Reconciliation workflow related to `.status`

[marshtompsxd]

Zookeeper

The status of the cr object:

// ZookeeperClusterStatus defines the observed state of ZookeeperCluster
type ZookeeperClusterStatus struct {
	// Members is the zookeeper members in the cluster
	Members MembersStatus `json:"members,omitempty"`

	// Replicas is the number of number of desired replicas in the cluster
	Replicas int32 `json:"replicas,omitempty"`

	// ReadyReplicas is the number of number of ready replicas in the cluster
	ReadyReplicas int32 `json:"readyReplicas,omitempty"`

	// InternalClientEndpoint is the internal client IP and port
	InternalClientEndpoint string `json:"internalClientEndpoint,omitempty"`

	// ExternalClientEndpoint is the internal client IP and port
	ExternalClientEndpoint string `json:"externalClientEndpoint,omitempty"`

	MetaRootCreated bool `json:"metaRootCreated,omitempty"`

	// CurrentVersion is the current cluster version
	CurrentVersion string `json:"currentVersion,omitempty"`

	TargetVersion string `json:"targetVersion,omitempty"`

	// Conditions list all the applied conditions
	Conditions []ClusterCondition `json:"conditions,omitempty"`
}

Some of the fields above come from the status of other objects. Like ReadyReplicas comes from the stateful set object's status.

Meanwhile, the status of other state objects also affects the controller's behavior. In each reconciliation round, the controller checks whether all the zookeeper replicas are up-to-date by checking the stateful set's .Status.UpdateRevision and .Status.CurrentRevision. If the two are inconsistent for more than 10 minutes, meaning the upgrading lasts for that long time, then the controller will mark this upgrading as failed (in its .Status.Conditions). If the upgrading is marked as failed, the controller will stop updating the stateful set object even if new changes from the CR come. The upgrading failure mark will be resolved soon as .Status.UpdateRevision becomes the same as .Status.CurrentRevision, and then the controller can continue to update the stateful set. See:

func (r *ZookeeperClusterReconciler) upgradeStatefulSet(instance *zookeeperv1beta1.ZookeeperCluster, foundSts *appsv1.StatefulSet) (err error) {
    ...
    // Setting the upgrade condition to true to trigger the upgrade
    // When the zk cluster is upgrading Statefulset CurrentRevision and UpdateRevision are not equal and zk cluster image tag is not equal to CurrentVersion
    ...
    // checking if the upgrade is in progress
    if upgradeCondition.Status == corev1.ConditionTrue {
        ...
        // updating the upgradecondition if upgrade is in progress
        if foundSts.Status.CurrentRevision != foundSts.Status.UpdateRevision {
            ...
            err = checkSyncTimeout(instance, zookeeperv1beta1.UpdatingZookeeperReason, foundSts.Status.UpdatedReplicas, 10*time.Minute)
            if err != nil {
                instance.Status.SetErrorConditionTrue("UpgradeFailed", err.Error())
                return r.Client.Status().Update(context.TODO(), instance)
            }
            ...
        }
    }
    ...
}

func (r *ZookeeperClusterReconciler) reconcileStatefulSet(instance *zookeeperv1beta1.ZookeeperCluster) (err error) {
    // we cannot upgrade if cluster is in UpgradeFailed
    if instance.Status.IsClusterInUpgradeFailedState() {
        return nil
    }
    ...
}

Rabbitmq

The status of the cr object:

// Status presents the observed state of RabbitmqCluster
type RabbitmqClusterStatus struct {
	// Set of Conditions describing the current state of the RabbitmqCluster
	Conditions []status.RabbitmqClusterCondition `json:"conditions"`

	// Identifying information on internal resources
	DefaultUser *RabbitmqClusterDefaultUser `json:"defaultUser,omitempty"`

	// Binding exposes a secret containing the binding information for this
	// RabbitmqCluster. It implements the service binding Provisioned Service
	// duck type. See: https://github.com/servicebinding/spec#provisioned-service
	Binding *corev1.LocalObjectReference `json:"binding,omitempty"`

	// observedGeneration is the most recent successful generation observed for this RabbitmqCluster. It corresponds to the
	// RabbitmqCluster's generation, which is updated on mutation by the API Server.
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}

The rabbitmq controller checks the status of the stateful set object to see if all replicas are ready and updated before running rabbitmq commands (commands for rebalance or reset plugins). See:

func allReplicasReadyAndUpdated(sts *appsv1.StatefulSet) bool {
    return sts.Status.ReadyReplicas == *sts.Spec.Replicas && !statefulSetBeingUpdated(sts)
}

func statefulSetBeingUpdated(sts *appsv1.StatefulSet) bool {
    return sts.Status.CurrentRevision != sts.Status.UpdateRevision
}

func (r *RabbitmqClusterReconciler) runRabbitmqCLICommandsIfAnnotated(ctx context.Context, rmq *rabbitmqv1beta1.RabbitmqCluster) (requeueAfter time.Duration, err error) {
    logger := ctrl.LoggerFrom(ctx)
    sts, err := r.statefulSet(ctx, rmq)
    if err != nil {
        return 0, err
    }
    if !allReplicasReadyAndUpdated(sts) {
        logger.V(1).Info("not all replicas ready yet; requeuing request to run RabbitMQ CLI commands")
        return 15 * time.Second, nil
    }
     ...
}

Fluent

It does not have any status:

// FluentBitStatus defines the observed state of FluentBit
type FluentBitStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
}

and does not check the status of other objects.