Random failure publishing new secrets on changes
mkherlakian opened this issue · 4 comments
We've successfully deployed the Doppler Operator to our environments a little over 2 months ago.
We're noticing however that while everything works as it should post-deployment (we can change a secret in our config, and run kubectl describe secrets --selector=secrets.doppler.com/subtype=dopplerSecret
, we will see the changes almost immediately), after a while (days) the operator doesn't seem to pick up the new secrets anymore.
In order to fix this, we have to delete the CRD for that environment and recreate it.
We dug through the logs, but didn't notice anything out of the ordinary.
Happy to provide more info/details/logs that would help in diagnosing this. We are on GKE, k8s 1.23.
Hey @mkherlakian, thanks for reporting this!
When the operator is in the "stuck" state, could you share the output from kubectl describe
on the CRD itself? There are attached conditions which are often useful.
Did you check out the operator manager logs with this script? https://github.com/DopplerHQ/kubernetes-operator/blob/main/tools/operator-logs.sh
They're a bit verbose but often tell the whole story.
Hi @mkherlakian,
Are you still running into this issue?
Hi @nmanoogian,
AS it turns out I had to do a deployment in a namespace that we don't use often, and sure enough that one was not picking up the changes. Here's a describe
of the CRD.
Name: dopplersecrets.secrets.doppler.com
Namespace:
Labels: owner=doppler
Annotations: controller-gen.kubebuilder.io/version: v0.4.1
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2022-07-29T17:47:45Z
Generation: 1
Managed Fields:
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:kind:
f:listKind:
f:plural:
f:singular:
f:conditions:
k:{"type":"Established"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
k:{"type":"NamesAccepted"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
Manager: kube-apiserver
Operation: Update
Time: 2022-07-29T17:47:45Z
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:controller-gen.kubebuilder.io/version:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:owner:
f:spec:
f:conversion:
.:
f:strategy:
f:group:
f:names:
f:kind:
f:listKind:
f:plural:
f:singular:
f:scope:
f:versions:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-29T17:47:45Z
Resource Version: 349291445
UID: 8054024a-1c72-410d-b85d-2fc72f8dadca
Spec:
Conversion:
Strategy: None
Group: secrets.doppler.com
Names:
Kind: DopplerSecret
List Kind: DopplerSecretList
Plural: dopplersecrets
Singular: dopplersecret
Scope: Namespaced
Versions:
Name: v1alpha1
Schema:
openAPIV3Schema:
Description: DopplerSecret is the Schema for the dopplersecrets API
Properties:
API Version:
Description: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
Type: string
Kind:
Description: Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
Type: string
Metadata:
Type: object
Spec:
Description: DopplerSecretSpec defines the desired state of DopplerSecret
Properties:
Host:
Default: https://api.doppler.com
Description: The Doppler API host
Type: string
Managed Secret:
Description: The Kubernetes secret where the operator will store and sync the fetched secrets
Properties:
Name:
Description: The name of the Secret resource
Type: string
Namespace:
Description: Namespace of the resource being referred to. Ignored if not cluster scoped
Type: string
Required:
name
Type: object
Processors:
Additional Properties:
Properties:
Type:
Description: The type of process to be performed, either "plain" or "base64"
Enum:
plain
base64
Type: string
Required:
type
Type: object
Description: A list of processors to transform the data during ingestion
Type: object
Resync Seconds:
Default: 60
Description: The number of seconds to wait between resyncs
Format: int64
Type: integer
Token Secret:
Description: The Kubernetes secret containing the Doppler service token
Properties:
Name:
Description: The name of the Secret resource
Type: string
Namespace:
Description: Namespace of the resource being referred to. Ignored if not cluster scoped
Type: string
Required:
name
Type: object
Verify TLS:
Default: true
Description: Whether or not to verify TLS
Type: boolean
Type: object
Status:
Description: DopplerSecretStatus defines the observed state of DopplerSecret
Properties:
Conditions:
Items:
Description: Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions. For example, type FooStatus struct{ // Represents the observations of a foo's current state. // Known .status.conditions.type are: "Available", "Progressing", and "Degraded" // +patchMergeKey=type // +patchStrategy=merge // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,1,rep,name=conditions"`
// other fields }
Properties:
Last Transition Time:
Description: lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
Format: date-time
Type: string
Message:
Description: message is a human readable message indicating details about the transition. This may be an empty string.
Max Length: 32768
Type: string
Observed Generation:
Description: observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
Format: int64
Minimum: 0
Type: integer
Reason:
Description: reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
Max Length: 1024
Min Length: 1
Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
Type: string
Status:
Description: status of the condition, one of True, False, Unknown.
Enum:
True
False
Unknown
Type: string
Type:
Description: type of condition in CamelCase or in foo.example.com/CamelCase. --- Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be useful (see .node.status.conditions), the ability to deconflict is important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
Max Length: 316
Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
Type: string
Required:
lastTransitionTime
message
reason
status
type
Type: object
Type: array
Required:
conditions
Type: object
Type: object
Served: true
Storage: true
Subresources:
Status:
Status:
Accepted Names:
Kind: DopplerSecret
List Kind: DopplerSecretList
Plural: dopplersecrets
Singular: dopplersecret
Conditions:
Last Transition Time: 2022-07-29T17:47:45Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: 2022-07-29T17:47:45Z
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1alpha1
Getting the logs now.
Thanks, @mkherlakian! Any luck on getting those logs? Things are looking normal from the CRD describe.