Secondary Secret not discovered when CR applied after operator start

[ebma16]

Environment

• Operator SDK: 5.1.2
• Operator SDK Spring Boot Starter: 6.1.1
• Kubernetes: Kind cluster, version 1.32

Issue Summary
I am observing unexpected behavior when watching a secondary Secret resource in my operator application. The operator application consists of two reconcilers: CrAReconciler and CrBReconciler. When the operator application starts, both reconcilers are registered, and I observe the following behavior:

• If the operator starts and a CrB Custom Resource (CR) already exists, CrBReconciler correctly finds the Secret in its cache (e.g., via context.getSecondaryResources(Secret::class.java)).
• If the operator starts and the CrB CR does not exist initially (but is applied later), CrBReconciler does not find the Secret in its cache when the CrB CR is created. The Secret is only discovered after either restarting the operator application or modifying/recreating the Secret.

Important Note
The Secret is not managed by the operator itself. In the Reconciler, only an additional event source is created to watch the external Secret.

Expectation
The Secret should also be discovered when the CrB CR is applied after the operator has already started.

Reproducer
I have created a reproducer demonstrating this behavior: https://github.com/ebma16/josdk-sample-secret-not-in-cache

This reproducer is available for review and testing for @csviri and @metacosm

[metacosm]

@ebma16 looks like the repo is private

[ebma16]

Hi @metacosm,

Yes, the repo is private. I’ve added you and @csviri as collaborators so you can take a look. Is that sufficient, or do you need a publicly accessible repository?

[csviri]

@ebma16 pls add also @metacosm and @xstefank or make it public

[ebma16]

@metacosm has already been added as a collaborator, and I’ve now added @xstefank as well. All three of you now have access to the repository.

[csviri]

Hi @ebma16 it looks that I won't have time to take a look on this in next 2 weeks, but note that in core almost all tests cover this scenario so almost certainly something wrong on your side.

@metacosm @xstefank if you have the time pls take a look

[ebma16]

Thanks for your reply!

To summarize, the only thing I did in my reconciler was register an EventSource (as described in the documentation) for a Secret that is not managed by the operator—essentially a read-only resource.

I have already double-checked whether I might be doing something wrong, but unfortunately, I couldn’t detect any changes I could make to get this working.

The behavior that I find unexpected is: the Secret can be found in the cache when the Primary exists, but cannot be found when the Primary is initially absent and created after the operator starts. This made me wonder if there might be an edge case in the SDK. I initially hoped that issue #2869 would resolve this, but it didn’t change this behavior.

Any guidance or help from anyone would be greatly appreciated!

[xstefank]

Hi @ebma16,

what you need is the other way around: primary -> secondary. Add something like this:

 .withPrimaryToSecondaryMapper { res: ResourceClass ->
                        val secretName = "my-secret"
                        mutableSetOf(ResourceID(secretName, res.metadata.namespace))
                    }

I ommitted the real values from your example but I can send this a patch to your repo if needed. But I think this should be OK for you to understand what you need to do. Just let me know if you would like a PR there.

[ebma16]

Hi @xstefank,

that did the trick, thank you so much!

Could you quickly explain why a PrimaryToSecondaryMapper is required in this context?

So far, my understanding based on this documentation was:

• A SecondaryToPrimaryMapper is used in a one-to-many relationship – one primary resource (CustomResource) and multiple secondary resources (in my case, multiple Secrets). That’s why I initially went for a SecondaryToPrimaryMapper.
• A PrimaryToSecondaryMapper is used in many-to-one or many-to-many relationships – multiple primary resources (CustomResources) and possibly multiple secondary resources (Secrets), where you need to explicitly define which secondary resources belong to which primary.

Sorry if I’m a bit confused about the exact intention of the two mappers – could you clarify this for me?

[xstefank]

@ebma16 great that it works!

TBH, I was also confused by this. The problem is that the index of SecondaryToPrimary mappings is created when the operator starts. And the SecondaryToPrimary requires a primary resource (your CR) to make the association from your secondary resource (your secret). This is why you saw your problematic behaviour:

1. Deployed secret.
2. Start the operator. The operator finds the secret, but there is no CR ready to be associated. So nothing is linked.
3. Deploy the CR. This doesn't trigger the rescan of secondary resources for the mapper to be invoked, so nothing happens.
4. Restart the operator. Now again, when the operator starts, it finds the secret, but this time, there is also the CR deployed for the mapping to be associated.

This is why you need PrimaryToSecondaryMapper. This reacts to the changes of the primary resource (your CR) and not the secondary resource (secret). So when the CR is deployed, this method runs and associates the secret with your CR.

[ebma16]

Thanks again for the explanation and the support, that really helps!

Issue can be closed!

[metacosm]

To be a little bit more explicit, if I understood your use case properly, you have a secret that exists on the cluster, a secret that is presumably not managed by the operator but that Custom Resources CrB need to check to get reconciled. Since the secret is managed outside of the operator, there is no information in it that would link it to any CrB resources. JOSDK uses the PrimaryToSecondaryMapper concept in that situation. This allows the developer of the operator to tell JOSDK which secondary resources need to be considered for a given primary resource. Without that mapper, as explained above, there is no way to know that the CrB reconciler needs to consider your secret when a primary resource is deployed and it would therefore not be found when getSecondaryResources is called.

We should probably clarify the documentation on that aspect.

[ds-akloskowski]

@metacosm Recently, we had a discussion about work of SecondaryToPrimaryMapper and PrimaryToSecondayMapper, and got a few thoughts about what could be added to documentation to make it more accurate. I hope that it will be helpful for you during update of documentation.

• Type of relationship between primary and secondary doesn't matter (I'm unsure about it, because didn't notice a case for which it may be problematic);
• If a secondary resource can determine its primaries (even not existing at the moment) without any cache (like context.getPrimaryCache()) then SecondaryToPrimaryMapper is enough, new primaries should be able to find their secondaries (then io/javaoperatorsdk/operator/processing/event/source/informer/DefaultPrimaryToSecondaryIndex.java will be used, which should be able to handle it);
• If a secondary resource requires cache to determine its primaries, then PrimaryToSecondaryMapper is required to able primary to find its own secondaries (without it, there can be a case when secondary exists before primary and DefaultPrimaryToSecondaryIndex won't be populated with data about relationship between primary and secondary, because of primary's absence in the cache).

[csviri]

Hi all create a page for this:
#2949