Resolve parameters JIT before executing bundle
Opened this issue · 2 comments
A little history: storage.Run was originally the CNAB claim document. Over time we evolved the resource to store what Porter needed to know about a bundle run, and it was renamed to Run. One thing that didn't change is that it is still being treated as the original intent of the claim, an append-only record of a bundle's execution.
One requirement of the CNAB claim document that we preserved on Run is the Run.Parameters field, which is the full set of resolved parameters (credentials are explicitly NOT supposed to be in the claim document). Porter doesn't persist all parameters on the Run, and excludes sensitive parameters. We need to double-check that we are writing sensitive parameters to the CNAB claim document that we inject into the bundle when it's run (i.e. we meet the CNAB requirement that all parameters are in the document, but we don't persist those sensitive parameters on the Run in our database). EDIT: We are storing secret names to lookup later (keyed by run id) on run.Parameters not raw sensitive values so we are 👍
The problem with the current solution is that when we add support for workflows, we want to be able to create a record of a PENDING run, where there could be a significant delay between when the workflow is created and the bundle is ultimately executed. We can't resolve the parameters so far in advance, and instead we need to resolve them just-in-time before execution.
We are very close to being able to do that already with Run. We already keep track of the parameter and credential set names used by the run, along with the parameter overrides. If we move the resolved parameters to the Run.Status field, we can hold off on populating that until immediately before running the bundle.
This mostly impacts the Run.ToCNAB function, and we'll need to make sure that we've resolved the parameters before using it to inject the claim into the bundle before running it.
This will allow us to schedule a run ahead of time and delay resolution to avoid stale data.
I am going to update Run directly and not introduce a Status field. The right way to handle such a change would be to modify how we access our records, and when we retrieve a resource that isn't the current version, JIT migrate it to the current version as part of our data layer. Eventually we may need to do that but for this it isn't worth it.
Since run isn't user-specified anyway, I'm not too worried about the inconsistency, and there's definitely and argument to be had that Run shouldn't have status anyway, the entire resource is a status essentially, auditing what was executed. So for now I am just stuffing new fields, such as a digest of the resolved parameters, directly onto run.
As part of this I am also going to calculate a hash of the resolved parameters and credentials. This will help us quickly determine if they have changed (when reconciling an installation) without requiring us to re-resolve the values from an old run, as long as the digest is set.