Source label does not reflect upstream pipeline label

[haimizrael]

Artifacts uploaded to S3 take on a label in the form {Pipeline Counter}.{Stage Counter}, which overrides the actual label of the pipeline that produced the artifact. This makes it more difficult to trace deployments of versions and respond to auditor requests.

For example, if you have a build/test/package pipeline, whose label is set to 2.1.3.${COUNT}, it will produce an artifact with version label like 2.1.3.103. The S3 package label, however, would be something like 103.1. The issue this causes is for example if you viewed possible versions to deploy in a deployment pipeline, you would see versions like 103.1, 102.1, 101.1 rather than the actual versions produced by the build pipeline.

While it's possible to find out the original version with the trackback URL, that's quite inconvenient, as you have to click each one. We have a temporary work-around in that we use both the upstream build pipeline as well as the s3 package as materials. We fetch the s3 package, but use the label of the upstream pipeline. Works, but obviously not ideal, and it won't work if the upstream pipeline is on a different Go server.

We're looking into possible remediation, including adding the original label as object metadata.

Please let me know if this makes sense or perhaps we missed something in how we have set up our pipelines. I'd appreciate any feedback.

Thanks

[ashwanthkumar]

When we built the publish plugin, we wanted to mimic the Go's artifact storage scheme. Reason being - We needed consistent way for the s3 poller to find new versions - Since GO_PIPELINE_LABEL can have any arbitrary string in it, it'll be impossible to find new versions.

One solution would be, If you want publish artifacts under custom directory scheme, you could use the "Destination" field in the Publish dialog box. It also accepts Environment variables based value rendering - ex. you can use GO_PIPELINE_LABEL in the path.

PS: If you use custom destination path for publishing, the behaviour of S3Poller is undefined.

[manojlds]

We're looking into possible remediation, including adding the original label as object metadata.

@haimizrael That approach should be possible. Would that be a satisfactory solution?

[haimizrael]

Thanks, @ashwanthkumar and @manojlds:

Using the pipeline/stage counters as a consistent way to poll makes total sense - I assumed that was the reason for the existing setup.

Using custom directory scheme won't work for us since we'd want the S3Poller to detect changes correctly (though the custom directory scheme is a pretty cool feature for conducting "deployments" into S3 using s3publish - I don't know if that was the original intent).

Adding the original label as object metadata would work, only if that could then be used by the S3Material to set the resulting label in Go. If that would actually work, we'd like to explore that option. Obviously, that would be a breaking change, so it'd likely need to be configurable to be backward compatible with how it works now.

[brewkode]

Implicitly making the poller use object metadata if present else falling
back to the existing convention should be enough, isn't it? A separate
flag to indicate which place to look for might be an overkill, I guess.
Some releases down the line, we can fully shift to using metadata, if that
helps fully.

On Wed, Feb 10, 2016 at 10:55 PM, Haim S Izrael notifications@github.com
wrote:

Thanks, @ashwanthkumar https://github.com/ashwanthkumar and @manojlds
https://github.com/manojlds:

Using the pipeline/stage counters as a consistent way to poll makes total
sense - I assumed that was the reason for the existing setup.

Using custom directory scheme won't work for us since we'd want the
S3Poller to detect changes correctly (though the custom directory scheme is
a pretty cool feature for conducting "deployments" into S3 using s3publish

• I don't know if that was the original intent).

Adding the original label as object metadata would work, only if that
could then be used by the S3Material to set the resulting label in Go. If
that would actually work, we'd like to explore that option. Obviously, that
would be a breaking change, so it'd likely need to be configurable to be
backward compatible with how it works now.

—
Reply to this email directly or view it on GitHub
#29 (comment)
.

[ashwanthkumar]

-1 for using object metadata for detecting new versions. Reasons being

1. There'll be performance impact while using that approach. Today we do one listing on the configured bucket, get the latest and do only one GetObjectMetadataRequest. If we have to use metadata - we'll end up using that many number of GetObjectMetadataRequest as there are listings.
2. While in-memory caching is an option for already seen directories - If the pipeline label for some reason doesn't use ${COUNT} - it becomes hard or unnecessarily complicated.

[haimizrael]

@manojlds @ashwanthkumar @Sriram-R
We came up with a workable solution by:

1. Adding the original pipeline label as object metadata during s3publish (e.g. go_pipeline_label: 'x.y.z')
2. s3Material, if it finds such metadata, sets the Revision Comment to "Original revision number: x.y.z"

This does not change the behavior of pipeline label used by the plugins (still is pipeline counter.stage counter), but with the revision comment one can easily see the original label, since the comment appears in multiple places:

• the pipeline changes hover window
• trigger with options window (in the revision to trigger with list)
• in the stage details window (right above the trackback URL)
• in the value stream map (in the package icon, above the trackback URL)

This solution works for us, and is backward-compatible (if there is no revision label metadata, the revision comment is empty string and Go does not show it). If you are OK with this, we'll do a PR with this change once the original PR is merged. As mentioned, we have developed a few other features, but I assume it would be cleaner to PR these individually so it is easier to review and control the changes.

[ashwanthkumar]

@haimizrael Just to be clear, you're suggesting we can replace the highlighted section in the image with the pipeline label instead of the default ${COUNT}.${GO_STAGE_COUNTER} ?

[haimizrael]

No, that label stays intact. You get a revision comment shown above the trackback:

[ashwanthkumar]

Ah Sweet! 👍
@manojlds @Sriram-R Thoughts?

[haimizrael]

Addressed by PR #33