eiffel-community/eiffel-intelligence

Add flowContext to Artifacts aggregation

Closed this issue · 11 comments

Description

We need the information from a linked FCD event to be added to the rules file for at least the Artifacts aggregation, to be able to subscribe on status updates on specific flow contexts.

Motivation

In some cases it is not enough to only subscribe to artifact identifiers. Those subscriptions need to be restricted by a flow context.

Exemplification

The same artifact identifier can be used in multiple different flow contexts (branches, projects, programs, etc) and that is specified using a link to an FCD event. To be able to subscribe only to updates on the branch/projec/program that I'm interested in, I need to be able to provide a flow context specification in the subscription.

Benefits

Unnecessary subscription triggers kicked off

Possible Drawbacks

None

I'll look into this - I will begin by extending the existing artifact rules for Agen version.

Thanks. FLOW_CONTEXT could come in any event I think. It's probably most interesting to filter subscriptions on flow contexts in the artifacts part of the aggregated object, but it could also be interesting to filter on flow contexts in the confidence level parts of the aggregated object. I would propose you read out the flow context from any incoming event to be aggregated into the object and add it to the respective part of the aggregated object.

By what we've discussed previously I assume the proposed structure of an aggregation made by EI would look something like this:

{
    "_id" : "df4cdb42-1580-4cff-b97a-4d0faa9b2b22",
    "fileInformation" : [ 
        {
            "extension" : "war",
            "classifier" : ""
        }
    ],
    "artifactCustomData" : [],
    "buildCommand" : "trigger",
    "identity" : "pkg:maven/com.mycompany.myproduct/artifact-name@2.1.7",
    "name" : "myArtifact",
    "flowContexts": [
        {
            "eventId": "aaaaaaaa-bbbb-5ccc-8ddd-dkt8ejeeeee0",
            "program": "NewMarket",
            "product": "MyProduct",
            "track": "CustomerXAdaptions",
            "version": "1.0.2",
            "project": "NewImportantFeatures"
        },
        {
            "eventId": "fffffttt-bbbb-5ccc-8ddd-dkt8ejeeeee0",
            "program": "SeparateProgram",
            "product": "MyProduct",
            "track": "MaintenanceRelease",
            "version": "1.0.2",
            "project": "HotFix"
        }
    ]
    "confidenceLevels" : [ 
        {
            "eventId" : "e3be0cf8-2ebd-4d6d-bf5c-a3b535cd084e",
            "name" : "dummy_1_stable",
            "time" : 1521452400324,
            "value" : "SUCCESS",
            "flowContexts": [
                {
                    "eventId": "ttttgggt-bbbb-5ccc-8ddd-dkt8ejeeeee0",
                    "program": "TestProgram",
                    "product": "MyProduct",
                    "track": "robust_testing",
                    "version": "4.0.8",
                    "project": "TestProjectA"
                }
            ]
        }
    ],
    "TemplateName" : "ARTIFACT_1",
    "id" : "df4cdb42-1580-4cff-b97a-4d0faa9b2b22",
    "time" : 1521452368194,
    "type" : "EiffelArtifactCreatedEvent",
   ...
}

The list of flowContexts objects on root level are connected to the ArtC event which starts the aggregation. In this example the ArtC had two links to separate FlowContext events. There can also be nested lists of flowContexts inside the aggregated object, which belongs together with data extracted from the other Eiffel events, such as inside the confidenceLevels object extracted from the CLM event. This scenario assumes the FlowContextDefined events already exist in ER so that EI can perform lookup for historical events and aggregate them.

Hopefully this is similar to what you requested, @e-backmark-ericsson ?

Having some difficulties with extracting the FlowContextDefined event at all into the aggregated object right now... I'll continue looking into this.

Yes, that seems to be inline with what I'd expect. We should note that FCD events can be 'recursive' - https://github.com/eiffel-community/eiffel/blob/master/eiffel-vocabulary/EiffelFlowContextDefinedEvent.md#flow_context, but I think we could exclude that in the aggregation to start with if it complicates things too much.

Not sure I follow your remark on FCD events being recursive, can you elaborate on that?
Update on the progress so far: I've managed to get EI to aggregate all FCD event links from the ArtC event which is also the start event of the aggregation.

However the additional functionality of being able to trace any FCD events from the CLM or ArtP events coming later in the aggregation chain does not seem to be possible, with the way EI behaves today. EI does not follow any linked event history unless it's a start event (which CLM and ArtP is not in this aggregation flow). I'm unsure if this was a deliberate design decision taken previously (which need to be documented better if so) or if it's something that needs fixing... @vasile-baluta any input?

vasba commented

Hi!

The decision to only do upstream search for start event was made because we wanted to fulfill the initial upstream aggregation which in initial requirements was only related to start event.

You can remove the condition for start event but then you need to implement rule that takes ER search parameters in order to limit the search to only one level and only wanted type.

Today the entire upstream tree is fetched and selection of the events to be retained is done by declaring history extraction rules. If an event type in the tree has history extraction rules defined it is agregated if not then it is skiped.

Thank you @vasba for the clarification! It could be a future improvement. I noticed that @e-backmark-ericsson created a new issue for this #365 . So the scope for this issue will be just adding FlowContextDefined into the artifact aggregation. I'm currently finishing up on testing and updating the documentation for these changes.

I've written a separate issue for extending the upstream event aggregation from non-start-events in #374. I do agree with limiting upstream searches for those non-start-events with a specific link type, like @vasba suggested. Also to limit to only one level to avoid a too large aggregation.

Right now I only added the FLOW_CONTEXT link type but perhaps CONTEXT could also be interesting? If you have any comments or additional benefits regarding that issue, please add them when you have time!

The FlowContextDefined addition to the ArtifactRules have been merged on a branch and released as 2.0.2. In the long run the changes will also be merged into master, so they can be included in the next 3.0.0 release we plan for Eiffel Intelligence (which contain some non-backwards compatible changes).

Should we close this issue now or wait until the changes are also available in the master branch?

I guess we should keep it open until it is merged to master, or we might forget to verify that it is merged correctly to master.

These change have been merged into master. EI will include the addition of FlowContextDefined in the artifact ruleset. This extension of the rules is available from 2.0.2 release and onward. Therefore I'm closing this issue.