Automated ingestion and access controls

[dylanmcreynolds]

I have an automated ingestion routine, triggered by a message from the beamline that a data set is ready to be ingested.

My ingestion service uses a functional account to ingest...generating a token through a call to catamel.

Currently, each object that ingest (Dataset, Proposal, OrigDatablock and Sample), I use the following:

{ 
  "ownerGroup": "ingestor",
  "accessGroups: ["groupA", "groupB"]
}

This works for ingesting the dataset. However, mys do not enjoy the "ingestor" role. So, non of them can edit files, add tags and attachments, etc.

What I would like to do is something like this:

{ 
  "ownerGroup": "proposalA",
  "accessGroups: ["groupA", "groupB"]
}

Where a user has proposalA in their profiles accessGroups field.

When I try this, the ingestion fails. The Dataset succeeds, but then subsequent objects fail because the ingestor user can't see the Dataset.

I have a slighly kludgy fix, setting ingestor in the accessGroups field of the Dataset.

{ 
 "ownerGroup": "proposalA",
 "accessGroups: ["groupA", "groupB", "ingestor"]
}

This works, but is a bit ungraceful. Is there a way to modify the ingestor user's access so that it can see the Dataset even if it's not in the ownerGroup or accessGroups of the Dataset?

[stephan271]

I would say that is a bug. The ingestor should see all datasets. What exactly is the error you get in your second case above ?

[dylanmcreynolds]

On the client I'm getting:

"Error creating datablock. {'statusCode': 403, 'name': 'Error', 'message': 'DatasetId not found. Could be access rule problem - test accessGroups for id: undefined'}")

And in the server:

Tue Sep 14 2021 15:55:19 GMT+0000 (Coordinated Universal Time): Update size error: Instance undefined can not be found. Could be access problem.

Unhandled error for request POST /api/v3/RawDatasets/als%2F203143a8-2188-4e57-b50e-ec0e501a87d1/origdatablocks?access_token=EwVhVCS4KXzlYxrp3l9VuqdEXJDLrbHBJiB6px012YF9QHa5vNC0H6eltDTAp4T5: Error: DatasetId not found. Could be access rule problem - test accessGroups for id: undefined

    at /usr/src/app/common/models/utils.js:33:23

    at /usr/src/app/node_modules/loopback-datasource-juggler/lib/dao.js:1801:62

    at /usr/src/app/node_modules/loopback-datasource-juggler/lib/dao.js:1737:9

    at /usr/src/app/node_modules/async/dist/async.js:1140:9

    at /usr/src/app/node_modules/async/dist/async.js:473:16

    at eachOfArrayLike (/usr/src/app/node_modules/async/dist/async.js:1057:9)

    at eachOf (/usr/src/app/node_modules/async/dist/async.js:1117:5)

    at _asyncMap (/usr/src/app/node_modules/async/dist/async.js:1133:5)

    at Object.map (/usr/src/app/node_modules/async/dist/async.js:1122:16)

    at allCb (/usr/src/app/node_modules/loopback-datasource-juggler/lib/dao.js:1648:13)

    at /usr/src/app/node_modules/loopback-connector-mongodb/lib/mongodb.js:1408:9

    at /usr/src/app/node_modules/mongodb/lib/utils.js:697:5

    at handleCallback (/usr/src/app/node_modules/mongodb/lib/utils.js:102:55)

    at cursor.close (/usr/src/app/node_modules/mongodb/lib/cursor.js:840:66)

    at /usr/src/app/node_modules/mongodb/lib/utils.js:697:5

    at _endSession (/usr/src/app/node_modules/mongodb/lib/cursor.js:925:9)

[stephan271]

Is this error appearing only recently, in the newer catamel versions ?

[dylanmcreynolds]

Unknown. I only started this setup recently.

[stephan271]

What is the result of "npm run test" on your catamel version ?