[Feature Request] ISA-JSON export and format alignment

[xiaoranzhou]

Is your feature request related to a problem? Please describe.
The current version of ARCtrl export only short ISA-JSON
{ "identifier": "Facultative-CAM-in-Talinum", "title": "Facultative CAM in Talinum triangulare", "description": "Drought tolerance is a key factor for agriculture in the 21st century as it is a major determinant of plant survival in natural ecosystems as well as crop productivity. Plants have evolved a range of mechanisms to cope with drought, including a specialized type of photosynthesis termed Crassulacean acid metabolism (CAM). CAM is associated with stomatal closure during the day as atmospheric CO2 is assimilated primarily during the night, thus reducing transpirational water loss. The tropical herbaceous perennial species Talinum triangulare is capable of transitioning, in a facultative, reversible manner, from C3 photosynthesis to weakly expressed CAM in response to drought stress. The transcriptional regulation of this transition has been studied. Combining mRNA-Seq with targeted metabolite measurements, we found highly elevated levels of CAM-cycle enzyme transcripts and their metabolic products in T. triangulare leaves upon water deprivation. The carbohydrate metabolism is rewired to reduce the use of reserves for growth to support the CAM-cycle and the synthesis of compatible solutes. This large-scale expression dataset of drought-induced CAM demonstrates transcriptional regulation of the C3–CAM transition. We identified candidate transcription factors to mediate this photosynthetic plasticity, which may contribute in the future to the design of more drought-tolerant crops via engineered CAM.", "submissionDate": "July 07, 2015", "publicReleaseDate": "Nov 05, 2015", "ontologySourceReferences": [], "publications": [], "people": [], "studies": [], "comments": [] }

while using
toJsonString(arc.ISA)

Describe the solution you'd like
I would like to use ARCtrl to export ISA-JSON use for MARC import.
The ISA-JSON should be similar to the ISA-JSON export generted by ARCcommander or ISAdocNET (in attachment)
The export can be done by using read ARC ans export ISA script (in attachment).
In the best case also aligned with the ISA-JSON generated by VIB (in attachement.)

Describe alternatives you've considered
As the ISA object can be seen by using the console.log(arc.ISA), I think only conversions are needed.

Additional context
Add any other context or screenshots about the feature request here.
Additional desired behaviors are:

1. Use "@id" to link the "characteristicCategories"

ARCcommander_ISA-JSON.json
Read_ARC_and_export_ISA-JSON.zip
Belgium_VIB.json

[Freymaurer]

Note to future me. This is the reason behind this:

[xiaoranzhou]

Hi, just a quick FYI. I am back from vacation and ready for questions and help to accelerate this up :D

[HLWeil]

Hey @xiaoranzhou, thanks for the offer. I'm currently working on #365. Will let you know when that is done and I can look into the ARCtrl<->MARS alignment.

[HLWeil]

@xiaoranzhou, the issue with the studies and assays missing in the exported json should be fixed in the latest version 2.0.0-alpha6. Could you confirm this?

[HLWeil]

I'll get to work on the @id mechanic.

[xiaoranzhou]

I tried to update the package by npm i ./ @nfdi4plants/arctrl@2.0.0-alpha.6
and then rerun the node readARC.js

Got the following error:

`node readARC.js

node:internal/errors:496

ErrorCaptureStackTrace(err);

^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/home/xr/ARCtrl/node_modules/@nfdi4plants/arctrl/ISA/ISA.Json/Investigation.js' imported from /home/xr/ARCtrl/readARC.js

at new NodeError (node:internal/errors:405:5)

at finalizeResolution (node:internal/modules/esm/resolve:327:11)

at moduleResolve (node:internal/modules/esm/resolve:946:10)

at defaultResolve (node:internal/modules/esm/resolve:1132:11)

at nextResolve (node:internal/modules/esm/loader:163:28)

at ESMLoader.resolve (node:internal/modules/esm/loader:835:30)

at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)

at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:77:40)

at link (node:internal/modules/esm/module_job:76:36) {

code: 'ERR_MODULE_NOT_FOUND'

}

Node.js v18.18.2

`

My "readARC.js" is here:
readARC.zip

[Freymaurer]

Sadly my browser tells me not to download this file 😅

But.. for ARCtrl >= v2.0.0 we decided to streamline the api, which sadly meant we had to introduce breaking changes.

You can now find all relevant apis by just importing @nfdi4plants/arctrl. If you are looking for json, which i assume, based on the path: @nfdi4plants/arctrl/ISA/ISA.Json/Investigation.js. You can import JsonController.

[xiaoranzhou]

readARC.txt

@Freymaurer Hi Kevin, I have uploaded the file as a .txt file. You can download it and then rename it to .js to run it.

As I can neither test the "study" export nor the "@id" feature. Could you please reopen this issue to keep track on it.
It will be great if you can offer me another example to export JSON from a created ARC.
Thanks in advance 😄

[Freymaurer]

Here is an example for isa json export:

1. Download the .zip below.
2. Unzip
3. npm i to install dependencies
4. node arctrl.js to run

arc_js_test.zip

[xiaoranzhou]

Dear Kevin,
Thanks for your nice code example. 😄
I tried your code and the code itself worked. But your code creates an ARC and then generated the output ISA.JSON. This type of ISA.JSON was correct even from the previous version.
Other than this, I can not find any example that reads a created ARC. It will be great if you can provide me such example. It will take more time from myside to dive deep and found the correct reference.
Best wishes,
Xiaoran

[Freymaurer]

Dear Kevin, Thanks for your nice code example. 😄 I tried your code and the code itself worked. But your code creates an ARC and then generated the output ISA.JSON. This type of ISA.JSON was correct even from the previous version. Other than this, I can not find any example that reads a created ARC. It will be great if you can provide me such example. It will take more time from myside to dive deep and found the correct reference. Best wishes, Xiaoran

@xiaoranzhou - we have examples for reading and writing arcs here: https://github.com/nfdi4plants/ARCtrl/blob/main/docs/scripts_js/ARC.js

[xiaoranzhou]

Dear Kevin, Thanks for your nice code example. 😄 I tried your code and the code itself worked. But your code creates an ARC and then generated the output ISA.JSON. This type of ISA.JSON was correct even from the previous version. Other than this, I can not find any example that reads a created ARC. It will be great if you can provide me such example. It will take more time from myside to dive deep and found the correct reference. Best wishes, Xiaoran

@xiaoranzhou - we have examples for reading and writing arcs here: https://github.com/nfdi4plants/ARCtrl/blob/main/docs/scripts_js/ARC.js

Yes, I created a script mentioned here to read ARCs from those scripts and the script was working before the current updated ARCtrl. But the script was not working anymore. I can only test it after I finish the current update of DataPLAN.

[xiaoranzhou]

OK, here is my current progress:
`
// JavaScript
import * as ARC from '@nfdi4plants/arctrl';
import fs from "fs";
import path from "path";

// Setup
function normalizePathSeparators (str) {
const normalizedPath = path.normalize(str)
return normalizedPath.replace(/\/g, '/');
}

export function getAllFilePaths(basePath) {
const filesList = []
function loop (dir) {
const files = fs.readdirSync(dir);
for (const file of files) {
const filePath = path.join(dir, file);

      if (fs.statSync(filePath).isDirectory()) {
          // If it's a directory, recursively call the function on that directory
          loop(filePath);
      } else {
          // If it's a file, calculate the relative path and add it to the list
          const relativePath = path.relative(basePath, filePath);
          const normalizePath = normalizePathSeparators(relativePath)
          filesList.push(normalizePath);
      }
  }

}
loop(basePath)
return filesList;
}

// put it all together
async function read(basePath) {
let allFilePaths = getAllFilePaths(basePath)
// Initiates an ARC from FileSystem but no ISA info.
let arc = ARC.ARC.fromFilePaths(allFilePaths)
// Read contracts will tell us what we need to read from disc.
let readContracts = arc.GetReadContracts()
console.log(readContracts)
let fcontracts = await Promise.all(
readContracts.map(async (contract) => {
let content = await fulfillReadContract(basePath, contract)
contract.DTO = content
return (contract)
})
)
arc.SetISAFromContracts(fcontracts);
console.log(fcontracts);
return arc
}

// execution

await read("Facultative-CAM-in-Talinum").then(
arc => console.log(toJsonString(arc.ISA))

)`

I have already found out that the previous "fromFilePaths" is now relocated at the "ARC.ARC.fromFilePaths".
However, I can not find fulfillReadContract anymore. I guess either the file name has been changed or it is now better to use SetISAFromContracts() to create contracts?

[HLWeil]

The second part of this issue, @id referencing implemented in #392