Creation of Links and External_links.

[Mautly]

Hello,

I am working at Helmholtz-Zentrum Berlin and I am currently developing a tool to create nexus files using a description of the nexus structure format as an input. The software runs on Javascript (NodeJS) and I am using H5Web. I have managed to create groups, attributes and dataset satisfactory, but I could not find a way to create links and external_links. Looking the source code I only see these two functions in the Group class:

• get_link(obj_path: string)
• get_external_link(obj_path: string)

I suppose the creation of links is not yet implemented and therefore I would like to ask if there are plans to implement them. Or if there is way to create them using the current version of the library, could you please guide me through the process?

Thank you in advance,
Hector

[bmaranville]

You are correct - it is not yet implemented. I don't imagine it will be very difficult. Do you have in mind to create soft links or hard links? (external links are always soft links)

[Mautly]

I want to use soft links to avoid inserting more than once any dataset. I am using Nexus format and normaly the dataset is set to the detector, but there is a group called Data that contains a link to this dataset. If this soft links can be implemented in H5wasm, that would be super helpful. I do not would like to use h5py just for the creation of links. Thank you in advance.

[bmaranville]

Oh yes, I know about NeXus and the NXdata folder and all that... I'm a neutron scatterer by day :)
I'll put in a function to create soft links when I get a chance.

Can you comment on why you prefer the (nodejs + h5wasm) stack to (python + h5py) for writing hdf5? We use h5py in-house for writing, and we're hoping to use H5Web for visualization, and I'm curious what advantages you find in using nodejs.

[Mautly]

Thank you very much! This will help us a lot. I have discuss the question about soft links and external links with the team and I have realized that the creation of external links will be also needed. But starting with the soft links will already provide us a lot of help.

I have made the choice of working with (Nodejs + h5wasm) for several reason:

• I want to create a tool that runs out of the box. With NodeJS it is possible to encapsulate everything in a single executable file. Therefore, there is no need to install depencies or ecosystems.
• Coffeescript allows you to write clean and undestandable code. Its syntax similarities to Python allows you to copy libraries writen in Python to NodeJS very easily.
• Javascript/Coffeescript is 2-5x faster than Python. Here is the [link](https://edemaine.github.io/coffeescript-for-python/#comments](https://edemaine.github.io/coffeescript-for-python/) where this claim has been made.
• My personal experience regarding Python versions ecosystem is not god. I am always praying to gods when I try run new packages inside my python ecosystem.
• The installation of a new Python ecosystem, at least in Windows, is not straight forward. Specially if you want to use miniconda to work with version 3.7, which is the version required for Bluesky.
• NodeJS can run almost everywhere. For example, I sometimes debug the code in Termux (Android terminal) while drinking a coffee ;-) Besides the comfort, it allows me to check how the code performs in deviced with poor computing performance.

To put everything in context, I can say that the tool I am working on, called NexusCreator, will be used by 39 line beamlines at HZB. The idea is that the final user do not have to write code lines, as debugging can become easily a big issue. I just want people to modify the nexus structure in a text file and execute NexusCreator to generate their nexus files. This is a new project and there is still a lot of discussion going on, therefore I want be flexible and Web technologies offers this flexibility. The proof is how H5wasm is being accepted as a tool for visualizing Nexus/HDF5 files, and how cool it is that it is possible to run it in a web browser, as a stand-along program or integrated to Visual Code.

As a note:
H5wasm is the only library that I am using that cannot be encapsulated into the executable. But It is possible to place everything (executable + H5wasm library) in a single folder for distribution.

[bmaranville]

I just published v0.4.9 on npm and github. Instructions for using create_soft_link, create_hard_link and create_external_link can be found in the Links section of the README.md https://github.com/usnistgov/h5wasm/blob/main/README.md#links

[bmaranville]

Thanks for the feedback on your use case! Can you provide a link to the NexusCreator code?

Also, it might be possible to embed the WASM binary in the output of the esbuild step - I will look into that. I think that might solve your problem of encapsulation.

For instance, the WASM binary is embedded in the IIFE build of h5wasm, which is a single file: dist/iife/h5wasm.js
This build is based on the ESM build, so it has the MEMFS, WORKERFS and IDBFS filesystem libraries included, while the build in dist/node/hdf5_hl.js and dist/node/hdf5_util.js uses NODERAWFS for direct access to the OS filesystem.

I'm guessing you want NODERAWFS for your use case?

[bmaranville]

I was able to build a standalone vercel pkg script with h5wasm:

// index.mjs
import { h5wasm } from './node_modules/h5wasm/dist/node/hdf5_hl.js';

async function main() {
  const cwd = process.cwd();
  const { FS } = await h5wasm.ready;
  const f = new h5wasm.File(`${cwd}/test.h5`, "w");  
  f.create_group("entry");
  f.close();
}
main();

npx esbuild --bundle --format=iife --platform=node --define:import.meta.url=__dirname --outfile=build.js index.mjs;
npx pkg --target host build.js;

then running ./build results in the creation of a new hdf5 file test.h5 in the local directory.

[Mautly]

Thank you very much!!! It is a nice New Year surprise to see that it has been implemented :-)

The code for NexusCreator is for the moment contained in a private HZB repository, but I'll share the link when the code become public. I am not an experte on emscripten but [NODERAWFS] is OK for me to access files locally.

And thank you for the tip on how to encapsulate. I am using "nexe" tool to create an executable. I will try "npx".

Best wishes for this 2023.

[bmaranville]

Happy New Year to you too!

The npx above is just using npm to execute locally-installed packages instead of globally-installed (I tend not to install things globally), and I'm using esbuild to first bundle the app to a single .js file, then using pkg (from the vercel project) to create an executable from that file, so in this case pkg would be the replacement for your nexe.