File-based module system

[iacore]

Currently, module.ens doesn't do much. I think it's better to specify dependencies with relative paths inside source files. This makes it easy to reason about dependencies, and if Neut would have a package manager, no changes needed to made to the core language. The package manager can simply download files to a location, and the Neut compiler would find files based on relative paths.

How other languages does this:
Zig: const mod = @import("dep.zig")
ES (Browser/Deno): import * as mod from './dep.js'

It would be easier to integrate neut with C/Zig too. Something like neut -c -o out.o hello.nt is easier to learn as well.

[vekatze]

Yes, I also loved the simplicity of file-based module system (actualy, the module system was file-based in earlier version of neut). But I eventually turned it into project-based to see how the latter approach behaves with respect to versioning.

I think this needs some explanation. Please allow me to be a bit verbose.

---

The first thing to note is that neut already contains a package manager (-ish mechanism). You can create a package via neut release SOME_NAME. For example:

~/Desktop
❯ neut init sample
~/Desktop
❯ cd sample
~/Desktop/sample
❯ neut release 0.1.0.0
~/Desktop/sample
❯ tree
./
├── release/
│  └── 0.1.0.0.tar.zst
├── source/
│  └── sample.nt
└── module.ens

Then you can publish the resulting tar.zst by, for example, committing and pushing it to GitHub.

Now you can add the package to a project via neut get SOME_ALIAS URL_TO_THE_PACKAGE. This, by default, downloads the package into: $XDG_CACHE_HOME/neut/ARCHNAME-OSNAME/COMPILER_VERSION/library/ENCODED_CHECKSUM/. This also writes package info (the checksum, the URL, and the alias) to module.ens.

Then you can import the package as follows:

import
- SOME_ALIAS.foo // assuming that the package contains /source/foo.nt
end

define bar(): i64 =
  SOME_ALIAS.foo::some-function(3) // assuming `foo.nt` contains a definition of `some-function`.
end

And here comes the interesting point. The SOME_ALIAS part in SOME_ALIAS.foo::some-function is resolved into the corresponding checksum by the compiler. Conceptually, a symbol like SOME_ALIAS.foo::some-function is automatically resolved into something like:

IIc6mUoHKN5bBYVxKcKQ6ajJu6IjbMMLz_p7dG_-khg=.foo::some-function

assuming that the encoded checksum of the tarball is IIc6mUoHKN5bBYVxKcKQ6ajJu6IjbMMLz_p7dG_-khg=. The compiler uses this symbol during compilation.

This means that every package is identified by its content, not by its name. In particular, this also means that technically there doesn't exist a thing like "the same package but different versions" in neut.

During investigation, I came up with this approach and felt like studying it. I felt like seeing how it works and fails. This is why project-based module system is used now (... I won't surprise if there exists a project with a similar approach, though).

The file-based approach can be better at FFI and other aspects. Still, please give me some time to evaluate the current approach. Some time to wait for the dust of trying to settle.

[iacore]

You should look at Unison Web. They hash the AST of every function, not tarball.

[vekatze]

Thanks for the pointer. I'll look into it.