inconsistent preopened dirs between wasmtime and spin

[rajatjindal]

originally started the discussion here: https://discord.com/channels/926888690310053918/950022897160839248/1261589910511681598

hey folks, while debugging an issue in a guest component using filesystem api, I noticed that the preopened directories are different between wasmtime and spin.

is that intentional? if not, can we consider making this consistent across wasmtime and spin?

a repo to demonstrate the said discripency: https://github.com/rajatjindal/print-preopened

$ wasmtime run --dir src --dir wit target/wasm32-wasi/debug/print_preopened.wasm
src
wit

$ spin up                                                                       
/

thank you

[rylev]

Thanks for the report. I first wanted to make sure I could reproduce this in an http trigger (to eliminate the possibility of this somehow being an issue in the command trigger), and I was able to. This issue isn't a bug per say as the guest still has access to the same files at the same points in the virtual file system. The only thing that differs is the list of "preopened" file descriptors.

What's even more interesting is that behavior differs between spin up and spin up --direct-mounts (as you can see here at least for the local loader (I believe the OCI loader always mounts to /). With the --direct-mounts flag, the behavior matches that of wasmtime run.

I'm not sure exactly what the right answer is here. It feels like we should be consistent between regular spin up and spin up --direct mounts, but does it actually matter? This is perhaps a question for the conformance tests. How much do we want to force runtimes to mount exactly the same way?

@rajatjindal is there a particular use case where this discrepancy is causing an issue or is it just that the inconsistent behavior seems wrong (even if not causing actual bugs)?

cc @lann for 👀

[rajatjindal]

Thanks for checking @rylev .

I found it while trying to understand why "ls" command was behaving differently between wasmtime and spin for "wasm-console" project.

I think tinygo also makes use of these preopened dirs to provide filesystem api's (on top of wasip2)

Although I have a fix for it in wasm-console now, but it means some pieces of functionality takes different code paths between wasmtime and spin. And i was hoping to avoid that as it adds a bit of complexity.

Thanks for consideration

[lann]

I'd be interested to see how different preopen patterns interact with wasi-libc. For instance, if I have a preopen with guest path x/y, does that appear as /x/y to libc?

[itowlson]

The origin of this is that Spin allows globs or specific files to be mounted e.g. files = ["src/lib.rs"] (and indeed this was originally the only way to do mounts). That drove us down an implementation route of copying the selected files under a temp directory and pre-opening the root of the temp directory. When we added directory mounts, we followed that model so that pattern and directory syntax would play nicely together. And it worked nicely with OCI with its snapshots of individual files. However the "open a single root" scheme couldn't work with --direct-mounts which is why that ends up different.

To be clear I'm not arguing what we have is correct, just adding some context on why we ended up this way!

[rajatjindal]

Hi @lann, could you please point me to how could I verify that with wasi-libc specifically?

Hi @itowlson, thank you for the additional context. 👍

[lann]

Hi @lann, could you please point me to how could I verify that with wasi-libc specifically?

Most wasi implementations are using wasi-libc (via wasi-sdk), so just experimenting with e.g. Rust's std::fs and wasmtime run should work.

[rajatjindal]

with the following component code:

#[allow(warnings)]
mod bindings;

use crate::bindings::exports::wasi::cli::run::Guest;
use crate::bindings::wasi::filesystem::preopens;

struct Component;

impl Guest for Component {
    fn run() -> Result<(), ()> {
        let dirs = std::fs::read_dir("/src").expect("read_dir on /src");
        dirs.into_iter().for_each(|f| println!("{:?}", f));

        let dirs = std::fs::read_dir("src").expect("read_dir on src");
        dirs.into_iter().for_each(|f| println!("{:?}", f));

        // Following does not work
        // let dirs = std::fs::read_dir("/").expect("read_dir on /");
        // dirs.into_iter().for_each(|f| println!("{:?}", f));

        Ok(())
    }
}

bindings::export!(Component with_types_in bindings);

and running it using wasmtime run --dir src --dir wit target/wasm32-wasi/debug/print_preopened.wasm

reading src or /src works. but reading / returns following error:

read_dir on /: Custom { kind: Uncategorized, error: "failed to find a pre-opened file descriptor through which \"/\" could be opened" }

[lann]

For interop with existing code I think its best for wasi's filesystem to always have a "conventional" full tree structure; I know I have personally written plenty of code that assumes that is always the case.

I think my preferred behavior would be for Spin to always present files as part of a "full tree", which makes direct mounts the odd duck here.