Unable to unpack WIT files from deep subdir in the unified WASI archive

[ritalin]

The unified WASI archive at
"https://github.com/WebAssembly/WASI/archive/refs/tags/v0.2.9.tar.gz"
places each proposal under the "proposals" folder. For example, the wasi:filesystem proposal is located at:

proposals/filesystem/wit

However, when running wit-deps update with this archive, the tarball is downloaded, but the WIT files are never unpacked.

A minimal deps.toml to reproduce the problem:

filesystem = { url = "https://github.com/WebAssembly/WASI/archive/refs/tags/v0.2.9.tar.gz", subdir = "proposals/filesystem/wit" }

Looking at the source code (crates/wit-deps/src/lib.rs, untar function),
it seems that certain cases are not handled correctly in the unpacking logic.
The subdir path "proposals/filesystem/wit" is deeper than what the current code supports,
so the WIT files are silently skipped.

https://github.com/bytecodealliance/wit-deps/blob/main/crates/wit-deps/src/lib.rs#L218

Is this the intended behavior?

[rvolosatovs]

Hey @ritalin, I've just merged #342 to address that issue, and will release 0.6.0 shortly

[rvolosatovs]

I was able to make it work locally with v0.6.0 using deps.toml like this:

clocks = { url = "https://github.com/WebAssembly/WASI/archive/refs/tags/v0.2.9.tar.gz", prefix = "WASI-0.2.9/proposals/clocks" }
filesystem = { url = "https://github.com/WebAssembly/WASI/archive/refs/tags/v0.2.9.tar.gz", prefix = "WASI-0.2.9/proposals/filesystem" }
io = { url = "https://github.com/WebAssembly/WASI/archive/refs/tags/v0.2.9.tar.gz", prefix = "WASI-0.2.9/proposals/io" }

Note, that we have to explicitly specify all transitive dependencies now due to this changed structure upstream (previously proposals would contain wit/deps directory, from which we could acquire the transitive deps)

[ritalin]

@rvolosatovs Thank you for fixing this so quickly.
This resolves the issue I reported.

One follow-up question:
wasi:filesystem depends on wasi:clocks and wasi:io,
but at the moment all of these need to be listed explicitly in deps.toml
(i.e. transitive WIT dependencies are not expanded automatically).

Is this an intentional design choice,
or are there plans to support automatic transitive dependency resolution?

[rvolosatovs]

Previously, each WASI proposal would include the wit/deps directory - Wasm tooling expects this directory to contain all dependencies of the WIT package used. Consequently, wit-deps would pull in the (transitive) dependencies from wit/deps directory.

Since WebAssembly/WASI#836, that's not the case anymore and so wit-deps can no longer pull in transitive dependencies.

I think there are a few things that can be done here:

1. https://github.com/WebAssembly/WASI should publish release artifacts with "self-contained" packages for each proposal, this way users can just use a single dependency specification/URL to pull in everything required, e.g. something like this:

http = "https://github.com/WebAssembly/wasi/releases/download/v0.2.9/wasi-http.tar.gz"

The layout of wasi-http.tar.gz is something like:

deps/cli/*.wit
deps/clocks/*.wit
deps/filesystem/*.wit
deps/io/*.wit
deps/random/*.wit
deps/sockets/*.wit
handler.wit
proxy.wit
types.wit

So it includes only the wasi-http WIT including WIT of it's dependencies.

Note, that such release artifact could also be a single Wasm file, but we'd need to add support for Wasm to wit-deps as well. (Refs #25 #24)

@ricochet @yoshuawuyts - would you be able to set something like this up? Otherwise, I'm happy to contribute a GitHub Action workflow doing that

2. Similar to 1., I think that https://github.com/WebAssembly/WASI repository should be usable out-of-the-box just after cloning it. I.e. I think that proposal dependencies (deps directory) should be present in the repo. It being a monorepo, perhaps we just use symlinks for that?

3. Something we can do on wit-deps side is add a --recursive mode, which would resolve dependencies from deps.toml files in dependencies. E.g. in the case of wasi:filesystem coming directly from https://github.com/WebAssembly/WASI in the shape it is in today, wit-deps would look in the deps.toml of proposals/filesystem and pull in the dependencies specified.

This way, just this:

filesystem = { url = "https://github.com/WebAssembly/WASI/archive/refs/tags/v0.2.9.tar.gz", prefix = "WASI-0.2.9/proposals/filesystem" }

would be enough to pull in wasi:filesystem and all of it's dependencies