Unable to build with vendored source

[happybeing]

I need to build offline so am using cargo vendor in the verso repo to create vendored sources for offline builds. When building it gets a long way, but I keep hitting and working around problems and finally am stuck. I've since tried the same with servo/servo because the issues here seem to relate to servo, but I can't get that to build vendored either.

Below is my progress with a vendored build of verso.

vendoring verso

The reason I need vendored builds is that I am off-grid an the servo related repos cause remote git operations to fail. I've tried many things to get that to work which I won't go into here, but it is a dead end and I need a way to build offline. So I start by getting the sources to a cloud server, vendoring there and then transferring the results to my laptop using rsync.

On cloud server I clone the fork I made of verso yesterday:

mkdir ~/verso && cd ~/verso
git clone https://github.com/happybeing/safe-verso
cd safe-verso
cargo vendor ../vendor-safe-verso/ > ../vendor-safe-verso.config.toml

On Laptop:
Note: as verso depends on servo I've already followed the servo build instructions and set up the build system locally with ./mach bootstrap in a clone of servo, downloaded mozjs and set the environment variable as below:
export MOZJS_MIRROR=~/src/safe-browser/safe-servo/libmozjs-x86_64-unknown-linux-gnu.tar.gz

Now I get my safe-verso/ clone and the vendor-safe-verso/ directory to the laptop:
cd ~/src/safe-browser && rsync --progress -lHpErz user@cloudserver:/home/user/verso/ safe-verso-vendored/

I add the following to the vendored sources template config file ~/src/safe-browser/safe-verso-vendored/vendor-safe-verso.config.toml

[net]
retry = 3                   # network retries
git-fetch-with-cli = true   # use the `git` executable for git operations
offline = true              # do not access the network

Then add the contents of the modified template to ~/src/safe-browser/safe-verso-vendored/safe-verso/.cargo/config.toml so verso will build off-line using the vendored sources.

Building vendored

Using cd ~/src/safe-browser/safe-verso-vendored/safe-verso && cargo run it builds successfully (offline) until it gets to embedder_traits.

Fails to compile embedder_traits

The build fails at embedder_traits because it can't find safe-verso/resources in the vendored source directory:

Compiling embedder_traits v0.0.1 (https://github.com/servo/servo.git?rev=d1fd8d5#d1fd8d55)
  error: couldn't read /home/user/src/safe-browser/safe-verso-vendored/safe-verso/../vendor-safe-verso/embedder_traits/../../../resources/prefs.json: No such file or directory (os error 2)

I worked around this with a symbolic link to place safe-verso/resources where it is looking:
cd ~/src/safe-browser/ && ln -s ~/src/safe-browser/safe-verso-vendored/safe-verso/resources resources

After that embedder_traits builds and it continues until the next issue.

Servo build script: can't find WebIDL

First it can't find the servo/third_party/ modules, and so errors importing WebIDL:

error: failed to run custom build command for `script v0.0.1 (https://github.com/servo/servo.git?rev=d1fd8d5#d1fd8d55)`

Caused by:
  process didn't exit successfully: `/home/mrh/src/safe-browser/safe-verso-vendored/safe-verso/target/debug/build/script-c5baaf37c41530a2/build-script-build` (exit status: 1)
  --- stderr
  Traceback (most recent call last):
	File "/home/mrh/src/safe-browser/safe-verso-vendored/vendor-safe-verso/script/dom/bindings/codegen/run.py", line 123, in <module>
	  main()
	File "/home/mrh/src/safe-browser/safe-verso-vendored/vendor-safe-verso/script/dom/bindings/codegen/run.py", line 23, in main
	  import WebIDL
  ModuleNotFoundError: No module named 'WebIDL'

I fixed this by adding the location to the environment, but as I didn't find the servo repo in the vendored sources I used the location of a servo/ clone:
export PYTHONPATH=~/src/safe-browser/safe-servo/servo/third_party

That enabled it to import WebIDL but immediately hit another error:

error: failed to run custom build command for `script v0.0.1 (https://github.com/servo/servo.git?rev=d1fd8d5#d1fd8d55)`

Caused by:
  process didn't exit successfully: `/home/mrh/src/safe-browser/safe-verso-vendored/safe-verso/target/debug/build/script-c5baaf37c41530a2/build-script-build` (exit status: 1)
  --- stderr
  Traceback (most recent call last):
    File "/home/mrh/src/safe-browser/safe-verso-vendored/vendor-safe-verso/script/dom/bindings/codegen/run.py", line 123, in <module>
      main()
    File "/home/mrh/src/safe-browser/safe-verso-vendored/vendor-safe-verso/script/dom/bindings/codegen/run.py", line 24, in main
      from Configuration import Configuration
    File "/home/mrh/src/safe-browser/safe-verso-vendored/vendor-safe-verso/script/dom/bindings/codegen/Configuration.py", line 8, in <module>
      from WebIDL import IDLExternalInterface, IDLSequenceType, IDLWrapperType, WebIDLError
  ImportError: cannot import name 'IDLExternalInterface' from 'WebIDL' (unknown location)

FYI: I also tried setting SERVO_ROOT but it didn't help.

At this point I gave up with verso and tried building servo vendored but that is failing as well so I'm stuck without a way to build verso.

Platform and Versions (please complete the following information):
OS: Ubuntu 22.04.04 LTS
cargo 1.74.1 (ecb9851af 2023-10-18)
rustc 1.74.1 (a28077b28 2023-12-04)

Additional context
I'd like to evaluate verso (and servo) for use as a browser for a new p2p platform. verso looks like a promising place to start putting together an initial version of a prototype browser if only I can get it building.

The reason I need to build offline is that I am off-grid, and while cloning and updating works for most repos, my mobile broadband can't handle servo and its dependencies. I can do some things on a cloud server but need to work on a local laptop without cargo trying to update dependencies. Hence trying the vendored approach.

I began with the wry servo demo and had the same issues, but was told that it had moved to this project.

[wusyong]

I think this is an interesting case because I believe even Servo hasn't explored this yet.
WebIDL implementation from Servo is generated by python script during build script.
And there are also lots of other resource are located in different directories during compilation.
Did it work with simple cargo run on your cloud? This is one of the goals of this verso project, which is to simplify the DX.
Perhaps you can also opened an issue in servo so more people can look into it and see what are exactly required during vendored compilation.

[happybeing]

Thanks. I will open an issue on Servo when I have some time.

I couldn't test the cloud build because it is Ubuntu 20 and doesn't install all the dependencies, such as Python 3.

[sagudev]

This is indeed servo's problem that is tracked in servo/servo#31812.