Unreliable Docker build
glyn opened this issue · 3 comments
glyn commented
The command ./src/wrap_in_docker.sh ninjafailed to install Raku due to a collection of downloads which were not found:
...
Step 23/24 : RUN apt-get install -y --no-install-recommends rakudo perl6-zef
---> Running in c291994bfd9b
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
fonts-glyphicons-halflings libgraph-perl libipc-system-simple-perl
libjs-angularjs libjs-bootstrap libpath-tiny-perl libtommath1 moarvm nqp
Suggested packages:
valgrind
Recommended packages:
libunicode-utf8-perl
The following NEW packages will be installed:
fonts-glyphicons-halflings libgraph-perl libipc-system-simple-perl
libjs-angularjs libjs-bootstrap libpath-tiny-perl libtommath1 moarvm nqp
perl6-zef rakudo
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 7080 kB of archives.
After this operation, 50.5 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy/universe amd64 fonts-glyphicons-halflings all 1.009~3.4.1+dfsg-1 [117 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy/universe amd64 libgraph-perl all 1:0.9704-1 [109 kB]
Get:3 http://archive.ubuntu.com/ubuntu groovy/main amd64 libipc-system-simple-perl all 1.30-1 [23.2 kB]
Get:4 http://archive.ubuntu.com/ubuntu groovy/universe amd64 libjs-angularjs all 1.8.0-1 [552 kB]
Get:5 http://archive.ubuntu.com/ubuntu groovy/universe amd64 libjs-bootstrap all 3.4.1+dfsg-1 [124 kB]
Get:6 http://archive.ubuntu.com/ubuntu groovy/main amd64 libpath-tiny-perl all 0.114-1 [42.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu groovy/main amd64 libtommath1 amd64 1.2.0-3 [53.0 kB]
Err:8 http://archive.ubuntu.com/ubuntu groovy/universe amd64 moarvm amd64 2020.05+dfsg-1
404 Not Found [IP: 91.189.88.142 80]
Err:9 http://archive.ubuntu.com/ubuntu groovy/universe amd64 nqp amd64 2020.05+dfsg-1
404 Not Found [IP: 91.189.88.142 80]
Err:10 http://archive.ubuntu.com/ubuntu groovy/universe amd64 rakudo amd64 2020.05.1-1
404 Not Found [IP: 91.189.88.142 80]
Err:11 http://archive.ubuntu.com/ubuntu groovy/universe amd64 perl6-zef all 0.8.4-3
404 Not Found [IP: 91.189.88.142 80]
Fetched 1020 kB in 0s (2130 kB/s)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/moarvm/moarvm_2020.05+dfsg-1_amd64.deb 404 Not Found [IP: 91.189.88.142 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/n/nqp/nqp_2020.05+dfsg-1_amd64.deb 404 Not Found [IP: 91.189.88.142 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/r/rakudo/rakudo_2020.05.1-1_amd64.deb 404 Not Found [IP: 91.189.88.142 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/p/perl6-zef/perl6-zef_0.8.4-3_all.deb 404 Not Found [IP: 91.189.88.142 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get install -y --no-install-recommends rakudo perl6-zef' returned a non-zero code: 100
I guess the Docker cache was out of date, so I edited ./src/wrap_in_docker.sh as follows:
...
docker build --no-cache -t "$target_image" "$script_dir"
...
(I don't want to check this in because it will increase everyone's build times, but maybe we need to say something in the README. I guess the problem could occur on any install where downloads have been removed.)
I then got another failure:
[87/10474] (cd implementations/Clojure_json-path && ./lein uberjar) && mv implementations/Clojure_json-path/target/uberjar/json-path-comparison-0.1.0-SNAPSHOT-standalone.jar implementations/Clojure_json-path/build/json-path-comparison.jar && rm -r implementations/Clojure_json-path/target
Downloading Leiningen to /root/.lein/self-installs/leiningen-2.9.1-standalone.jar now...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 650 100 650 0 0 1975 0 --:--:-- --:--:-- --:--:-- 1975
100 13.9M 100 13.9M 0 0 1134k 0 0:00:12 0:00:12 --:--:-- 1437k
Retrieving org/clojure/clojure/1.10.0/clojure-1.10.0.pom from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.pom from central
Retrieving org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.pom from central
Retrieving cheshire/cheshire/5.8.1/cheshire-5.8.1.pom from clojars
Retrieving com/fasterxml/jackson/core/jackson-core/2.9.6/jackson-core-2.9.6.pom from central
Retrieving com/fasterxml/jackson/jackson-base/2.9.6/jackson-base-2.9.6.pom from central
Retrieving com/fasterxml/jackson/jackson-bom/2.9.6/jackson-bom-2.9.6.pom from central
Retrieving com/fasterxml/jackson/jackson-parent/2.9.1.1/jackson-parent-2.9.1.1.pom from central
Retrieving com/fasterxml/oss-parent/33/oss-parent-33.pom from central
Retrieving com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.9.6/jackson-dataformat-smile-2.9.6.pom from central
Retrieving com/fasterxml/jackson/dataformat/jackson-dataformats-binary/2.9.6/jackson-dataformats-binary-2.9.6.pom from central
Retrieving com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.9.6/jackson-dataformat-cbor-2.9.6.pom from central
Retrieving tigris/tigris/0.1.1/tigris-0.1.1.pom from clojars
Retrieving org/clojure/clojure/1.5.1/clojure-1.5.1.pom from central
Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom from central
Retrieving json-path/json-path/2.1.0/json-path-2.1.0.pom from clojars
Retrieving org/clojure/clojure/1.10.1/clojure-1.10.1.pom from central
Retrieving org/clojure/clojure/1.10.0/clojure-1.10.0.jar from central
Retrieving com/fasterxml/jackson/core/jackson-core/2.9.6/jackson-core-2.9.6.jar from central
Retrieving org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar from central
Retrieving org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar from central
Retrieving com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.9.6/jackson-dataformat-cbor-2.9.6.jar from central
Retrieving com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.9.6/jackson-dataformat-smile-2.9.6.jar from central
Retrieving tigris/tigris/0.1.1/tigris-0.1.1.jar from clojars
Retrieving cheshire/cheshire/5.8.1/cheshire-5.8.1.jar from clojars
Retrieving json-path/json-path/2.1.0/json-path-2.1.0.jar from clojars
Compiling json-path-comparison.core
Created /json-path-comparison/implementations/Clojure_json-path/target/uberjar/json-path-comparison-0.1.0-SNAPSHOT.jar
Created /json-path-comparison/implementations/Clojure_json-path/target/uberjar/json-path-comparison-0.1.0-SNAPSHOT-standalone.jar
ninja: build stopped: subcommand failed.
So I removed the --no-cache option and tried again. I hit another failure and increased the Docker VM size to 10 GB to be on the safe side and tried again...
Next failure:
...
[9767/10292] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_implementation_report.sh build/results build/implementations_matching_majority build/consensus implementations/Python_jsonpath2 > regression_suite/Python_jsonpath2.yaml
FAILED: regression_suite/Python_jsonpath2.yaml
LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_implementation_report.sh build/results build/implementations_matching_majority build/consensus implementations/Python_jsonpath2 > regression_suite/Python_jsonpath2.yaml
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath2: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath2: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath2: No such file or directory
src/shared.sh: line 54: build/results/union_with_duplication_from_array/Python_jsonpath2: No such file or directory
[9768/10292] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_results_report.sh build/results build/implementations_matching_majority build/consensus queries/filter_expression_with_equals_number_with_fraction > build/markdown/results/filter_expression_with_equals_number_with_fraction.md
[9769/10292] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_results_report.sh build/results build/implementations_matching_majority build/consensus queries/bracket_notation_with_number_on_object > build/markdown/results/bracket_notation_with_number_on_object.md
[9770/10292] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_bug_reports.sh build/results build/implementations_matching_majority build/consensus implementations/Clojure_json-path > bug_reports/Clojure_json-path.md
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Clojure_json-path: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_object/Clojure_json-path: No such file or directory
ninja: build stopped: subcommand failed.
After several failures and having spent most of the day trying, I am running out of steam. The latest failure is:
[1157/1674] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_bug_reports.sh build/results build/implementations_matching_majority build/consensus implementations/Elixir_jaxon > bug_reports/Elixir_jaxon.md
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Elixir_jaxon: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_object/Elixir_jaxon: No such file or directory
ninja: build stopped: subcommand failed.
glyn commented
Ran it again with only 1 CPU assigned to the Docker VM, but it still failed, thus:
...
[1140/1674] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_implementation_report.sh build/results build/implementations_matching_majority build/consensus implementations/Python_jsonpath > regression_suite/Python_jsonpath.yaml
FAILED: regression_suite/Python_jsonpath.yaml
LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_implementation_report.sh build/results build/implementations_matching_majority build/consensus implementations/Python_jsonpath > regression_suite/Python_jsonpath.yaml
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath: No such file or directory
src/shared.sh: line 54: build/results/union_with_duplication_from_array/Python_jsonpath: No such file or directory
[1141/1674] LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_implementation_report.sh build/results build/implementations_matching_majority build/consensus implementations/Python_jsonpath-rw > regression_suite/Python_jsonpath-rw.yaml
FAILED: regression_suite/Python_jsonpath-rw.yaml
LANG=en_US.UTF-8 LC_ALL= LC_COLLATE=C ./src/compile_implementation_report.sh build/results build/implementations_matching_majority build/consensus implementations/Python_jsonpath-rw > regression_suite/Python_jsonpath-rw.yaml
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath-rw: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath-rw: No such file or directory
src/shared.sh: line 29: build/results/union_with_duplication_from_array/Python_jsonpath-rw: No such file or directory
src/shared.sh: line 54: build/results/union_with_duplication_from_array/Python_jsonpath-rw: No such file or directory
ninja: build stopped: subcommand failed.
cburgmer commented
Sorry to hear you had to step into this.
I've pushed two changes to address some of the issues as you suggested.
Some context:
-
Ubuntu failing on re-builds of existing layers is a problem I believe is inherent to Ubuntu on Docker. As suggested I added something to the README.
-
Sometimes a build fails due to timeouts, I made the general mechanism a bit more forgiving (doubling the timeout). That hopefully alleviates some of the pain. Also documented this in the README.
-
The issue that reporting steps (e.g ./src/compile_implementation_report.sh or ./src/compile_bug_reports.sh) fail because files are missing is interesting. Currently my only explanation is that Ninja might have considered those files as built, but for some reason they were skipped. One quick way for you to force a re-build is to remove the directory of the missing files (
rm -r build/results/union_with_duplication_from_array) as this forces Ninja to rebuild all of them.
I re-ran the Docker solution twice now (I usually use the native environment on OSX), and it works for me. I wonder whether I can try to simulate a different hardware setup somewhat to try to reproduce some of your issues.
glyn commented
Sorry to hear you had to step into this.
No problem. The fact that this projects regularly builds cleanly for anyone is quite a feat.
I've pushed two changes to address some of the issues as you suggested.
Thank you!
I re-ran the Docker solution twice now (I usually use the native environment on OSX), and it works for me. I wonder whether I can try to simulate a different hardware setup somewhat to try to reproduce some of your issues.
Not sure. I'm hosting Docker on OSX using Docker Desktop. Rebuilding with your changes (and 1 CPU and 6 GB of memory) was successful, thanks.