zkevm-proverjs buildsetup returns immediately without generating the verifier contract
antonis19 opened this issue · 10 comments
I am trying to reproduce the Mainnet verifier contract using the guide here:
https://github.com/0xPolygonHermez/zkevm-contracts/blob/main/verifyMainnetDeployment/verifyMainnetProofVerifier.md
For this I have created a simple Docker container to perform the steps outlined in the doc. However, I think there might be a bug with the following step:
tmux -c "npm run buildsetup --bctree=../zkevm-prover/build/bctree"
as it immediately returns with the following output:
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildsetup
> tools/build_all.sh ${npm_package_config_steps_setup}
Using 13565 MB
The document says this step should take a few hours since it involves downloading the power of tau file (288 GB) , however this is not happening for me as the command returns immediately instead.
This is my Dockerfile ( zkevm-prover and zkevm-proverjs are checked out locally at the appropriate commits, and copied to the image) :
# Use Ubuntu 22.04 LTS as the base image
FROM --platform=linux/amd64 ubuntu:22.04
WORKDIR /root
# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive
ENV NODE_OPTIONS="--max-old-space-size=230000"
# Basic OS preparation
RUN apt-get update && \
apt-get install -y git tmux git curl jq && \
tee -a /etc/sysctl.conf && \
sysctl -w vm.max_map_count=655300
RUN apt install -y build-essential libomp-dev libgmp-dev nlohmann-json3-dev libpqxx-dev nasm libgrpc++-dev libprotobuf-dev grpc-proto libsodium-dev uuid-dev libsecp256k1-dev vim
# Install Node.js version 18
RUN curl -sL https://deb.nodesource.com/setup_18.x -o nodesource_setup.sh && \
bash nodesource_setup.sh && \
apt-get install -y nodejs
# Verify Node.js version
RUN node -v
# Download and prepare circom
RUN cd ~ && \
git clone https://github.com/iden3/circom.git && \
cd circom && \
git checkout v2.1.8 && \
git log --pretty=format:'%H' -n 1
# Install and compile circom (RUST)
RUN cd ~ && \
apt install -y cargo && \
cd circom && \
cargo build --release && \
cargo install --path circom
ENV PATH="$PATH:~/.cargo/bin"
## Prepare fast build constant tree tool
COPY ./zkevm-prover /root/zkevm-prover
RUN cd ~ && \
cd zkevm-prover && \
make -j bctree
# git checkout 40cde45deacede2b10a91ce2dd926abd2ba67541 && \
# git submodule init && \
# git submodule update && \
## Prepare and launch setup (zkevm-proverjs)
COPY ./zkevm-proverjs /root/zkevm-proverjs
RUN cd zkevm-proverjs && \
npm install
ENTRYPOINT ["/bin/bash", "-c", "cd zkevm-proverjs && tmux -c 'npm run buildsetup --bctree=../zkevm-prover/build/bctree'"]The problem is with npm version. When the document was written, version 18.19 was installed, but now it installs version 18.20 and newest version of npm that not supports arrays as string on package.json config.
You could replace:
# Install Node.js version 18
RUN curl -sL https://deb.nodesource.com/setup_18.x -o nodesource_setup.sh && \
bash nodesource_setup.sh && \
apt-get install -y nodejs
for this:
# Install Node.js version 18
ENV NVM_DIR=/usr/local/nvm
ENV NODE_VERSION=18.19.1
RUN mkdir -p $NVM_DIR \
&& curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash \
&& . $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
ENV NODE_PATH=$NVM_DIR/versions/node/v$NODE_VERSION/lib/node_modules
ENV PATH=$NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH
@zkronos73 Thanks for the update! I managed to get further along, and now I am getting an Illegal instruction error on the buildconstanttree command. This is my log:
cd zkevm-proverjs && tmux -c 'npm run buildsetup --bctree=../zkevm-prover/build/bctree'
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildsetup
> tools/build_all.sh ${npm_package_config_steps_setup}
Using 103662 MB
####### buildrom #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildrom
> . ./pre.sh && ROM_DIR=node_modules/@0xpolygonhermez/zkevm-rom && (cd $ROM_DIR && npm run build) && cp $ROM_DIR/build/rom.json $ROM_DIR/build/metadata-rom.txt $BDIR/
Using 103662 MB
> @0xpolygonhermez/zkevm-rom@6.0.0 build
> mkdir -p build && npx zkasm main/main.zkasm -o build/rom.json && npm run build:metadata
GLOBAL memory: 2663 4.06%
LOCAL memory: 59
> @0xpolygonhermez/zkevm-rom@6.0.0 build:metadata
> echo "COMMIT=$(git rev-parse HEAD)
> SHA256SUM=$(sha256sum ./build/rom.json)" > build/metadata-rom.txt
buildrom ...[OK] 0:00:04 / 0:00:05
####### buildpil #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildpil
> . ./pre.sh && mkdir -p $BDIR/pil/zkevm && node $NODE node_modules/.bin/pilcom $PIL -o $BDIR/pil/zkevm/$PIL_JSON -c $BDIR/pols_generated -n fork_1 && for PF in `tools/included_pils.sh $BDIR/pil/zkevm/$PIL_JSON`; do cp -v $PIL_DIR/$PF $BDIR/pil/zkevm/$PF; done
Using 103662 MB
Input Pol Commitmets: 751
Q Pol Commitmets: 403
Constant Pols: 234
Im Pols: 605
plookupIdentities: 34
permutationIdentities: 21
connectionIdentities: 4
polIdentities: 772
'pil/arith.pil' -> 'build/proof/pil/zkevm/arith.pil'
'pil/binary.pil' -> 'build/proof/pil/zkevm/binary.pil'
'pil/bits2field.pil' -> 'build/proof/pil/zkevm/bits2field.pil'
'pil/bits2field_sha256.pil' -> 'build/proof/pil/zkevm/bits2field_sha256.pil'
'pil/climb_key.pil' -> 'build/proof/pil/zkevm/climb_key.pil'
'pil/keccakf.pil' -> 'build/proof/pil/zkevm/keccakf.pil'
'pil/main.pil' -> 'build/proof/pil/zkevm/main.pil'
'pil/mem.pil' -> 'build/proof/pil/zkevm/mem.pil'
'pil/mem_align.pil' -> 'build/proof/pil/zkevm/mem_align.pil'
'pil/padding_kk.pil' -> 'build/proof/pil/zkevm/padding_kk.pil'
'pil/padding_kkbit.pil' -> 'build/proof/pil/zkevm/padding_kkbit.pil'
'pil/padding_pg.pil' -> 'build/proof/pil/zkevm/padding_pg.pil'
'pil/padding_sha256.pil' -> 'build/proof/pil/zkevm/padding_sha256.pil'
'pil/padding_sha256bit.pil' -> 'build/proof/pil/zkevm/padding_sha256bit.pil'
'pil/poseidong.pil' -> 'build/proof/pil/zkevm/poseidong.pil'
'pil/sha256f.pil' -> 'build/proof/pil/zkevm/sha256f.pil'
'pil/storage.pil' -> 'build/proof/pil/zkevm/storage.pil'
buildpil ...[OK] 0:00:02 / 0:00:10
####### buildstoragerom #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildstoragerom
> . ./pre.sh && ROM_DIR=node_modules/@0xpolygonhermez/zkevm-storage-rom && (cd $ROM_DIR && npm run build:rom) && cp $ROM_DIR/build/storage_sm_rom.json src/sm/sm_storage && cp src/sm/sm_storage/storage_sm_rom.json $BDIR/
Using 103662 MB
> @hermeznetwork/zkasm@4.0.0 build:rom
> node src/zkasmstorage.js zkasm/storage_sm.zkasm -o build/storage_sm_rom.json
buildstoragerom ...[OK] 0:00:02 / 0:00:14
####### buildconstants #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildconstants
> . ./pre.sh && node $NODE src/main_buildconstants -p $PIL -r $BDIR/rom.json -o $BDIR/zkevm.const
Using 103662 MB
compile PIL pil/main.pil
N = 8388608
Arith...
Binary...
Global...
KeccakF...
keccak build constants done
MemAlign...
Bits2Field...
PaddingKK...
PaddingKKBit...
PaddingPG...
PoseidonG...
Rom...
Storage...
StorageRom Done
PaddingSha256...
PaddingSha256Bit...
Bits2FieldSha256...
Sha256F...
ClimbKey...
Constants generated succefully!
buildconstants ...[OK] 0:09:28 / 0:09:44
####### buildstarkinfo #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 prebuildstarkinfo
> . ./pre.sh && if [ "$npm_config_starkstruct" = "debug" ]; then node tools/gen_debug_starkstruct.js -t GL -p $PIL -s $BDIR/zkevm.starkstruct.json; else cp src/zkevm.starkstruct.json $BDIR; fi
Using 103662 MB
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildstarkinfo
> . ./pre.sh && $PILSTARK/main_genstarkinfo.js -p $PIL -s $BDIR/zkevm.starkstruct.json -i $BDIR/zkevm.starkinfo.json
Using 103662 MB
maxDeg: 2, nIm: 539, d: 2
maxDeg: 3, nIm: 139, d: 3
--------------------- POLINOMIALS INFO ---------------------
Columns stage 1: 751 -> Columns in the basefield: 751
Columns stage 2: 68 -> Columns in the basefield: 168
Columns stage 3: 198 (139 intermediate polinomials) -> Columns in the basefield: 408
Columns stage 4: 2 -> Columns in the basefield: 6
Total Columns: 1019 -> Total Columns in the basefield: 1333
Total Constraints: 1029
Number of evaluations: 1973
------------------------------------------------------------
files Generated Correctly
buildstarkinfo ...[OK] 0:00:03 / 0:09:49
####### buildchelpers #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildchelpers
> . ./pre.sh && $PILSTARK/main_buildchelpers.js -m -p $PIL -s $BDIR/zkevm.starkinfo.json -c $BDIR/zkevm.chelpers/zkevm.chelpers.cpp -C ZkevmSteps -o
Using 103662 MB
39 2 39 2
step2prev_first
NOPS: 2094 2094
NARGS: 6804 6804
[343,31,3,25,6,1,0,0,23,0,1,0,322,0,0,0,0,0,0,0,0,1,4,0,0,0,0,8,0,0,1,0,1,0,0,3,0,4,0,0,0,0,0,4,8,4,20,3,0,0,115,0,0,98,170,1,4,1,0,50,0,9,0,0,3,0,0,0,0,0,310,0,0,0,0,0,0,0,41,211,3,1,158,0,0,0,3,25,38,8,4,0,0,2,0,0,5,0,0,1,17,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
Non used code: 76
113 4 113 4
step3prev_first
NOPS: 6673 6673
NARGS: 21766 21766
[399,94,6,25,11,2,0,2,23,0,1,1,1373,136,0,29,9,102,68,75,25,1,4,3,1,0,0,15,0,0,3,0,1,0,0,3,0,4,3,0,0,0,0,50,8,4,28,5,0,0,161,0,31,164,170,1,6,4,0,130,0,52,9,18,3,3,9,4,9,0,1393,42,0,50,0,0,25,0,296,830,74,59,172,145,4,3,3,25,38,8,46,0,0,2,0,0,5,0,76,1,18,3,0,21,0,4,0,2,0,0,0,4,0,9,24,0,0,0,0,1,2]
Non used code: 41
66 4 66 4
step3_first
NOPS: 14972 14972
NARGS: 55745 55745
[3529,76,53,25,10,4,12,1,23,0,1,0,612,26,0,5,3,13,43,10,10,2332,163,4,3,0,45,20,0,0,10,1,1,0,0,3,0,6,3,0,0,0,0,27,8,7,172,22,0,224,4015,2,32,1563,202,1,13,5,11,64,0,29,0,3,3,3,8,4,8,0,568,21,0,10,0,0,0,0,158,361,58,44,42,20,4,3,27,25,11,8,23,0,1,4,13,15,6,0,21,1,8,3,0,6,0,4,0,2,0,0,0,4,0,3,24,0,0,0,0,1,0]
Non used code: 36
1196 175 1196 175
step42ns_first
step52ns_first
files Generated Correctly
buildchelpers ...[OK] 0:00:02 / 0:09:54
####### buildconstanttree #######
> @0xpolygonhermez/zkevm-proverjs@6.0.0 buildconstanttree
> . ./pre.sh && $BCTREE -c $BDIR/zkevm.const -p $PIL -s $BDIR/zkevm.starkstruct.json -t $BDIR/zkevm.consttree -v $BDIR/zkevm.verkey.json
Using 103662 MB
Illegal instruction
buildconstanttree ...[FAIL] 0:00:01 / 0:09:57
Could you confirm the hardware characteristics of the server where you are doing buildsetup?
@zkronos73 I think it was a memory issue since I was running it on a Macbook Pro with 128 GB RAM which may not have been sufficient.
So I tried it on a r6a.8xlarge aws instance, running Ubuntu 22.04 LTS , with 256GB RAM, and 512GB of SSD like described in the doc, and while the process went further, I encountered an error in the FFLONK setup stage this time. See the log below:
Using 232679 MB
[INFO] snarkJS: FFLONK SETUP STARTED
[INFO] snarkJS: > Reading PTau file
[INFO] snarkJS: > Getting curve from PTau settings
[INFO] snarkJS: > Reading r1cs file
[INFO] snarkJS: > Processing FFlonk constraints
[INFO] snarkJS: processing r1cs constraints 500000/7774507
[INFO] snarkJS: processing r1cs constraints 1000000/7774507
[INFO] snarkJS: processing r1cs constraints 1500000/7774507
[INFO] snarkJS: processing r1cs constraints 2000000/7774507
[INFO] snarkJS: processing r1cs constraints 2500000/7774507
[INFO] snarkJS: processing r1cs constraints 3000000/7774507
[INFO] snarkJS: processing r1cs constraints 3500000/7774507
[INFO] snarkJS: processing r1cs constraints 4000000/7774507
[INFO] snarkJS: processing r1cs constraints 4500000/7774507
[INFO] snarkJS: processing r1cs constraints 5000000/7774507
[INFO] snarkJS: processing r1cs constraints 5500000/7774507
[INFO] snarkJS: processing r1cs constraints 6000000/7774507
[INFO] snarkJS: processing r1cs constraints 6500000/7774507
[INFO] snarkJS: processing r1cs constraints 7000000/7774507
[INFO] snarkJS: processing r1cs constraints 7500000/7774507
[INFO] snarkJS: ----------------------------
[INFO] snarkJS: FFLONK SETUP SETTINGS
[INFO] snarkJS: Curve: bn128
[INFO] snarkJS: Circuit power: 24
[INFO] snarkJS: Domain size: 16777216
[INFO] snarkJS: Vars: 16498185
[INFO] snarkJS: Public vars: 1
[INFO] snarkJS: Constraints: 16509843
[INFO] snarkJS: Additions: 8735335
[INFO] snarkJS: ----------------------------
[INFO] snarkJS: > computing k1 and k2
[INFO] snarkJS: > computing w3
[INFO] snarkJS: > computing w4
[INFO] snarkJS: > computing w8
[INFO] snarkJS: > computing wr
[INFO] snarkJS: > Writing the zkey file
[INFO] snarkJS: ··· Writing Section 1. Zkey Header
[INFO] snarkJS: ··· Writing Section 3. Additions
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
fflonk_setup ...[FAIL] 1:00:29 / 1:18:29I tried running the script again in case this was just a one time unlucky occurrence, but it happens every time.
It's a typical error because there is not enough memory. We probably need more memory in the last versions. Let me check.
I'm using a server with 512GB of RAM. I finished the setup on this server by limiting the memory used to 232679MB, the same as you. I suppose the problem is that the remaining memory isn't enough for the OS. You have three options:
- Activate swap.
- Use a larger instance, such as r6a.12xlarge, with 384GiB of memory.
- Force that setup uses less memory to leave more memory for OS.
I tested the last option, adding the parameter --mem=200, and I could finish the setup well. The command is:
tmux -c "npm run buildsetup --bctree=../zkevm-prover/build/bctree --mem=200"
@zkronos73 even with a r6a.16xlarge aws instance that has 512GB of RAM I still get the same bad alloc error.
Forcing the set up to use less memory on this bigger instance also resulted in the same error.
How much swap should I allocate? I tried with 2GB but I presume that's not enough.
I have just finished the build setup on a specific rented google instance like the one you used to try to reproduce the same error that appears to you. It's a n2-highmem-64 instance with 64 vCPU and 512 GB of RAM, ubuntu 22.04 without docker.
As you can see, the buildsetup process was finished correctly.
I have followed the document step by step, with the exception that I already mentioned to force the use of version 18.19.1 of nodejs with nvm. I could not reproduce the error.
Check the configuration and the operating system, and run everything as it is in the document. Follow the instructions to install the specific version of the nodejs.
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
logout / login
nvm install 18.19.1
In this instance swap is disabled; I don't need to modify it.
MemTotal: 528338404 kB
MemFree: 161553448 kB
MemAvailable: 522263132 kB
Buffers: 113988 kB
Cached: 353791016 kB
SwapCached: 0 kB
Active: 984420 kB
Inactive: 353011440 kB
Active(anon): 96988 kB
Inactive(anon): 4172 kB
Active(file): 887432 kB
Inactive(file): 353007268 kB
Unevictable: 27616 kB
Mlocked: 27616 kB
SwapTotal: 0 kB
SwapFree: 0 kB
So I tried again on a fresh AWS instance with 512GB RAM, and this time the process completed. The generated final.fflonk.verifier.sol contract is exactly the same as the verifier contract on mainnet (https://etherscan.io/address/0x0775e11309d75aA6b0967917fB0213C5673eDf81) .
The sha256 is :
ad7faf985475359b115d73ba216e7f6feb9cb3181889e65f62e23904da40b33awhich is what was expected.
Ok, perfect.