Downloading SOLC behind proxy - HardhatError: HH502: Couldn't download compiler versions list. Please check your connection.
cdesch opened this issue · 24 comments
I cannot compile the contracts because the compiler goes out to https://solc-bin.ethereum.org/windows-amd64/list.json to fetch them. This is blocked by a proxy.
How do I...
- A) Configure HardHat to work with a proxy?
OR
- B) Manually install solc (with something like
yarn add solc) and disable HardHat from checking for a compiler?
Error:
HardhatError: HH502: Couldn't download compiler versions list. Please check your connection.
at CompilerDownloader.downloadCompilersList (C:\Projects\myProject\node_modules\hardhat\src\internal\solidity\compiler\downloader.ts:185:13)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Caused by: FetchError: request to https://solc-bin.ethereum.org/windows-amd64/list.json failed, reason: read ECONNRESET
at ClientRequest.<anonymous> (C:\Projects\myProject\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (events.js:315:20)
at TLSSocket.socketErrorListener (_http_client.js:426:9)
at TLSSocket.emit (events.js:315:20)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
I found the task name subtask(TASK_COMPILE_SOLIDITY_RUN_SOLCJS)
to compile using solcjs. Is there a way to configure solcjs as the default compiler? Possibly through an ENV variable?
@fvictorio Do you think that there is any way to use this check for the solidity compiler in PR#913 to skip the binary process entirely and just use the solcjs compiler?
how do I set this option _forceSolcJs? Can it be set in the config.hardhat.js
Hi @okwme! I made PR#1291 to add proxy support. @fvictorio is going to take a look at it this week.
Hope this feature can be added soon, thanks hardhat team.
Would it be easier if HardHat allowed the solc to be used in a Docker Container (solc docker containers)?
I don't like to compare HardHat with Truffle, but it is a nice feature in Truffle to specify that the solc can be used from the docker container:
module.exports = {
networks: {
development: {
host: "localhost",
port: 8545,
network_id: "*",
},
},
compilers: {
solc: {
version: "0.8.4",
docker: true,
},
},
};
@cdesch Thank you for pushing this pr that seems to be introduced with 2.3.3. Did this version finally work out for you?
Still facing this issue behind the corporate proxy (using Fiddler as local proxy to perform required NTLM authentication against the corporate proxy).
This was included in version 2.4.0. You can now set the HTTP_PROXY or HTTPS_PROXY variables and they will be used to download the solc binaries.
@martinoss it works with the HTTP_PROXY and HTTPS_PROXY, although it is still problematic in some cases where there intermediate steps or CI build steps that don't have internet access even with the proxies.
It would be nice if you could manually provide the solidity compiler solc binary and set the path in the hardhat.config.js so there isn't the default behavior of reaching out and pulling the list of compilers (https://binaries.soliditylang.org/linux-amd64/list.json) and then the compiler.
To get around this, I did some reverse engineering and provided the list.json and solc manually by side loading them into a docker image. Here is a sample, but the paths and params might change based on your NodeJS based docker image:
FROM my_nodejs_image
ENV HTTP_PROXY=http://myproxy.mycompany.com:8080
ENV HTTPS_PROXY=http://myproxy.mycompany.com:8080
ENV NO_PROXY=".local,mycompany.com"
WORKDIR /opt/app-root/src
COPY .npmrc /opt/app-root/src
COPY package.json /opt/app-root/src
COPY yarn.lock /opt/app-root/src
RUN yarn install
# Copy source code
COPY . /opt/app-root/src
# Manually Copy solidity compiler
COPY list.json /opt/app-root/src/.cache/hardhat-nodejs/compilers/linux-amd64/list.json
COPY solc-linux-amd64-v0.7.3+commit.9bfce1f6 /opt/app-root/src/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.7.3+commit.9bfce1f6
COPY solc-linux-amd64-v0.8.4+commit.c7e474f2 /opt/app-root/src/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2
RUN npx hardhat compile --verbose
CMD ["node", "/opt/app-root/src/server.js"]
Of course, you have to provide the right list.json for the right platform and the right binary for the solc version your using. Same with the paths.
I think on MacOS, the default path might be:
/Users/myusername/Library/Caches/hardhat-nodejs/compilers/macos-amd64/
and Windows might be something like:
C:\Users\myusername\AppData\Local\hardhat-nodejs\Cache\compilers\windows-amd64
Your mileage may vary. Good Luck.
@cdesch i'm curious about this. Can you describe more about this offline use case? Why don't you have internet access? Thanks!
Hi @alcuadrado. I wouldn't call it completely offline, but isolated or restricted mostly from many internet resources and offline at points in time.
For example, let's that you had a Continuous Integration and Continuous Delivery pipeline (like, Jenkins, GitHub CI or GitLab CI) on an internal platform like GitHub Enterprise or GitLab Enterprise. This is pretty normal for many businesses and can be setup a variety of ways on different projects. We now want to use those tools with a HH project to automatically build, lint, test, security audit, and deploy our Smart Contracts into dev/review environment and possibly promote them into production. To do this, each step might be broken down into smaller steps that run on docker containers and also deployed on docker containers to where-ever they may run.
In some or many instances, the deployment of docker containers that run apps assume that they have everything on everything on it to when running and may not need internet access. For example, if we had an ExpressJS app, the Dockerfile would install of the NodeJS/npm dependencies, copy the source code and build (or transpile for TypeScript) and produce a runnable image (kind of like this example from dockerhub). Internet access to NPM or an internal NPM registry was needed for the build of the docker image, but not running it. Applications with greater complexity probably have multiple build steps in place of just one Dockerfile.
If we took that ExpressJS app container image that we just made, we could then run it in something like kubernetes, AWS or on a internal server. The ExpressJS app this example has no expectation using any internet access because all of the resources were provided for it and it can just run. Locking down network access to the environment adds additional security in many cases and we know all it needs incoming network path to the port running Express.
Let's add HardHat to our ExpressJS App and let's say it does something. Everything works great until we run npx hardhat compile or some of the other npx hardhat * commands because HardHat wants to reach out to internet to get list.json and solc and then everything comes crashing down.
This isn't a knock against the way HardHat does this currently. I think it is awesome that HardHat sorts things out for you by automatically getting the right solidity compilers for your system! The complexity of solc compilers and solidity versions is a different issue completely and it is a great feature that hardhat can make the sometimes painful process of getting SmartContracts or solidity to work correctly much easier right out of the box. The drawback is that when we have to deviate from the pattern at all it makes things a bit more difficult.
For those who might say something like "What about Mainnet or Rinky? Don't you need internet access for that?". Forget about Mainnet and Rinky. You can run your own geth node or you can run Quorum, or HyperLedger Fabric or the AWS blockchain or something else. Just because you are working with SmartContracts and solidity doesn't mean they have to go on the Mainnet.
Does that help a bit?
Even since I posted the Dockerfile, I added:
COPY wasm_list.json /opt/app-root/src/.cache/hardhat-nodejs/compilers/wasm/list.json
COPY soljson-v0.8.4+commit.c7e474f2.js /opt/app-root/src/.cache/hardhat-nodejs/compilers/wasm/soljson-v0.8.4+commit.c7e474f2.js
Why? I dunno. It should be the same in the docker container but for some reason it wants wasm... idk.
Yes! This works! Thank you!
@cdesch thanks for your detailed response! What you described makes a lot of sense.
As @fvictorio pointed out, this is now handled by overriding a task. We may add a less involved way to achieve it in the future.
If you found your way to this issue because your @FleekHQ builds that rely on @nomiclabs Hardhat are flaky and fail 80% of the time because the build step says, it can't download the soldity compiler, you might be able to work around that by using a custom build image (Note, you can also commit solidity compilers to your repo and overwrite the hardhat subtask to enforce using it, like demonstrated here, but that'd add another 10-20 MBs to your repo).
Since I need pnpm for my builds I'm using my own build image on Fleek anyway. You can find elmariachi/node16-pnpm:v1.1 on docker hub. This is its the whole Dockerfile:
FROM node:16-buster
RUN mkdir -p /root/.cache/hardhat-nodejs/compilers/linux-amd64 \
&& wget -O /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.7+commit.e28d00a7 https://binaries.soliditylang.org/linux-amd64/solc-linux-amd64-v0.8.7+commit.e28d00a7
RUN npm i -g pnpmV1.1 of my image contains the 0.8.7 compiler at the path where hardhat's compiler loader is looking for it and hence doesn't need to download it and can't fail anymore \0/
That is similar to how I configured my docker builds. I had a base container image with the solc compilers added to it, then used the solution that @fvictorio provided in the above thread to add a subtask to the hardhat.config.js to specify the compiler.
I faced similar issue. However, for me hardhat is not able to download list of compiler versions behind corporate proxy. Tried a lot of things. No luck. After a lot of trial and errors, found a way to make this work. Download list.json
and keep it locally at C:\Users\<your-user>\AppData\Local\hardhat-nodejs\Cache\compilers\windows-amd64 (windows). Also, download the solidity compiler executable for windows (required version) from https://github.com/ethereum/solidity/releases and place it in the same folder mentioned above. Now, peruse your list.json and identify the object for the solc compiler version of interest. Rename the executable as with the long name given in path in that object. For e.g., if you need solc version 0.8.0, the path in list.json for 0.8.0 version is solc-windows-amd64-v0.8.4+commit.c7e474f2.exe. Use this to rename the downloaded executable. Now, run npm test. This should fix the issue.
My issue is resolved by simply adding my own proxy config in the "hardhat.config.js" file:
// set proxy
const { ProxyAgent, setGlobalDispatcher } = require("undici");
const proxyAgent = new ProxyAgent('http://127.0.0.1:7890'); // change to yours
setGlobalDispatcher(proxyAgent);My issue is resolved by simply adding my own proxy config in the "hardhat.config.js" file:
// set proxy const { ProxyAgent, setGlobalDispatcher } = require("undici"); const proxyAgent = new ProxyAgent('http://127.0.0.1:7890'); // change to yours setGlobalDispatcher(proxyAgent);
thank you
My issue is resolved by simply adding my own proxy config in the "hardhat.config.js" file:
// set proxy const { ProxyAgent, setGlobalDispatcher } = require("undici"); const proxyAgent = new ProxyAgent('http://127.0.0.1:7890'); // change to yours setGlobalDispatcher(proxyAgent);
thank you
My issue is resolved by simply adding my own proxy config in the "hardhat.config.js" file:
// set proxy const { ProxyAgent, setGlobalDispatcher } = require("undici"); const proxyAgent = new ProxyAgent('http://127.0.0.1:7890'); // change to yours setGlobalDispatcher(proxyAgent);
thx
Hi, I solved the problem by letting hardhat to cache locally.
Just in case you are using compose pluging and Docker.
Dockerfile.
FROM node:alpine as base
ENV PUID=1000
ENV PGID=1000
COPY . /usr/app/
RUN npm install --save-dev @nomiclabs/hardhat-waffle 'ethereum-waffle@^3.0.0' @nomiclabs/hardhat-ethers 'ethers@^5.0.0'
compose.yml
services:
app: &app
build:
context: .
environment:
dapk: ""
NODE_OPTIONS: "--openssl-legacy-provider"
ETHERSCAN_API_KEY: ""
test:
<<: *app
volumes:
- ./contract/contracts:/usr/app/contract/contracts
- ./contract/artifacts:/usr/app/contract/artifacts
- ./root/.cache/hardhat-nodejs/compilers/linux-amd64:/root/.cache/hardhat-nodejs/compilers/linux-amd64/
command: "npx hardhat test --verbose