SourceScan integration

Question

SourceScan integration

Canvinus opened this issue a year ago · 2 comments

For the integration purpose, I've established a preview API server that leverages test functionality not currently available in the production API. Further in this guide, I will outline all the necessary steps for cargo-near to successfully utilize SourceScan verification.

Preview API Server

Access the preview API server at https://test-api.sourcescan.dev.

Docker Image for Compilation

Use the Docker image sourcescan/cargo-near:0.6.0 for compilation. Details and the Dockerfile are available on GitHub at cargo-near-image.

It's essential that cargo-near employs this exact image when executing the cargo near build command. Containerizing the cargo-near deploy process is more complex, requires several workarounds. Specifically, it requires a method to securely pass the access keys to the container. This would enable the container to execute the deploy command by using the access keys located on the host machine.

`cargo near build` Command

The command for executing cargo near build {args} would work as:

docker run \
  --name {container-name} \
  --mount type=bind,source="$(pwd)",target=/host \
  --rm -it sourcescan/cargo-near:0.6.0 \
  bash -c "cd /host && cargo near build {args}"

This command mounts the current directory, executes the build command with the provided arguments, and ensures the container is removed after the process.

To avoid attempting to mount the container again, the environment variable CARGO_NEAR_NO_REPRODUCIBLE is introduced within the container. If this environment variable is detected, the standard cargo near build process should be executed.

Deploying Smart Contracts

After compilation, deploy the smart contract to the blockchain.

SourceScan API Integration

Step 1: Creating a temporary folder on remote server

Create a temporary folder with a POST request:

POST https://test-api.sourcescan.dev/api/temp/github

{
  "repo": "https://github.com/SourceScan/verifier-contract",
  "sha": "a24feb56fdc581b18fd39eb781a6d34a6673879f"
}

GitHub public repository URL
Full SHA256 hash of the commit

Returns

STATUS 200:

{
  "accessToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2VQYXRoIjoiL3RtcC82MWM3YWI5NzRhMDAiLCJnaXRodWIiOnsib3duZXIiOiJuZWFyIiwicmVwbyI6ImNhcmdvLW5lYXItbmV3LXByb2plY3QtdGVtcGxhdGUiLCJzaGEiOiI5MzJhYTZkNTRiNTBkNGRjODYzMTE0MjYzOTU5ZDg1NWI5ODhkYWY4In0sImlhdCI6MTcwNzQ5MDMxMCwiZXhwIjoxNzA3NDkyMTEwfQ.imvgAmbUXnNp_8N57bSfbh9rB207SQaaI1HywjeDIkU", 
  "files": ["cargo-near-new-project-template/Cargo.toml"]
}

The accessToken serves as the JWT Bearer token required to access the temporary folder, which contains the specific commit, this token is available for 10 minutes, after that the token is revoked and the temp folder is deleted. Also, it returns files such as Cargo.toml, which may act as entry points for the project compilation.

Step 2: Verifying the Smart Contract

Send a POST request for verification:

POST https://test-api.sourcescan.dev/api/verify/rust
Authorization: Bearer {JWT token}

{
  "entryPoint": "cargo-near-new-project-template/Cargo.toml",
  "networkId": "mainnet",
  "accountId": "sourcescan.near",
  "uploadToIpfs": true,
  "attributes": ["--no-abi"]
}

networkId: Either mainnet or testnet.
accountId: The identifier of the smart contract being verified.
uploadToIpfs: Boolean value indicating whether to upload to IPFS.
entryPoint: Selection from the entry points received in the first step.
attributes: An array containing all attributes called during cargo near build compilation. If no attributes were specified during compilation, send an empty array [].

Returns

STATUS 200 with message in body:
- "Contract verified successfully" - Successfully verified the contract
- "Code hash didn't change" - Code hash didn't change since last verification
- "Code hash mismatch" - discrepancy between the on-chain code and the compiled version
STATUS 500: Compilation error with detailed information.

After the successful execution, the temp folder is deleted, so sending the request once again would return an error.

Verification Contracts

Mainnet: dev.sourcescan.near
Testnet: dev.sourcescan.testnet

Verification results are accessible on BOS apps:

Mainnet: https://near.social/sourcescan.near/widget/SourceScan
Testnet: https://test.near.social/sourcescan.testnet/widget/SourceScan

Compilation Errors

A STATUS 500 response includes error details to assist with troubleshooting.

{
  "status": 500,
  "message": "Error occurred during command \"docker exec contract-builder /bin/bash /app/scripts/compiler/rust.sh /tmp/63482cf5276a/cargo-near-new-project-template \" execution",
  "detail": {
	  "stderr":" • Checking the host environment...done\n • Collecting cargo project metadata...done\n • Generating ABI\n │ Compiling proc-macro2 v1.0.78\n │ Compiling unicode-ident v1.0.12\n │ Compiling autocfg v1.1.0\n │ Compiling libc v0.2.153\n │ Compiling quote v1.0.35\n │ Compiling syn v2.0.48\n │ Compiling cfg-if v1.0.0\n │ Compiling version_check v0.9.4\n │ Compiling once_cell v1.19.0\n │ Compiling syn v1.0.109\n │ Compiling pin-project-lite v0.2.13\n │ Compiling hashbrown v0.14.3\n │ Compiling equivalent v1.0.1\n │ Compiling serde v1.0.196\n │ Compiling serde_derive v1.0.196\n │ Compiling indexmap v2.2.2\n │ Compiling itoa v1.0.10\n │ Compiling smallvec v1.13.1\n │ Compiling typenum v1.17.0\n │ Compiling generic-array v0.14.7\n │ Compiling lock_api v0.4.11\n │ Compiling parking_lot_core v0.9.9\n │ Compiling log v0.4.20\n │ Compiling scopeguard v1.2.0\n │ Compiling bytes v1.5.0\n │ Compiling futures-core v0.3.30\n │ Compiling tracing-core v0.1.32\n │ Compiling fnv v1.0.7\n │ Compiling parking_lot v0.12.1\n │ Compiling tokio-macros v2.2.0\n │ Compiling tracing-attributes v0.1.27\n │ Compiling num_cpus v1.16.0\n │ Compiling mio v0.8.10\n │ Compiling signal-hook-registry v1.4.1\n │ Compiling socket2 v0.5.5\n │ Compiling futures-sink v0.3.30\n │ Compiling tracing v0.1.40\n │ Compiling tokio v1.36.0\n │ Compiling proc-macro-error-attr v1.0.4\n │ Compiling getrandom v0.2.12\n │ Compiling slab v0.4.9\n │ Compiling anyhow v1.0.79\n │ Compiling ppv-lite86 v0.2.17\n │ Compiling rand_core v0.6.4\n │ Compiling proc-macro-error v1.0.4\n │ Compiling either v1.9.0\n │ Compiling memchr v2.7.1\n │ error: could not write output to /tmp/63482cf5276a/cargo-near-new-project-template/target/debug/deps/memchr-e8ff810912b191b7.memchr.b4e9aa3b105be35f-cgu.3.rcgu.o: No such file or directory\n │ \n │ error: could not compile `memchr` (lib) due to 1 previous error\n │ warning: build failed, waiting for other jobs to finish...\n │ error: couldn't create a temp dir: No such file or directory (os error 2) at path \"/tmp/63482cf5276a/cargo-near-new-project-template/target/debug/deps/rmeta5OEO0h\"\n │ \n │ error: could not compile `tokio` (lib) due to 1 previous error\nHere is the console command if you ever need to re-run it again:\ncargo near build\n\nError: \n 0: �[91m`cd \"/tmp/63482cf5276a/cargo-near-new-project-template\" && CARGO_PROFILE_DEV_DEBUG=\"0\" CARGO_PROFILE_DEV_LTO=\"off\" CARGO_PROFILE_DEV_OPT_LEVEL=\"0\" RUSTFLAGS=\"-C link-arg=-s -Awarnings\" \"/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo\" \"build\" \"--message-format=json-render-diagnostics\" \"--features\" \"near-sdk/__abi-generate\" \"--color\" \"auto\"` failed with exit code: Some(101)�[0m\n\nLocation:\n �[35mcargo-near/src/util/mod.rs�[0m:�[35m127�[0m\n\nBacktrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.\nRun with RUST_BACKTRACE=full to include source snippets.\n",
	  "stdout": ""
  }
}

(Optional) Debugging API Endpoint

POST https://test-api.sourcescan.dev/api/compile/rust
Authorization: Bearer {JWT token}

{
  "entryPoint": "Cargo.toml",
  "attributes": ["--no-abi"]
}

Returns

STATUS 200:

{
    "wasmBase64": "...",
    "checksum": "CgWXgxXnpfmZ7kEnBEsQvk4Dbj3DWMMD7UJm8ij37egF"
}

WASM base64 encoded string and checksum which is base58 encoded sha256 of WASM

STATUS 500: Compilation error with detailed information.

This endpoint could be used for debugging purposes.

(Optional) Contract Metadata

To facilitate future implementations, it is recommended to save the required information in the contract metadata. Currently, SourceScan's backend cannot work with contract metadata, but storing this information would be beneficial for future use. The necessary information includes:

{
  "repo": "https://github.com/SourceScan/verifier-contract",
  "sha": "a24feb56fdc581b18fd39eb781a6d34a6673879f",
  "entryPoint": "Cargo.toml",
  "attributes": []
}

This includes the repository URL, the SHA of the commit, the entry point (Cargo.toml), and any compilation attributes used during the cargo near build execution.

Answer 1 · 2024-02-11T12:45:38.000Z

@Canvinus Thanks for the details about SourceScan. I hope we can adjust SourceScan to use contract metadata, so it can handle contracts verification passively.

It is not unique setup, so let me provide an example from Rust world; here is how crates publishing works in Rust at the high level:

cargo publish (we can view it as cargo near deploy) builds and tests the crate locally and pushes the snapshot of code to crates.io (view it as NEAR blockchain where we publish Wasm files instead)
docs.rs (view it as SourceScan service) picks up recently published crates, and starts building them with cargo docs and using Cargo.toml as a reference (view it as SourceScan checks contract metadata and builds the contract and verify reproducibility)
docs.rs serves the docs from their website: https://docs.rs/near-sdk (SourceScan should just push the status to the smart contract with the verification status)

In such case, anyone can run their own SourceScan and also verify contracts.

So currently, it is a question of what details needs to be included in the contract metadata to allow SourceScan to reproduce the build without any other inputs. The current Contract Metadata is quite limited:

struct ContractMetadata {
    version: Option<String>,
    link: Option<String>,
    standards: Vec<Standard>,
}

struct Standard {
    standard: String,
    version: String,
}

https://github.com/near/near-sdk-rs/blob/e78e06adabb532f6ca29aab58cf22beba48be01f/near-sdk-macros/src/core_impl/contract_metadata/mod.rs#L11-L22

For reproducibility we need:

source code snapshot (archive) or at least a reference (commit hash)
frozen environment (docker image or at least a reference)
build command with all the flags that were used for compilation (anything that can influence the result - whether ABI was embedded, whether it is release or debug build, etc - keep in mind that cargo-near is not the only tool that is going to be out there to build contracts - near-sdk-js has its own process)

These are currently missing in the ContractMetadata and I would suggest to add all these details into a new section (build):

struct ContractMetadata {
    version: Option<String>,
    link: Option<String>,
    standards: Vec<Standard>,
    build_details: Option<BuildDetails>,
}

struct BuildDetails {
    /// The exact link to the contract source code, e.g. git, archive on IPFS, etc.
    ///
    /// Examples:
    /// * `"git+https://github.com/near-DevHub/neardevhub-contract.git#335e89edec95d56a4744e7160c3fd590e41ec38e"`
    /// * `"ipfs://<ipfs-hash>"`
    source_code_snapshot_link: Option<Url>,

    /// Reference to a reproducible build environment, e.g. Docker image reference:
    /// "docker.io/sourcescan/cargo-near:0.6.0"
    build_environment_ref: Option<String>,

    /// Contract folder within the source code snapshot.
    /// Often, it is the root of the repository, so can be omitted, but in case it is a monorepo, this is the way to specify the path to the contract folder.
    contract_work_dir: Option<PathBuf>,
    
    /// The exact command that was used to build the contract, with all the flags that could affect the final result.
    build_command: ["cargo", "near", "build"]
}

Factory contracts will be a challenge since they often try to embed a child contract Wasm file inside. We would need to ensure that factory contracts have build.rs script that will build all its child contracts.

Let's create a NEP extension to NEP-330 and start implementing it into near-sdk-rs, cargo-near, and SourceScan.

Answer 2 · 2024-04-04T18:13:30.000Z

Docker Integration and Permission Enhancements

Docker Engine Availability Check

Check if Docker is installed before performing operations requiring Docker.

pub fn check_docker_installed() -> color_eyre::eyre::Result<bool> {
    use std::process::Command;
    let output = Command::new("docker").arg("--version").output();
    output.map(|o| o.status.success()).unwrap_or(false)
}

User Notification: "Docker is not detected on your system. Please install Docker by following the guide at https://docs.docker.com/engine/install/."

Permission Error Handling for Docker Commands

Handle permission errors by suggesting the user add themselves to the Docker group or use sudo.

pub fn docker_run(args: BuildCommand) -> color_eyre::eyre::Result<()> {
    let output = Command::new("docker").args(["run", ...]).output();
    if let Err(e) = output {
        if e.kind() == std::io::ErrorKind::PermissionDenied {
            println!("Permission denied for Docker commands. Try adding your user to the Docker group with: sudo usermod -aG docker $USER. Alternatively, run the command with `sudo` to elevate privileges. Then, log out and log back in for the changes to take effect.");
        }
    }
    Ok(())
}

Suggestion Message: "Permission denied for Docker commands. Try adding your user to the Docker group with: sudo usermod -aG docker $USER. Alternatively, run the command with sudo to elevate privileges. Then, log out and log back in for the changes to take effect."

Check Docker Image Availability

Ensure required Docker image is available locally, or attempt to pull it from Docker Hub.

pub fn check_docker_image_available(image_name: &str) -> color_eyre::eyre::Result<bool> {
    let output = Command::new("docker").args(["image", "inspect", image_name]).output();
    if let Ok(o) = output {
        Ok(o.status.success())
    } else {
        println!("Attempting to pull the latest version of the image...");
        let pull_output = Command::new("docker").args(["pull", image_name]).output();
        pull_output.map(|p| p.status.success()).unwrap_or(false)
    }
}

Image Check Message: "The required Docker image is not available. Attempting to pull the latest version of the image..."

Alternative Approach: Stderr Parsing

As an additional method for error handling, parse stderr from Docker command executions to provide more detailed advice.

fn parse_docker_stderr(stderr: &str) -> Result<(), String> {
    if stderr.contains("permission denied") {
        Err("Permission denied. Try running the command with `sudo`, or add your user to the Docker group: `sudo usermod -aG docker $USER`. Then, log out and log back in for the changes to take effect.".into())
    } else if stderr.contains("Network timed out") || stderr.contains("server gave HTTP response") {
        Err("Network error detected. Please check your internet connection or try again later.".into())
    } else if stderr.contains("No such image") {
        Err("The required Docker image is not available locally. Please ensure the image name is correct or pull it from Docker Hub.".into())
    } else if stderr.contains("Cannot connect to the Docker daemon") {
        Err("Cannot connect to the Docker daemon. Is the docker daemon running?".into())
    } else {
        Err("An unspecified error occurred while executing the Docker command. Please check the Docker command and try again.".into())
    }
}

Preview API Server

Docker Image for Compilation

cargo near build Command

Deploying Smart Contracts

SourceScan API Integration

Step 1: Creating a temporary folder on remote server

Returns

Step 2: Verifying the Smart Contract

Returns

Verification Contracts

Compilation Errors

(Optional) Debugging API Endpoint

Returns

(Optional) Contract Metadata

Docker Integration and Permission Enhancements

Docker Engine Availability Check

Permission Error Handling for Docker Commands

Check Docker Image Availability

Alternative Approach: Stderr Parsing

`cargo near build` Command