ipfs/ipfs-webui

bug: pinning fails during CI

SgtPooki opened this issue · 6 comments

SgtPooki commented

Describe the bug
Pinning is failing during CI actions: https://github.com/ipfs/ipfs-webui/actions/runs/6165506297/job/16733763437. We currently use both estuary and our own ipfs-cluster; both are failing.

We should investigate using fleek.co (since most of our applications are using fleek.co now) or web3.storage for pinning webui files instead.

Things that could reduce the complexity of CI by moving to...

Fleek

  • creating CID with contents of build/ -

    ipfs-webui/.github/workflows/ci.yml

    Lines 69 to 74 in 346c367

    - uses: ipfs/start-ipfs-daemon-action@v1
    - name: Import build/ to IPFS
    id: ipfs
    run: |
    root_cid=$(ipfs add --cid-version 1 -Q -r ./build)
    echo "cid=$root_cid" >> $GITHUB_OUTPUT
  • ? create car file -

    ipfs-webui/.github/workflows/ci.yml

    Lines 78 to 80 in 346c367

    - name: Create ipfs-webui.car file
    run: |
    ipfs dag export ${{ steps.ipfs.outputs.cid }} > ipfs-webui_${{ github.sha }}.car
  • pinning -

    ipfs-webui/.github/workflows/ci.yml

    Lines 88 to 125 in 346c367

    - name: Pin to estuary.tech
    id: pin-estuary
    continue-on-error: true
    run: |
    curl -X POST https://api.estuary.tech/pinning/pins -d '{ "name": "'$PIN_NAME'", "cid": "'$PIN_CID'" }' -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
    curl -X GET https://api.estuary.tech/pinning/pins -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
    env:
    ESTUARY_API_TOKEN: ${{ secrets.ESTUARY_API_TOKEN }}
    PIN_CID: ${{ steps.ipfs.outputs.cid }}
    PIN_NAME: "ipfs-webui@${{ github.sha }}"
    - name: Pin to ipfs-websites.collab.ipfscluster.io
    id: pin-cluster
    continue-on-error: true
    run: |
    ipfs-cluster-ctl --enc=json \
    --host "${CLUSTER_HOST}" \
    --basic-auth "$CLUSTER_USER:$CLUSTER_PASSWORD" \
    peers ls > cluster-peers-ls
    for maddr in $(jq -r '.ipfs.addresses[]?' cluster-peers-ls); do
    ipfs swarm peering add "$maddr" &
    ipfs swarm connect "$maddr" &
    done
    ipfs-cluster-ctl --enc=json \
    --host "${CLUSTER_HOST}" \
    --basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \
    pin add \
    --name "${PIN_NAME}" \
    --replication-min 1 \
    --replication-max 3 \
    --wait \
    $PIN_CID
    env:
    CLUSTER_HOST: "/dnsaddr/ipfs-websites.collab.ipfscluster.io"
    CLUSTER_USER: ${{ secrets.CLUSTER_USER }}
    CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }}
    PIN_CID: ${{ steps.ipfs.outputs.cid }}
    PIN_NAME: "ipfs-webui@${{ github.sha }}"
  • dnslink -

    ipfs-webui/.github/workflows/ci.yml

    Lines 134 to 144 in 346c367

    # dev dnslink is updated on each main branch update
    - run: npx dnslink-dnsimple --domain dev.webui.ipfs.io --link /ipfs/${{ steps.ipfs.outputs.cid }}
    if: github.ref == 'refs/heads/main'
    env:
    DNSIMPLE_TOKEN: ${{ secrets.DNSIMPLE_TOKEN }}
    # production dnslink is updated on release (during tag build)
    - run: npx dnslink-dnsimple --domain webui.ipfs.io --link /ipfs/${{ steps.ipfs.outputs.cid }}
    if: github.ref == 'refs/heads/main' && github.event_name == 'workflow_dispatch'
    env:
    DNSIMPLE_TOKEN: ${{ secrets.DNSIMPLE_TOKEN }}

web3.storage

  • pinning -

    ipfs-webui/.github/workflows/ci.yml

    Lines 88 to 125 in 346c367

    - name: Pin to estuary.tech
    id: pin-estuary
    continue-on-error: true
    run: |
    curl -X POST https://api.estuary.tech/pinning/pins -d '{ "name": "'$PIN_NAME'", "cid": "'$PIN_CID'" }' -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
    curl -X GET https://api.estuary.tech/pinning/pins -H "Content-Type: application/json" -H "Authorization: Bearer $ESTUARY_API_TOKEN"
    env:
    ESTUARY_API_TOKEN: ${{ secrets.ESTUARY_API_TOKEN }}
    PIN_CID: ${{ steps.ipfs.outputs.cid }}
    PIN_NAME: "ipfs-webui@${{ github.sha }}"
    - name: Pin to ipfs-websites.collab.ipfscluster.io
    id: pin-cluster
    continue-on-error: true
    run: |
    ipfs-cluster-ctl --enc=json \
    --host "${CLUSTER_HOST}" \
    --basic-auth "$CLUSTER_USER:$CLUSTER_PASSWORD" \
    peers ls > cluster-peers-ls
    for maddr in $(jq -r '.ipfs.addresses[]?' cluster-peers-ls); do
    ipfs swarm peering add "$maddr" &
    ipfs swarm connect "$maddr" &
    done
    ipfs-cluster-ctl --enc=json \
    --host "${CLUSTER_HOST}" \
    --basic-auth "${CLUSTER_USER}:${CLUSTER_PASSWORD}" \
    pin add \
    --name "${PIN_NAME}" \
    --replication-min 1 \
    --replication-max 3 \
    --wait \
    $PIN_CID
    env:
    CLUSTER_HOST: "/dnsaddr/ipfs-websites.collab.ipfscluster.io"
    CLUSTER_USER: ${{ secrets.CLUSTER_USER }}
    CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }}
    PIN_CID: ${{ steps.ipfs.outputs.cid }}
    PIN_NAME: "ipfs-webui@${{ github.sha }}"
SgtPooki commented

i'm investigating using w3cli for this, but running into some issues.. will need some more debugging and testing: https://filecoinproject.slack.com/archives/C027XP5BTGB/p1694813264220989

SgtPooki commented

we can potentially use fleek (even though pinning is currently behind closed alpha) via their site:deploy command and then getting the CID via dig or other: dig +noall +answer TXT \_dnslink.ipfs-webui.on.fleek.co

SgtPooki commented

pinning seems to be succeeding on our ipfs cluster now: https://github.com/ipfs/ipfs-webui/actions/runs/6150397438/job/16688431637

SgtPooki commented

Process for pinning with w3cli

The below steps are finalized and working in this repo now

Given the following env vars:

export W3CLI_SPACE_NAME="foobar123"
export W3CLI_SPACE_DELEGATION_PROOF_FILENAME="ipfs-webui-ci-space-delegation.proof"
export W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME="ipfs-webui-ci-space-delegation.proof.base64"
export W3_STORE_NAME="ipfs-webui-store"
export UCAN_PRINCIPAL_FOR_CI_FILENAME="w3cli-ci-principal.json"

on your computer

# install w3cli, exposes command `w3`
npm install -g @web3-storage/w3cli 

# verify email if you don't already have an account
w3 login <email> 

# creates a space and returns a DID we store into `W3CLI_SPACE_DID`
W3CLI_SPACE_DID=$(w3 space create $W3CLI_SPACE_NAME) 

# Create a private key and DID our CI agent will use. 
# See https://github.com/web3-storage/w3cli#w3_principal & the `const principal = Signer.parse(process.env.KEY)` line at https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation for more information
npx -y ucan-key ed --json > $UCAN_PRINCIPAL_FOR_CI_FILENAME

# this is a private key for your CI "agent". This needs uploaded to github as a SECRET.
W3_AGENT_PRINCIPAL=$(cat $UCAN_PRINCIPAL_FOR_CI_FILENAME | jq -r .key)

# this is the DID for your CI Agent
W3_AGENT_DID=$(cat $UCAN_PRINCIPAL_FOR_CI_FILENAME | jq -r .did)


# create a delegation that has permissions for the CI runner, and save delegation to a file
w3 delegation create $W3_AGENT_DID -n 'ipfs-webui-CI' --can 'store/*' --can 'upload/*' --output $W3CLI_SPACE_DELEGATION_PROOF_FILENAME 

# convert space delegation proof car to a base64 encoded string and save in a file with the name $W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME
# Note that on my machine, base64 is BSD and gbase64 is from GNU-tools, so I use gbase64
cat $W3CLI_SPACE_DELEGATION_PROOF_FILENAME | gbase64 -w0 > $W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME

on CI

Make sure you set your GitHub repo secrets:

Set CI SECRETS:

  • W3_AGENT_DID = the content of $W3_AGENT_DID from above
  • W3_AGENT_PRINCIPAL = the content of $W3_AGENT_PRINCIPAL from above
  • W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING = the result of $(cat $W3CLI_SPACE_DELEGATION_PROOF_BASE64_FILENAME)

When running your action, you will need to ensure you do:

    env:
      W3_STORE_NAME: ipfs-webui-ci
      W3_AGENT_DID: ${{ secrets.W3_AGENT_DID }}
      # set w3cli principal from https://github.com/web3-storage/w3cli#w3_principal
      W3_PRINCIPAL: ${{ secrets.W3_AGENT_PRINCIPAL }}
      W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING: ${{ secrets.W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING }}
### Non w3cli stuff:

# create car file with contents you want to upload, stored in `my-car.car`

### web3cli stuff

# auth/delegation with w3cli still unclear.. 
# some instructions at https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation

# ensure whoami shows we are who we think we are. This should be the same as $W3_AGENT_DID
npx -y --package=@web3-storage/w3cli@latest -- w3 whoami

# convert W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING to a file:
echo $W3CLI_SPACE_DELEGATION_PROOF_BASE64_STRING | base64 -d > $W3CLI_SPACE_DELEGATION_PROOF_FILENAME


# I'm not sure if I need this, or I just need to use `space use`
npx -y --package=@web3-storage/w3cli@latest -- w3 proof add $W3CLI_SPACE_DELEGATION_PROOF_FILENAME


# add that space
W3CLI_SPACE_DID=$(npx -y --package=@web3-storage/w3cli@latest -- w3 space add $W3CLI_SPACE_DELEGATION_PROOF_FILENAME)

# now ensure the w3cli is using that space
npx -y --package=@web3-storage/w3cli@latest -- w3 space use $W3CLI_SPACE_DID

# upload the car 
npx -y --package=@web3-storage/w3cli@latest -- w3 up --no-wrap -c my-car.car