Fetching a CID causes an infinite loop of delegated routing and block requests

[2color]

Problem

Since the 1.4.0 release which introduces the blockstore sessions, using verified fetch to fetch a CID leads to an infinite loop of delegated routing and block requests.

This can be observed in the following codepen: https://codepen.io/2color/pen/oNOyarL

[SgtPooki]

FYI: I've done some testing here, and I can absolutely reproduce this in the browser when sessions are enabled, but not when sessions are disabled.

However, this works fine in nodejs whether sessions are enabled or not.

Browser testing

Sessions enabled

... infinite loop

Sessions disabled

NodeJS testing

// test.js in ./packages/verified-fetch on this repo
import { verifiedFetch } from './dist/src/index.js'

const resp = await verifiedFetch('ipfs://baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa', { session: false })
const obj = await resp.json()

// eslint-disable-next-line no-console
console.log(JSON.stringify(obj, null, '\t'))

Sessions enabled

╰─ ✔ ❯ DEBUG="helia*,helia*:trace" node test.js                                                                                                         14.98   27.1G   63%   80%  ─╯
  helia:verified-fetch:trace created VerifiedFetch instance +0ms
  helia:verified-fetch fetch ipfs://baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa +0ms
  helia:verified-fetch:get-resolved-accept-header no explicit IPLD content-type requested, returning incoming accept header undefined +0ms
  helia:verified-fetch output type undefined +8ms
  helia:verified-fetch:trace finding handler for cid code "297" and output type "undefined" +10ms
  helia:verified-fetch:trace calling handler "handleJson" +0ms
  helia:verified-fetch:trace fetching baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa/ +0ms
  helia:trustless-gateway:session:baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa finding 1-5 new provider(s) for baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa +0ms
  helia:trustless-gateway:session:baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa found 0/5 new providers +11s
  helia:trustless-gateway:session:baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa session is ready +0ms
  helia:trustless-gateway:session:baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa found initial session peers for baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa +0ms
  helia:trustless-gateway:session:baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa fetching BLOCK for baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa from https://dag.w3s.link/ +1ms
  helia:trustless-gateway-block-broker:dag.w3s.link GET https://dag.w3s.link/ipfs/baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa?format=raw 200 +0ms
  helia:trustless-gateway:session:baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa:trace got block for baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa from https://dag.w3s.link/ +0ms
{
	"hello": "@helia/verified-fetch makes verified IPFS retrieval easy"
}

Sessions disabled

╰─ ✔ ❯ DEBUG="helia*,helia*:trace" node test.js
  helia:verified-fetch:trace created VerifiedFetch instance +0ms
  helia:verified-fetch fetch ipfs://baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa +0ms
  helia:verified-fetch:get-resolved-accept-header no explicit IPLD content-type requested, returning incoming accept header undefined +0ms
  helia:verified-fetch output type undefined +8ms
  helia:verified-fetch:trace finding handler for cid code "297" and output type "undefined" +9ms
  helia:verified-fetch:trace calling handler "handleJson" +0ms
  helia:verified-fetch:trace fetching baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa/ +0ms
  helia:trustless-gateway-block-broker getting block for baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa from https://dag.w3s.link/ +0ms
  helia:trustless-gateway-block-broker:dag.w3s.link GET https://dag.w3s.link/ipfs/baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa?format=raw 200 +0ms
  helia:trustless-gateway-block-broker:trace got block for baguqeeradnk3742vd3jxhurh22rgmlpcbzxvsy3vc5bzakwiktdeplwer6pa from https://dag.w3s.link/ +0ms
{
	"hello": "@helia/verified-fetch makes verified IPFS retrieval easy"
}

[SgtPooki]

I feel like this might be related to https://github.com/ipfs/helia/blob/be86ac7200264e1671a0091a683db83cac0865da/packages/utils/src/abstract-session.ts#L121-L124

Because I'm seeing a number of found new providers re-retrieving when fetching from the browser with sessions enabled

[SgtPooki]

we're getting the block from the trustless gateway but the session is unaware:

Also, there are only two providers being returned and queried, but it's repeatedly querying them, so the bloom filter, toEvictionKey or equals is not working properly.

[SgtPooki]

So, it gets the block, and then says no providers had the block. is validation function failing somehow?

[SgtPooki]

FYI that I cannot reproduce this outside of codepen in a browser.

see https://codepen.io/SgtPooki/pen/vYwOWbB?editors=0010 & ipfs-examples/helia-examples@f93596a

both are almost the exact same code, and I cannot reproduce the error in the non-codepen one. something is happening inside of codepen that I can't reproduce outside of it.

[2color]

I also haven't been able to reproduce the problem outside of Codepen. I also tried the reproduction branch you created and it it doesn't cause the request loop.

[2color]

My suspicion is that it's related to how verifiedFetch is loaded from the esm.sh CDN.

For example, when I run the example from the Helia Examples repo (https://github.com/ipfs-examples/
helia-examples/commit/7c6674bc68293f33afbf5ddcacefcc5eb1f24c8b) and use a dymic import from esm.sh, I get the following log:

Whereas using the local dependency works fine:

[2color]

I just tried using jsdelivr (https://cdn.jsdelivr.net/npm/@helia/verified-fetch@1.4.1/+esm) both in the helia-examples reproduction branch and in codepen and it solved the problem.

You can see it working in https://codepen.io/2color/pen/oNOyarL

I wonder what it is about esm.sh that causes these bugs.

[2color]

Ok, I found a solution that also solves this with esm.sh. Add the ?bundle-deps query option which bundles all dependencies into a single js file.

Also fixes the codepen.

[SgtPooki]

awesome.. thanks for posting the fix, this will likely bite us again in the future.