Default block size too small for go-ipfs interop
rklaehn opened this issue · 8 comments
The maximum block size in go-ipfs is 4 mb, so it is possible to create a block in go-ipfs that can not be bitswapped.
Either we should increase the default block size to 4mb (exact size is somewhere in the maze of repos that is go-ipfs), or at least document that the default size will not work seamlessly for go-ipfs interop.
Note that at Actyx we might have some old blocks from the go-ipfs times laying around which are >1mb.
I'm completely fine with increasing it, it can just never be decreased again once it is increased. also I'm still unhappy about protocol labs thinking it's ok to make this "implementation defined". @vmx can protocol labs finally decide on an arbitrary number without falling into analysis paralysis?
I'm on it trying to find it out once for good.
The maximum block size bitswap supports is actually 2MiB. However, if you're using the default UnixFS encoder, there's some framing overhead so the maximum chunk size is less than 2MiB.
The following rules should cover it:
- Don't chunk larger than 1MiB. If a block is slightly larger than 1MiB, that should be fine.
- Support blocks up to 2MiB.
I stand corrected. The max message size is actually 4MiB. I still recommend not creating blocks larger than 1MiB+epsilon, but it would be safer to support blocks up to 4MiB.
(note: supporting blocks up to 2MiB should still be enough given the recommended 1MiB chunk size max)
It's not quite that easy. js-ipfs uses a 16k block size [0]. I'd appreciate this to be added to some spec somewhere.
Not quite. That's the target message size, not the max message size. Basically, if the message size is less than 16KiB, we'll try to pack more into the message. Once it hits 16KiB, we'll send it off.
But yeah, we do need to spec this (both in IPLD as a "recommendation" and in bitswap).
@Stebalien what's the status of this? Is there a spec or PR I can reference in the code?