Problems getting root hash for static sites re-uploaded as Buffers
john-osullivan opened this issue · 8 comments
I am trying to create an automated tool for deploying static websites to IPFS and am not getting an updated root hash when I add a new build. Note that I'm running ipfs-http-client@40.0.2 on node@10.x
Usage
I'm calling ipfs.add([...], { wrapWithDirectory: true }) with an array of objects from an unzipped static build -- each object has the file path and then the contents of the file in a Buffer. I'm testing the system by re-deploying new builds of the same source. Some of the file names change for each build, but some of them stay the same. Source snippet below.
Expected Behavior
I expect that even if some of the files stay the same, changing any of them would return a new root hash which has the appropriate links to both the unchanged and updated files. If I tried to add the exact same content, I would expect it to be idempotent and simply return the first set of hashes.
Current Behavior
The first time I called the function, I got back a full list of hashes, including the root hash where the site could be accessed. After that, though, I only get back a subset of the files added. It's a different subset each time, I'm having trouble seeing any pattern in which files get returned from add() and which ones don't.
Anyone have any tips for resolving this? Am I misunderstanding how IPFS would work in this scenario, or is this a bug with the client? My first guess was that the Promise was resolving too soon, before all the files had a chance to finish uploading.
Code
const ipfsClient = new ipfshttpclient({
host: 'ipfs.infura.io', protocol: 'https', port: 5001
});
const files = [];
// zipStream is the zipped build as a Readable
await zipStream
.pipe(unzipper.Parse())
.on('entry', async (entry: Entry) => {
if (entry.path !== '' && entry.type === 'File') {
const content = await entry.buffer();
const path = entry.path;
// Only add files with proper paths to the array
files.push({ content, path });
} else {
// Implementation detail for unzipper
entry.autodrain()
}
})
.promise()
// Returns a different subset of hashes depending on the run
const uploadedFilesArray: ipfsCreateResponse[] = await ipfsClient.add(files, { wrapWithDirectory: true });Debug Logs
Here are the paths added for successive add()s which are using identical builds and run immediately one after the other. Notice how this first one has paths for all intermediate directories, while the following two do not, and then the fourth one has it again:
Infura
Run 1 - Success!:
Uploaded the following path & hash pairs : [ 'index.html: QmciqyBJNyZSbbvrD8bLL9CJto456a8P8jdC9xRnJvP7z1',
'static/js/2.8f2728aa.chunk.js.map: QmYBZehyKGSa9yANtS9XeaaHGNgRBmRYGEh4ZyXLmkt9Dp',
'precache-manifest.43b2906cf16c91a3b27f89659eed165a.js: QmcwUccAY6JRh5SuZ9vErCgTmNupkDw6CnEM5XLPHMdaTD',
'static/css/main.6e9dd309.chunk.css: QmXFEbTyGABGvGbey8hhiuModvRkZSxR6JPTQ7aidu11ed',
'static/js/runtime~main.d653cc00.js: QmZr8DQ21vnJSnkQhnanHM5P8fppSCt8BHAkoEqhA7S5sd',
'static/js/runtime~main.d653cc00.js.map: QmbwbsrQmRY31cCJ4dXkminAPasSe1UKSZ9mqwv2sL1u5P',
'favicon.ico: QmcFc6EPhavNSfdjG8byaxxV6KtHZvnDwYXLHvyJQPp3uN',
'asset-manifest.json: QmVPt8canjAHijvELoUHCkRJPTUiDTSPrtVDFirdYEapes',
'service-worker.js: QmZdyXZvyrjQyJdvzB1dAtXToRZTgDacoGnwf7UHb3MsNh',
'manifest.json: QmfKqCqGsesAk1JhovJCQajk8awxuqjDPc8QnHa39oDx6g',
'static/js/main.772c1abb.chunk.js.map: QmRS669c8iYbzEWhy5K1PKqwKT5AEJCjnyXbbfu7UVm1vK',
'static/js/main.772c1abb.chunk.js: QmNbxSyj9VhXMbqkLF1yhMo5JkTUJSGpFiCGUR9FU8Z6cR',
'static/css/main.6e9dd309.chunk.css.map: QmPpcvLLnMkhE3PxtG5DWBT4Dhwhax18MnfgKL7RaWy6AS',
'static/js/2.8f2728aa.chunk.js: QmQZDepGSrPUxwYSAHdeiscjModJ2pnjpx1nyvG4rH1k8N',
'static/css: QmP3yzARoP7MTpuqZRwQop6oWs4RMPzrHS634473qC1igq',
'static/js: QmXjqfBzsaULbJppYyX92KnnbucuAY4FY2UbUPYwxNHYRQ',
'static: QmVq6unKm7zU4bfPAkR3ivnjK2ncJJqUEFHzEXF6RHLRgn',
': Qma5JXuFfewjVMhnzxfKcsaqsTK6mUFAikxZ5CMbVXZCKJ' ]
Run 2 - Failure, missing directories:
Uploaded the following path & hash pairs : [ 'index.html: QmciqyBJNyZSbbvrD8bLL9CJto456a8P8jdC9xRnJvP7z1',
'static/js/2.8f2728aa.chunk.js.map: QmYBZehyKGSa9yANtS9XeaaHGNgRBmRYGEh4ZyXLmkt9Dp',
'precache-manifest.43b2906cf16c91a3b27f89659eed165a.js: QmcwUccAY6JRh5SuZ9vErCgTmNupkDw6CnEM5XLPHMdaTD',
'static/css/main.6e9dd309.chunk.css: QmXFEbTyGABGvGbey8hhiuModvRkZSxR6JPTQ7aidu11ed',
'static/js/runtime~main.d653cc00.js: QmZr8DQ21vnJSnkQhnanHM5P8fppSCt8BHAkoEqhA7S5sd',
'static/js/runtime~main.d653cc00.js.map: QmbwbsrQmRY31cCJ4dXkminAPasSe1UKSZ9mqwv2sL1u5P',
'favicon.ico: QmcFc6EPhavNSfdjG8byaxxV6KtHZvnDwYXLHvyJQPp3uN',
'asset-manifest.json: QmVPt8canjAHijvELoUHCkRJPTUiDTSPrtVDFirdYEapes',
'service-worker.js: QmZdyXZvyrjQyJdvzB1dAtXToRZTgDacoGnwf7UHb3MsNh',
'manifest.json: QmfKqCqGsesAk1JhovJCQajk8awxuqjDPc8QnHa39oDx6g',
'static/js/main.772c1abb.chunk.js.map: QmRS669c8iYbzEWhy5K1PKqwKT5AEJCjnyXbbfu7UVm1vK',
'static/js/main.772c1abb.chunk.js: QmNbxSyj9VhXMbqkLF1yhMo5JkTUJSGpFiCGUR9FU8Z6cR',
'static/css/main.6e9dd309.chunk.css.map: QmPpcvLLnMkhE3PxtG5DWBT4Dhwhax18MnfgKL7RaWy6AS' ]
Run 3, Failure, missing almost everything:
Uploaded the following path & hash pairs : [ 'index.html: QmciqyBJNyZSbbvrD8bLL9CJto456a8P8jdC9xRnJvP7z1' ]
Run 4 - Success!:
Uploaded the following path & hash pairs : [ 'index.html: QmciqyBJNyZSbbvrD8bLL9CJto456a8P8jdC9xRnJvP7z1',
'static/js/2.8f2728aa.chunk.js.map: QmYBZehyKGSa9yANtS9XeaaHGNgRBmRYGEh4ZyXLmkt9Dp',
'precache-manifest.43b2906cf16c91a3b27f89659eed165a.js: QmcwUccAY6JRh5SuZ9vErCgTmNupkDw6CnEM5XLPHMdaTD',
'static/css/main.6e9dd309.chunk.css: QmXFEbTyGABGvGbey8hhiuModvRkZSxR6JPTQ7aidu11ed',
'static/js/runtime~main.d653cc00.js: QmZr8DQ21vnJSnkQhnanHM5P8fppSCt8BHAkoEqhA7S5sd',
'static/js/runtime~main.d653cc00.js.map: QmbwbsrQmRY31cCJ4dXkminAPasSe1UKSZ9mqwv2sL1u5P',
'favicon.ico: QmcFc6EPhavNSfdjG8byaxxV6KtHZvnDwYXLHvyJQPp3uN',
'asset-manifest.json: QmVPt8canjAHijvELoUHCkRJPTUiDTSPrtVDFirdYEapes',
'service-worker.js: QmZdyXZvyrjQyJdvzB1dAtXToRZTgDacoGnwf7UHb3MsNh',
'manifest.json: QmfKqCqGsesAk1JhovJCQajk8awxuqjDPc8QnHa39oDx6g',
'static/js/main.772c1abb.chunk.js.map: QmRS669c8iYbzEWhy5K1PKqwKT5AEJCjnyXbbfu7UVm1vK',
'static/js/main.772c1abb.chunk.js: QmNbxSyj9VhXMbqkLF1yhMo5JkTUJSGpFiCGUR9FU8Z6cR',
'static/css/main.6e9dd309.chunk.css.map: QmPpcvLLnMkhE3PxtG5DWBT4Dhwhax18MnfgKL7RaWy6AS',
'static/js/2.8f2728aa.chunk.js: QmQZDepGSrPUxwYSAHdeiscjModJ2pnjpx1nyvG4rH1k8N',
'static/css: QmP3yzARoP7MTpuqZRwQop6oWs4RMPzrHS634473qC1igq',
'static/js: QmXjqfBzsaULbJppYyX92KnnbucuAY4FY2UbUPYwxNHYRQ',
'static: QmVq6unKm7zU4bfPAkR3ivnjK2ncJJqUEFHzEXF6RHLRgn',
': Qma5JXuFfewjVMhnzxfKcsaqsTK6mUFAikxZ5CMbVXZCKJ' ]
Run 5 - Failure, same as Run 3:
Uploaded the following path & hash pairs : [ 'index.html: QmciqyBJNyZSbbvrD8bLL9CJto456a8P8jdC9xRnJvP7z1' ]
Can you share some working code that exhibits the behaviour? Also what kind and version of IPFS node are you connecting to?
Thanks for taking a look, Alan! I'm connecting to Infura's IPFS Gateway, ipfs.infura.io -- not sure what version it's running. My code snippet looks like this:
(moved to OP)
Do you see the same issue when connecting to a locally running IPFS node?
I haven't really tried it out, as I intend to execute in a Lambda function -- ideally I don't need to run my own dedicated node.
Is your instinct that it's probably an issue with Infura?
We have a lot of tests for adding files to IPFS. It would be good to rule out infura for the purposes of debugging this issue.
Is my understanding correct that the call ought to always return a root hash, even if we're adding files which are duplicates of some previous ones?
I'll try plugging in a different provider and see how it behaves, thank you!
Edit: Just in case, here's Infura's version information: {"Version":"0.4.22","Commit":"c6489e5bc","Repo":"7","System":"amd64/linux","Golang":"go1.12.13"}
Yes
Okay, so this behavior doesn't happen when using a local daemon, but given that this is a JS client, I'm probably not the only person who would prefer to instead send to Infura. I added logs at the top to show how the uploaded set seems to change cyclically.
Any tips on who I would poke at Infura to let them know about their erratic behavior here?
Edit: The final solution here, in case someone comes across this, was to just keep retrying the request until Infura behaved correctly. It usually happens within a few tries -- worst I've seen was a little over ten minutes for a 1.41MB build.
Edit 2: Given that it's certainly an Infura issue, I'm going to close this issue and link to it in a thread on their support forum. Thanks for following up and giving me a sanity check!