Zip file too big

Question

Zip file too big

xavdid opened this issue 6 years ago · 14 comments

Getting a couple of reports about users hitting the size limit for zip files during push under normal app creation circumstances. Our current build.zip limit is 6MB and the source.zip is 10MB.

My gut is that we're not walking the tree well enough and too many files are being included in the build. This is in puzzling contrast to #332, where our builds are too small. In either case, build is misbehaving.

In either case, I think we should tweak the build command to do any number of the following:

use browserify/webpack/rollup to put all the code in a single large file (aka tree shaking) that only contains the necessary functions. I'm not sure how well the current strategy of pulling entire required files in is working
minify all the code

That's a more involved change though. In the shorter term, here are two devs who hit this and accompanying logs, etc.

Case 1 (slack thread)

zapier build when depending on core 6.1.0 fails, but 6.0.0 passes. Files for each of those builds are in this gist. The sizes of the two builds are as follows:

5296766 Jul 12 17:12 build-6.0.0.zip
6388386 Jul 12 17:09 build-6.1.0.zip

One is over the 6MB limit, the other isn't. In hindsight, I should have had the user change the version of cli they built with (the only difference here was the core dependency). In any case, diffing those lists shows inclusion of two more full copies of lodash in 6.1.0, which is probably where the extra size is coming from.

Case 2 (slack thread)

In this case, adding the moment.js dependency made the source.zip balloon up (which is weird, since that shouldn't be in source, only build). This will update with logs when we have them

logs: https://gist.github.com/xavdid/2a6a374989bcbfbe39199ce4b7289d6d

Answer 1 · 2018-07-19T20:20:10.000Z

Received an email from a Partner running into this issue when trying to push the latest version. He upgraded to zapier-platform-cli@7.0.1 but then all the pushes were failng with an Internal Server Error 500. He rolled back the change to 7.0.0, but is still getting the error.

Internal logs here of "Exception: Source file is too big."
https://logs.internal.zapier.com/search?rangetype=absolute&from=2018-07-19T18%3A13%3A36.000Z&to=2018-07-19T18%3A13%3A46.000Z&q=pid%3A%2214993%22%20AND%20source%3A%22ip%5C-10%5C-0%5C-10%5C-232%22&highlightMessage=77a946d0-8b7f-11e8-a348-0a98fb293f02&fields=message%2Csource

It did look like from our audit logs that the app was changed to the latest version.

Answer 2 · 2018-09-24T09:26:49.000Z

@xavdid would it be an option to check for the maximum size on the CLI size and perhaps even drop source.zip if it's to big, and only send build.zip?

Answer 3 · 2018-09-26T20:28:00.000Z

I think (and it's a guess, I haven't looked at this in a while) that source.zip is unrelated. This is an issue with which files are pulled into the build.zip file.

Answer 4 · 2018-09-27T14:22:14.000Z

Ah, never mind - my angle was https://github.com/zapier/zapier/issues/18574. I've opened https://github.com/zapier/zapier/pull/20154 to improve the error message but I wonder if the CLI should retry with just the build when this happens?

Answer 5 · 2018-09-27T17:20:36.000Z

Not sure! it's weird that we have a total upload limit vs individual limits for both files. maybe that's worth addressing.

Though honestly, I'd rather fix the root cause of the build function misbehaving (see OP) than spend a lot of time on error messages our users hopefully won't have to see.

Answer 6 · 2018-09-27T17:25:26.000Z

Sure, I just wondered if us adding source.zip makes it more likely that we hit limits, but since the limit of build.zip is 6MB anyway, only weird edge cases would get the total over 50MB.

It'd maybe be good to error on the CLI-side if either reach their max.

Answer 7 · 2018-11-27T18:21:20.000Z

Circling back here, it seems like the culprit for source.zip being too big is node_modules not being listed in the .gitignore (or not having a .gitignore at all. We should expand that code to look for other common ignore files

Answer 8 · 2019-01-17T11:14:05.000Z

Same issue here, source.zip 31MB when updated to platform-core-7.6.0, .gitignore is ok. @FokkeZB can you help me ? I tried any possible way, still too big
ID: 6297

Answer 9 · 2019-01-23T10:25:37.000Z

@lucianfcorodeanu could you share the ZIP files in your build directory? You can send it to contact@zapier.com and mention this URL. Thanks!

Answer 10 · 2019-03-19T05:43:04.000Z

Our project add the "zapier-platform-legacy-scripting-runner": "3.0.1" as a dependency, then the build.zip exceed 6MB.

Answer 11 · 2019-03-28T08:36:30.000Z

Declaring the zapier-platform-cli as a dev dependency:

  "dependencies": {
    "zapier-platform-core": "7.6.1",
    "zapier-platform-legacy-scripting-runner": "3.0.1"
  },
  "devDependencies": {
    "mocha": "^5.2.0",
    "should": "^13.2.0",
    "zapier-platform-cli": "^8.0.1"
  },

Shrink the bundle size to 4.5MB.

Answer 12 · 2019-04-08T21:36:31.000Z

Circling back here, it seems like the culprit for source.zip being too big is node_modules not being listed in the .gitignore (or not having a .gitignore at all. We should expand that code to look for other common ignore files

my files were too large (~30MB) and included all node_modules. I found this thread and reading this comment helped me: i had no .gitignore at all and created one which solved my problem.

Answer 13 · 2019-04-08T21:38:14.000Z

glad to hear that! i've got some work slotted to clean up the messaging around this and add a warning if we're not excluding anything.

Answer 14 · 2019-05-02T17:49:04.000Z

Given that we now throw a warning if there's no gitignore (source), and throw a better server error when the zip is too big, i'm going to mark as closed. Thanks y'all!