Build fails without @aws-sdk/client-s3 package in package.json
MaksimDedov opened this issue · 17 comments
Hey!
When I install the unzipper npm package and don't install separately @aws-sdk/client-s3 I get the build exception like below.
Please move @aws-sdk/client-s3 to required dependencies for automatic installation, or fix it usage.
Thanks in advance
ERROR in ./node_modules/unzipper/lib/Open/index.js 97:52-81
Module not found: Error: Can't resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
Parsed request is a module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./lib/Open)
resolve as module
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open\node_modules doesn't exist or is not a directory
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\node_modules
single file module
using description file: E:\Work\finsemble-mono\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\node_modules doesn't exist or is not a directory
E:\node_modules doesn't exist or is not a directory
@ ./node_modules/unzipper/unzip.js 5:0-36
@ ./src/main/ExternalApplicationManager.ts 4:0-32 70:27-43
@ ./src/main/Main.ts 4:0-74 120:43-69
@ ./src/app.ts 12:0-35 100:17-21
ERROR in E:\Work\finsemble-mono\packages\electron-adapter\src\main\ExternalApplicationManager.ts
33:20-40
[tsl] ERROR in E:\Work\finsemble-mono\packages\electron-adapter\src\main\ExternalApplicationManager.ts(33,21)
TS2307: Cannot find module '@aws-sdk/client-s3' or its corresponding type declarations.
ts-loader-default_04bb2ec108ef140d
webpack 5.91.0 compiled with 2 errors and 2 warnings in 18167 ms
@MaksimDedov plus one facing the same issue.
same here
To keep node-unzipper super small, a decision was made to not include optional third party sdks as a part of the library itself. unzipper has plenty of users that do not require the s3 features and it would be very inefficient to force them to do so (example: https://www.npmjs.com/package/exceljs)
If you want to use the aws_s3 you will have to ensure that the sdk is installed, i.e. npm install unzipper @aws-sdk/client-s3, perhaps the right thing to do is clarify this in the README
@MaksimDedov can you confirm the build error only occurs because you are actually use the aws_v3 in the build process?
@ZJONSSON In my code I use unzipper in the next way and only once:
const unzipperStream = unzipper.Extract({ path: downloadAsset.assetFolder });
fileStream.pipe(unzipperStream);
I don't call any other functions. The downloadAsset.assetFolder is the path to local folder on machine, fileStream is Node.js transform stream. As I understood the Extract uses aws_v3 somewhere under the hood in ./node_modules/unzipper/lib/Open/index.js 97:52-81
The build error occurs if I don't add the aws_v3 in my package.json near unzipper
Thank that is very helpful. This seems to be a typescript/ts-loader compilation error not a runtime error. Seems like the appropriate fix here is a simple @ts-ignore.
@MaksimDedov can you try with unzipper@0.12.3 that was just published?
unfortunately I see the same issue for unzipper@0.12.3:
ERROR in ./node_modules/unzipper/lib/Open/index.js 98:52-81
Module not found: Error: Can't resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
resolve '@aws-sdk/client-s3' in 'E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open'
Parsed request is a module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./lib/Open)
resolve as module
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\Open\node_modules doesn't exist or is not a directory
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\lib\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\unzipper\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\packages\electron-adapter\node_modules
single file module
using description file: E:\Work\finsemble-mono\packages\electron-adapter\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\packages\electron-adapter\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\finsemble-mono\packages\node_modules doesn't exist or is not a directory
looking for modules in E:\Work\finsemble-mono\node_modules
single file module
using description file: E:\Work\finsemble-mono\package.json (relative path: ./node_modules/@aws-sdk/client-s3)
no extension
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
.ts
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.ts doesn't exist
.js
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.js doesn't exist
.json
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.json doesn't exist
.node
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3.node doesn't exist
E:\Work\finsemble-mono\node_modules\@aws-sdk\client-s3 doesn't exist
E:\Work\node_modules doesn't exist or is not a directory
E:\node_modules doesn't exist or is not a directory
@ ./node_modules/unzipper/unzip.js 5:0-36
@ ./src/main/ExternalApplicationManager.ts 4:0-32 70:27-43
@ ./src/main/Main.ts 4:0-74 120:43-69
@ ./src/exports.ts 31:24-46
webpack 5.91.0 compiled with 1 error and 2 warnings in 19696 ms
Same error on my side ^^"
"unzipper": "^0.12.3"
Stack trace from Next:
Module not found: Can't resolve '@aws-sdk/client-s3'
https://nextjs.org/docs/messages/module-not-found
Import trace for requested module:
./node_modules/unzipper/unzip.js
./src/server/actions/files.ts
./src/server/data/mods.ts
./node_modules/next/dist/build/webpack/loaders/next-flight-action-entry-loader.js?actions=%5B%5B%22E%3A%5C%5CGitHub%5C%5Cfaithful%5C%5Cwebsite%5C%5Csrc%5C%5Cserver%5C%5Cdata%5C%5Cmods.ts%22%2C%5B%22getModsWithVersions%22%2C%22deleteMod%22%2C%22updateModPicture%22%2C%22voidMods%22%2C%22getMods%22%2C%22getModsFromIds%22%2C%22getModWithModVersions%22%2C%22updateMod%22%2C%22createMod%22%2C%22modHasUnknownVersion%22%5D%5D%2C%5B%22E%3A%5C%5CGitHub%5C%5Cfaithful%5C%5Cwebsite%5C%5Csrc%5C%5Cserver%5C%5Cdata%5C%5Cmods-version.ts%22%2C%5B%22getModVersionProgression%22%2C%22createModVersion%22%2C%22getSupportedMinecraftVersions%22%2C%22removeModpackFromModVersion%22%2C%22getModsVersionsFromResources%22%2C%22updateModVersion%22%2C%22deleteModVersion%22%2C%22getModVersionsWithModpacks%22%2C%22getNumberOfTextureFromModVersion%22%2C%22getModVersions%22%2C%22addModVersionsFromJAR%22%5D%5D%5D&__client_imported__=true!
GET / 500 in 1049ms
GET /_next/static/webpack/a1548000c16fc3ef.webpack.hot-update.json 500 in 1028ms
⚠ Fast Refresh had to perform a full reload due to a runtime error.
And the only thing I'm doing with unzipper is the following:
const bytes = await jar.arrayBuffer(); // where jar is a File
const buffer = Buffer.from(bytes);
const archive = await unzipper.Open.buffer(buffer);Similar issue when using "unzipper": "^0.12.3" and NextJS
> next build
▲ Next.js 14.2.5
- Environments: .env
Creating an optimized production build ...
Failed to compile.
./node_modules/unzipper/lib/Open/index.js
Module not found: Can't resolve '@aws-sdk/client-s3'
https://nextjs.org/docs/messages/module-not-found
Temporary workaround: Add client-s3 to devDependencies
npm i -D @aws-sdk/client-s3@latest
I'm getting this error in my vue app
My Code
This is the only instance where I'm using the library, and it builds fine without it.
const directory = await Unzipper.Open.buffer(newFile)Terminal Error
13:29:14 [vite] hmr update /src/pages/media/add/media.vue, /src/components/media/AddFiles.vue (x2)
✘ [ERROR] Could not resolve "@aws-sdk/client-s3"
node_modules/.pnpm/unzipper@0.12.3/node_modules/unzipper/lib/Open/index.js:98:60:
98 │ const { GetObjectCommand, HeadObjectCommand } = require('@aws-sdk/client-s3');
╵ ~~~~~~~~~~~~~~~~~~~~
You can mark the path "@aws-sdk/client-s3" as external to exclude it from the bundle, which will
remove this error and leave the unresolved path in the bundle. You can also surround this
"require" call with a try/catch block to handle this failure at run-time instead of bundle-time.
13:29:17 [vite] error while updating dependencies:
Error: Build failed with 1 error:
node_modules/.pnpm/unzipper@0.12.3/node_modules/unzipper/lib/Open/index.js:98:60: ERROR: Could not resolve "@aws-sdk/client-s3"
at failureErrorWithLog (C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\esbuild@0.19.12\node_modules\esbuild\lib\main.js:1651:15)
at C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\esbuild@0.19.12\node_modules\esbuild\lib\main.js:1059:25
at C:\Users\ryanvarley\Code\cms.frontend.ui\node_modules\.pnpm\esbuild@0.19.12\node_modules\esbuild\lib\main.js:1527:9
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
UPDATE 1
I tried adding in the error handling as described by the message in my terminal and it got me past the error. Unfortunately I'm getting some other errors now:
TypeError: util.inherits is not a function
at node_modules/.pnpm/unzipper@0.12.3/node_modules/unzipper/lib/PullStream.js (PullStream.js:18:6)
at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
at node_modules/.pnpm/unzipper@0.12.3/node_modules/unzipper/lib/parse.js (parse.js:4:20)
at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
at node_modules/.pnpm/unzipper@0.12.3/node_modules/unzipper/unzip.js (unzip.js:2:17)
at __require2 (chunk-GFT2G5UO.js?v=9e3c5976:18:50)
at unzip.js:5:16
Is Unzipper not designed to work within a non-server environment or something? If it's any help, I'm using Node v20.14.0 and I'm compiling to ESNext
UPDATE 2
util is a node specific module that is not available in the browser. A step towards solving this is to polyfill those modules. As my project uses vite, this is best achieved with a package like https://github.com/davidmyersdev/vite-plugin-node-polyfills#vite-plugin-node-polyfills. That said, this still didn't quite work for me, so I'm going to try a browser first package like jszip.
This package does look really nice though!
@ZJONSSON
it is a good idea to keep the package small. but since you are using require then it failed to bundle or build in the absence of the optional packages
i recommend to use something like "optional-require" so it wont fail and you can check the dependency you need before using it
This is a bundling issue. Bundler can't tell at build time whether the dependency will be used or not, so tries to bundle it but it's not installed.
For those using Nextjs (or a webpack based framework), something like the following will work as an alternative to installing the dependency:
const nextConfig: NextConfig = {
webpack: (config) => {
config.resolve.alias = {
...config.resolve.alias,
"@aws-sdk/client-s3": false,
};
}@ZJONSSON how about having different entry points for this? Something like:
import { Open } from `unzipper/s3`I think you can do this within the one package by splitting out the code and modifying your package.json:
"name": "unzipper",
"exports": {
".": "./unzip.js",
"./s3": "./lib/Open/s3.js",
}
}
Alternatively listing @aws-sdk/client-s3 as a peerDependency might generate more helpful logs.
PS: great package, thanks for shipping! ❤️
I am also facing the problem Error: Cannot find module '@aws-sdk/client-s3' with vite and version 0.12.3
"@aws-sdk/client-s3" is imported by "@aws-sdk/client-s3?commonjs-external", but could not be resolved – treating it as an external dependency.
0.11.6 fixed the problem