'extend google.protobuf.FileOptions' in .milvus.proto.milvus Langchainjs 0.0.83 Nextjs

Question

'extend google.protobuf.FileOptions' in .milvus.proto.milvus Langchainjs 0.0.83 Nextjs

ssawant opened this issue 2 years ago · 18 comments

Describe the bug:
Langchainjs nextjs application Milvus.fromExistingCollition request is failing due to below error. compiled milvus-sdk-node is failing

Steps to reproduce:

Install latest version on LangChainjs 0.0.83
Dockerize as Langchainjs nextjs application
Error thrown in following line

const vectorStoreMilvus = await Milvus.fromExistingCollection(
      new OpenAIEmbeddings({}),
      {
        collectionName:  <>
      }
    );

Error :

error Error: unresolvable extensions: 'extend google.protobuf.FileOptions' in .milvus.proto.milvus
    at Root.resolveAll (/app/node_modules/protobufjs/src/root.js:256:15)
    at loadProtosWithOptionsSync (/app/node_modules/@grpc/proto-loader/build/src/util.js:68:16)
    at loadSync (/app/node_modules/@grpc/proto-loader/build/src/index.js:197:61)
    at getGRPCService (/app/node_modules/@zilliz/milvus2-sdk-node/dist/utils/Grpc.js:26:57)
    at MilvusClient.BaseClient (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/BaseClient.js:58:56)
    at MilvusClient.Collection [as constructor] (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/Collection.js:87:42)
    at MilvusClient.Data [as constructor] (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/Data.js:81:47)
    at MilvusClient.Index [as constructor] (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/MilvusIndex.js:72:42)
    at MilvusClient.Partition [as constructor] (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/Partition.js:61:42)
    at MilvusClient.Resource [as constructor] (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/Resource.js:61:42)

Milvus-node-sdk version:

Milvus version:

Answer 1 · 2023-05-27T08:19:08.000Z

Let me do some test.

Answer 2 · 2023-05-27T08:20:32.000Z

Would you like to share you code?

Answer 3 · 2023-06-01T12:18:23.000Z

I have the same issue, running on vercel, sveltekit setup.
"@zilliz/milvus2-sdk-node": "^2.2.10",

relevant code 👇
`
import { DataType, MilvusClient } from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient(address, secure, user, password);

	const collection_name = 'foobar';
	await client.loadCollectionSync({
		collection_name
	});

            const vector = [...Array(512)].map(() => Math.random());

	const res = await client.search({
	  collection_name,
	  vectors: [vector],
	  search_params: {
	    anns_field: "vector",
	    metric_type: "L2",
	    params: JSON.stringify({ nprobe: 64 }),
	    topk: '5',
	  },
	  output_fields: ['baz'],
	  vector_type: DataType.FloatVector,
	});

`

Answer 4 · 2023-06-01T12:21:09.000Z

vercel

Thanks. maybe there is some limitation on vercel platform. let me test it.

Answer 5 · 2023-06-01T13:17:50.000Z

it's the same for lower vector dims like 128

Answer 6 · 2023-06-01T19:23:41.000Z

It breaks on Netlify as well, edge function related maybe.

Answer 7 · 2023-06-04T00:49:00.000Z

Nextjs doesn't load proto files by default, you should modify webpack config: copy these file into the build folder.

This config works for me.

const path = require("path");
const CopyWebpackPlugin = require("copy-webpack-plugin");

/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      // Copy the proto files to the server build directory
      config.plugins.push(
        new CopyWebpackPlugin({
          patterns: [
            {
              from: path.join(
                __dirname,
                "node_modules/@zilliz/milvus2-sdk-node/dist"
              ),
              to: path.join(__dirname, ".next"),
            },
          ],
        })
      );
    }
    // Important: return the modified config
    return config;
  },
};

module.exports = nextConfig;

Answer 8 · 2023-06-17T10:09:48.000Z

Nextjs 默认不加载 proto 文件，你应该修改 webpack 配置：将这些文件复制到构建文件夹中。

这个配置对我有用。

const path = require("path");
const CopyWebpackPlugin = require("copy-webpack-plugin");

/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      // Copy the proto files to the server build directory
      config.plugins.push(
        new CopyWebpackPlugin({
          patterns: [
            {
              from: path.join(
                __dirname,
                "node_modules/@zilliz/milvus2-sdk-node/dist"
              ),
              to: path.join(__dirname, ".next"),
            },
          ],
        })
      );
    }
    // Important: return the modified config
    return config;
  },
};

module.exports = nextConfig;

👋 我也碰到了类似的问题：ENOENT: no such file or directory, open 'C:/Users/Administrator/Desktop/official-website/.next/server/app/proto/proto/schema.proto'，你这个方案貌似解决了我的问题，棒棒哒🤣

Answer 9 · 2023-08-12T03:07:32.000Z

@shanghaikid

Can the protoPath variable be configured? From the code, it seems to use __dirname, which would cause the protoPath to constantly change by caller route in Next.js, making it more troublesome to use.

Answer 10 · 2023-08-12T06:09:23.000Z

Or make it nextjs-milvus

Or is prisma-milvus

Answer 11 · 2023-08-28T07:31:02.000Z

v2.3.0 works like a charm. Thanks @shanghaikid

Answer 12 · 2023-09-04T23:40:39.000Z

Hello,
I have the some problem with Nuxt3 when build app.

Package version : @zilliz/milvus2-sdk-node": "^2.3.1"

Error message:

unresolvable extensions: 'extend google.protobuf.FileOptions' in .milvus.proto.milvus
at Root.resolveAll (//.output/server/node_modules/protobufjs/src/root.js:256:15)
at loadProtosWithOptionsSync (//.output/server/node_modules/@grpc/proto-loader/build/src/util.js:68:16)
at loadSync (//.output/server/node_modules/@grpc/proto-loader/build/src/index.js:197:61)

Answer 13 · 2023-10-26T18:30:41.000Z

Nextjs doesn't load proto files by default, you should modify webpack config: copy these file into the build folder.

This config works for me.

const path = require("path");
const CopyWebpackPlugin = require("copy-webpack-plugin");

/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      // Copy the proto files to the server build directory
      config.plugins.push(
        new CopyWebpackPlugin({
          patterns: [
            {
              from: path.join(
                __dirname,
                "node_modules/@zilliz/milvus2-sdk-node/dist"
              ),
              to: path.join(__dirname, ".next"),
            },
          ],
        })
      );
    }
    // Important: return the modified config
    return config;
  },
};

module.exports = nextConfig;

so this problem can not resolve? I think copy-webpack is a dirty code for me.

Answer 14 · 2023-10-27T09:26:24.000Z

Nextjs doesn't load proto files by default, you should modify webpack config: copy these file into the build folder.
This config works for me.

const path = require("path");
const CopyWebpackPlugin = require("copy-webpack-plugin");

/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      // Copy the proto files to the server build directory
      config.plugins.push(
        new CopyWebpackPlugin({
          patterns: [
            {
              from: path.join(
                __dirname,
                "node_modules/@zilliz/milvus2-sdk-node/dist"
              ),
              to: path.join(__dirname, ".next"),
            },
          ],
        })
      );
    }
    // Important: return the modified config
    return config;
  },
};

module.exports = nextConfig;

so this problem can not resolve? I think copy-webpack is a dirty code for me.

You can define the milvus proto file in the clientOptions. or you can create a pr to make it better.

Answer 15 · 2023-11-14T16:23:58.000Z

Hey @shanghaikid .
I managed to run nextjs with your solution but just on "npm run dev"(when changing to what you said) but when I'm doing npm run build I get this error:
Collecting page data ..Error: Unable to load service: milvus.proto.milvus.MilvusService from XXXXXX/.next/proto/proto/milvus.proto but the file exists in the direcoty

Answer 16 · 2024-02-23T07:26:15.000Z

Hey @shanghaikid . I managed to run nextjs with your solution but just on "npm run dev"(when changing to what you said) but when I'm doing npm run build I get this error: Collecting page data ..Error: Unable to load service: milvus.proto.milvus.MilvusService from XXXXXX/.next/proto/proto/milvus.proto but the file exists in the direcoty

same question here.

Answer 17 · 2024-02-23T07:28:12.000Z

Hey @shanghaikid . I managed to run nextjs with your solution but just on "npm run dev"(when changing to what you said) but when I'm doing npm run build I get this error: Collecting page data ..Error: Unable to load service: milvus.proto.milvus.MilvusService from XXXXXX/.next/proto/proto/milvus.proto but the file exists in the direcoty

same question here.

https://github.com/milvus-io/milvus-sdk-node/tree/main/examples/nextjs you can check this example.

Answer 18 · 2024-03-06T02:56:20.000Z

https://github.com/milvus-io/milvus-sdk-node/tree/main/examples/nextjs

Readme upgraded, close the issue.