/milvus-migration

Primary LanguageGo

Milvus Migration: migrate data to Milvus 2.x

Overview

milvus-migration is a data migration tool for Milvus that supports importing Milvus 0.9.x ~ 1.x / faiss / elasticsearch7.x+ datas to milvus 2.x.

Architecture

Getting started

Prerequisites

  • Operating System Requirements
Operating System Supported Versions
CentOS 7.5 or later
Ubuntu LTS 18.04 or later
  • Software Requirements
Software Version
Milvus 0.9.x, 1.x or 2.x
Elasticsearch 7.x or 8.x
go 1.20.2 or later
  • Data Format Support
Source Data Type Target Data Type
Milvus 0.9.x - 1.x Milvus 2.x
Elasticsearch 7.x-8.x Milvus 2.x
Faiss (Beta) Milvus 2.x
Milvus 2.x (in progress) Milvus 2.x

How to use this tool?

  1. Download the source code and build it:
git clone git@github.com:zilliztech/milvus-migration.git
go build
  1. After building, you will obtain the milvus-migration binary file. Place this file and the migration.yaml file in the directory structure like below:
directory
---- milvus-migration
---- configs/migration.yaml
  1. Milvus1.x or Faiss to Milvus2.x migration

Run the command dump to dump the source data to numpy:

./milvus-migration dump

Finally, load the numpy files to Milvus 2.x successfully by using the load command:

./milvus-migration load
  1. Elasticsearch to Milvus2.x migration:

Run the command start to migration es->2.x. here start == dump && load cmd

./milvus-migration start

Migration Examples (migration.yaml details)

how to learn more about using migration tool, see examples doc below:

  1. es -> milvux2.x : migrate_es_doc.
  2. milvus1.x -> milvux2.x : migrate_1.x_doc.
  3. faiss -> milvux2.x ( Beta) : migrate_faiss_doc.

How to verify migration result

When migration finished, you can use visual tool Attu or use Milvus SDK verify your new collection data rows.

Build Index and search

After the Milvus collection Data migration is completed, we can use SDK or Attu to create index and load collection for the next search operation.

Future work

  • Support Redis to Milvus 2.x
  • Support Mongodb to Milvus 2.x
  • Milvus 2.x to Milvus 2.x
  • Support others datasource to Milvus 2.x
  • Supports binary vectors

Contributing

Contributions to milvus-migration are welcome from everyone. See Contributing for details of the contribution workflow.