[Bug]: run milvus-backup in k8s pod, retrurn "failed to connect \"Addr\": \"milvus:tcp:\" "

[jackiedlh]

Current Behavior

When I run milvus-backup command on k8s pod, the target milvus address is wrong.

in backup.yaml:

milvus:
address: milvus
port: 19530
authorizationEnabled: false

in log:

[2023/09/08 07:35:33.943 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2023/09/08 07:35:33.943 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2023/09/08 07:35:33.943 +00:00] [INFO] [core/backup_impl_create_backup.go:25] ["receive CreateBackupRequest"] [requestId=4b980b8b-4e1a-11ee-8398-82b39bdfe663] [backupName=backup] [collections="[]"] [databaseCollections=] [async=false]
[2023/09/08 07:35:34.168 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n "Addr": "milvus:tcp:",\n "ServerName": "milvus:tcp:",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp: address milvus:tcp:: too many colons in address""]

Expected Behavior

expect connect to:

milvus:19530

Steps To Reproduce

1 copy milvus-backup tool to k8s milvus proxy pod
2 copy config files to milvus pod
3 run command:  ./milvus-backup create -n backup

Environment

aws EKS

Anything else?

No response

[wayblink]

address should be a ip or a domain.

[Archalbc]

address should be a ip or a domain.

"milvus" is correct in a Kubernetes context, it will resolve to the milvus-proxy pod, actually this look like a bug.

In debug mode you can have the proof that "milvus" is correctly resolved but badly interpreted by sub-libraries :

[2023/09/18 10:24:01.536 +00:00] [DEBUG] [core/backup_context.go:58] ["Start Milvus client"] [endpoint=milvus:tcp://10.224.44.247:19530]
[2023/09/18 10:24:01.536 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n  \"Addr\": \"milvus:tcp:\",\n  \"ServerName\": \"milvus:tcp:\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp: address milvus:tcp:: too many colons in address\""]

Same issue using : milvus.my-namespace.svc.cluster.local

I guess related code is : https://github.com/xushaoxiao/milvus-backup/blob/80c754eb9bfbb96961f244404ba7a3a5054d3e24/core/backup_context.go#L63C22-L63C50
and
https://github.com/milvus-io/milvus-sdk-go/blob/master/client/client.go

The used func is deprecated as described in comment : https://github.com/milvus-io/milvus-sdk-go/blob/master/client/client.go#L285

No issue with v0.3.2

[2023/09/18 13:01:26.991 +00:00] [DEBUG] [core/backup_context.go:58] ["Start Milvus client"] [endpoint=milvus:443]     

[nightmorning0]

It is definitely a bug. Add PORT# at the tail of the address. Something likes this:

milvus:
address: milvus:19530#
port: 19530
authorizationEnabled: false

It will comment :tcp: from milvus:tcp:: in your error as a tricky solution.

[adeelamin15]

Yup it is definitely a bug. For me workaround is to set the port using MILVUS_PORT environment variable. Apparently the bug is causing garbage value in port field which can be overridden using environment variable.

[wayblink]

/assign

[cin-eren]

When you deploy on K8s pod, it is recommended to set MinIO and Milvus configuration via environment variables, example:

MILVUS_ADDRESS: milvus
MILVUS_PORT: 19530
MINIO_BUCKET_NAME: s3_bucket
MINIO_ROOT_PATH: ""
MINIO_USE_IAM: true
MINIO_USE_SSL: true
MINIO_BACKUP_BUCKET_NAME: s3_backup_bucket
MINIO_BACKUP_ROOT_PATH: backup
MINIO_BACKUP_USE_SSL: true
MINIO_BACKUP_USE_IAM: true

Some configuration must be specified in backup.yaml like minio.storageType, minio.backupStorageType, you can consider to use Configmap for them

milvus:
  authorizationEnabled: false
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  tlsMode: 0

# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  # Milvus storage configs, make them the same with milvus config
  storageType: s3
  address: s3.us-west-2.amazonaws.com
  port: 443

  # Backup storage configs, the storage you want to put the backup data
  backupStorageType: s3 # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
  backupAddress: s3.us-west-2.amazonaws.com # Address of MinIO/S3
  backupPort: 443   # Port of MinIO/S3