Atinoda/text-generation-webui-docker

EXTRA_LAUNCH_ARGS are not honored

jrsperry opened this issue · 6 comments

jrsperry commented

Running in kubernetes, it works fine except the EXTRA_LAUNCH_ARGS are not honored as env vars.

The following I would expect to enable the api and preload a model, neither are enabled though.

            - name: EXTRA_LAUNCH_ARGS
              value: "--listen --verbose --api --extensions api --model TheBloke/vicuna-13B-v1.5-GPTQ --gpus all"

full kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: text-gen-webui
  namespace: text-gen-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      component: text-gen-webui
  template:
    metadata:
      labels:
        component: text-gen-webui
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          value: present
          effect: NoSchedule
      containers:
        - name: text-gen-demo-container
          image: atinoda/text-generation-webui
          ports:
            - containerPort: 7860
            - containerPort: 5000
            - containerPort: 5005
          resources:
            limits:
              nvidia.com/gpu: "1"
          env:
            - name: EXTRA_LAUNCH_ARGS
              value: "--listen --verbose --api --extensions api --model TheBloke/vicuna-13B-v1.5-GPTQ --gpus all"
            - name: TORCH_CUDA_ARCH_LIST
              value: "7.5"
          volumeMounts:
            - name: text-gen-demo-pvc
              mountPath: /app/loras
              subPath: loras
            - name: text-gen-demo-pvc
              mountPath: /app/models
              subPath: models
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: text-gen-demo-pvc
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 1Gi
accountForIssues commented

I use Kubernetes as well and spent a long time trying to figure this out.

In the end, though, it appears that only the first argument is honored i.e., anything after the first whitespace character is being ignored.

If I escape the whitespace however, it seems to work.

This is the only app I have noticed that didn't work with quotes and required escaping spaces.

(part of) my deployment:

          env:
          - name:  EXTRA_LAUNCH_ARGS
            value: --listen\ --verbose
  • 👍2
Atinoda commented

Thank you for sharing this issue, and even bigger thanks for sharing your solution with the escaped spaces! From your findings, I suspect it is the way that the args are being parsed into the launch command. I don't use Kubernetes, but I will try to replicate and fix this by testing a few different parsing strategies.

  • 👍1
redashes1984 commented

Same problem when dockerized deploying on my Unraid NAS. I'm also trying to deploy the source code (from "oobabooga/text-generation-webui") by docker-compose and it working well with env item "CLI_ARGS=--listen --api --xformers --verbose".
also thanks for your work!

jrsperry commented

Ya escaping the chars worked for me. Thanks for the tip!

Artem-B commented

Another workaround is to pass a single quoted string as an argument:

EXTRA_LAUNCH_ARGS="'--api --listen --verbose'"

The root cause of the problem is in

text-generation-webui-docker/scripts/docker-entrypoint.sh

Line 66 in 6726a71

LAUNCHER=($@ $extra_launch_args) 

eval "extra_launch_args=($EXTRA_LAUNCH_ARGS)"
LAUNCHER=($@ $extra_launch_args)

extra_launch_args is set as an array, but used as a scalar. Bash appears to expand it to the first element of the array.
It should be:

eval "extra_launch_args=($EXTRA_LAUNCH_ARGS)"
LAUNCHER=($@ "${extra_launch_args[@]")
  • 👍1
Atinoda commented

@Artem-B - thank you for spotting that and sharing the fix, really nice catch! I've also never really been happy with use of eval here but the parser code is suffering from "nothing more permanent than a temporary fix" syndrome. Happy to be able to improve it without needing to do a deployment-breaking refactor.

This is great timing because the upstream project has not changed since the last stable release, so I will make this fix and push it as the new stable version. Any problems should then be traceable to the entrypoint script and not project code changes. I believe that the changes won't break anybody's install... (he said, ominously). According to my testing, and from snapshot-2024-03-24 onwards:

WORKS

  • "--listen --verbose"
  • --listen --verbose
  • --listen\ --verbose

DOES NOT WORK

  • "--listen\ --verbose"
  • ❤️1