[Bug]: PermissionError: [Errno 13] Permission denied: '/app/aphrodite-engine/.triton'
theobjectivedad opened this issue · 3 comments
Your current environment
Note I am running alpindale/aphrodite-engine:latest
as of 2024-05-08.
Collecting environment information...
PyTorch version: 2.2.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000
Nvidia driver version: 535.86.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Vendor ID: AuthenticAMD
Model name: AMD Ryzen Threadripper PRO 5975WX 32-Cores
CPU family: 25
Model: 8
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 1
Stepping: 2
Frequency boost: enabled
CPU max MHz: 7006.6401
CPU min MHz: 1800.0000
BogoMIPS: 7186.95
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization: AMD-V
L1d cache: 1 MiB (32 instances)
L1i cache: 1 MiB (32 instances)
L2 cache: 16 MiB (32 instances)
L3 cache: 128 MiB (4 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.2.0
[pip3] triton==2.2.0
[conda] Could not collect ROCM Version: Could not collect
Aphrodite Version: 0.5.2
Aphrodite Build Flags:
CUDA Archs: 6.1 7.0 7.5 8.0 8.6 8.9 9.0+PTX; ROCm: Disabled
Additionally, here is the docker run command I'm using (triggered via a Makefile):
docker run -it -d \
--name=$(APHRODITE_NAME)-$(APHRODITE_PROFILE) \
--restart=always \
--shm-size=15g \
--ulimit memlock=-1 \
--ipc=host \
--entrypoint=python3 \
--gpus='"device=$(APHRODITE_GPUS)"' \
--publish=$(APHRODITE_PORT):8000 \
--volume=/models:/models:ro \
--health-cmd="timeout 5 bash -c 'cat < /dev/null > /dev/tcp/localhost/8000'" \
--health-start-period=240s \
--health-interval=15s \
--health-timeout=8s \
--health-retries=3 \
$(APHRODITE_CONTAINER) \
-m aphrodite.endpoints.openai.api_server \
--model /models/$(APHRODITE_MODEL) \
--worker-use-ray \
--context-shift \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--served-model-name $(APHRODITE_MODEL_NAME) \
--kv-cache-dtype auto \
--load-format auto \
--tokenizer-mode auto \
--dtype auto \
--response-role gpt \
--port 8000 \
--host 0.0.0.0
🐛 Describe the bug
Description
When executing a completion request I'm getting the exception below. I believe the root cause is the official container user is running as UID 1000 and /app/aphrodite-engine
is owned by root:root
. Seems to only happen under heavy load. Can replicate with unquantized version of miqu-1-70b-sf.
Full error message
INFO: 172.17.0.1:3226 - "POST /v1/completions HTTP/1.1" 500
ERROR: Exception in ASGI application
ERROR: Traceback (most recent call last):
ERROR: File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR: result = await app( # type: ignore[func-returns-value]
ERROR: File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR: return await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
ERROR: await super().__call__(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
ERROR: await self.middleware_stack(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR: await self.app(scope, receive, _send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR: await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR: await app(scope, receive, sender)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 758, in __call__
ERROR: await self.middleware_stack(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 778, in app
ERROR: await route.handle(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 299, in handle
ERROR: await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 79, in app
ERROR: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR: await app(scope, receive, sender)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
ERROR: response = await func(request)
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
ERROR: raw_response = await run_endpoint_function(
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR: return await dependant.call(**values)
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 281, in create_completion
ERROR: generator = await openai_serving_completion.create_completion(
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 177, in create_completion
ERROR: async for i, res in result_generator:
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 81, in consumer
ERROR: raise item
ERROR: File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR: result = await app( # type: ignore[func-returns-value]
ERROR: File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR: return await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
ERROR: await super().__call__(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
ERROR: await self.middleware_stack(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR: await self.app(scope, receive, _send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR: await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR: await app(scope, receive, sender)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 758, in __call__
ERROR: await self.middleware_stack(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 778, in app
ERROR: await route.handle(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 299, in handle
ERROR: await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 79, in app
ERROR: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR: await app(scope, receive, sender)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
ERROR: response = await func(request)
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
ERROR: raw_response = await run_endpoint_function(
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR: return await dependant.call(**values)
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 281, in create_completion
ERROR: generator = await openai_serving_completion.create_completion(
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 177, in create_completion
ERROR: async for i, res in result_generator:
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 81, in consumer
ERROR: raise item
ERROR: File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR: result = await app( # type: ignore[func-returns-value]
ERROR: File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR: return await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
ERROR: await super().__call__(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
ERROR: await self.middleware_stack(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR: await self.app(scope, receive, _send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR: await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR: await app(scope, receive, sender)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 758, in __call__
ERROR: await self.middleware_stack(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 778, in app
ERROR: await route.handle(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 299, in handle
ERROR: await self.app(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 79, in app
ERROR: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR: raise exc
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR: await app(scope, receive, sender)
ERROR: File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
ERROR: response = await func(request)
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
ERROR: raw_response = await run_endpoint_function(
ERROR: File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR: return await dependant.call(**values)
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 281, in create_completion
ERROR: generator = await openai_serving_completion.create_completion(
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 177, in create_completion
ERROR: async for i, res in result_generator:
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 81, in consumer
ERROR: raise item
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 66, in producer
ERROR: async for item in iterator:
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 625, in generate
ERROR: raise e
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 619, in generate
ERROR: async for request_output in stream:
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 72, in __anext__
ERROR: raise result
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 66, in producer
ERROR: async for item in iterator:
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 625, in generate
ERROR: raise e
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 619, in generate
ERROR: async for request_output in stream:
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 72, in __anext__
ERROR: raise result
ERROR: File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 66, in producer
ERROR: async for item in iterator:
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 625, in generate
ERROR: raise e
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 619, in generate
ERROR: async for request_output in stream:
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 72, in __anext__
ERROR: raise result
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 33, in _raise_exception_on_finish
ERROR: task.result()
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 468, in run_engine_loop
ERROR: has_requests_in_progress = await asyncio.wait_for(
ERROR: File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
ERROR: return fut.result()
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 442, in engine_step
ERROR: request_outputs = await self.engine.step_async()
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 208, in step_async
ERROR: all_outputs = await self._run_workers_async(
ERROR: File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 293, in _run_workers_async
ERROR: all_outputs = await asyncio.gather(*coros)
ERROR: File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR: result = self.fn(*self.args, **self.kwargs)
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR: return func(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/task_handler/worker.py", line 235, in execute_model
ERROR: output = self.model_runner.execute_model(seq_group_metadata_list,
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR: return func(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/task_handler/model_runner.py", line 692, in execute_model
ERROR: hidden_states = model_executable(
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR: return self._call_impl(*args, **kwargs)
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR: return forward_call(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 415, in forward
ERROR: hidden_states = self.model(input_ids, positions, kv_caches,
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR: return self._call_impl(*args, **kwargs)
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR: return forward_call(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 340, in forward
ERROR: hidden_states, residual = layer(
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR: return self._call_impl(*args, **kwargs)
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR: return forward_call(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 287, in forward
ERROR: hidden_states = self.self_attn(
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR: return self._call_impl(*args, **kwargs)
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR: return forward_call(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 228, in forward
ERROR: attn_output = self.attn(q, k, v, k_cache, v_cache, input_metadata,
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR: return self._call_impl(*args, **kwargs)
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR: return forward_call(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/modeling/layers/attention.py", line 213, in forward
ERROR: context_attention_fwd(
ERROR: File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR: return func(*args, **kwargs)
ERROR: File "/app/aphrodite-engine/aphrodite/modeling/layers/triton_kernel/prefix_prefill.py", line 684, in context_attention_fwd
ERROR: _fwd_kernel[grid](
ERROR: File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 532, in run
ERROR: self.cache[device][key] = compile(
ERROR: File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 484, in compile
ERROR: fn_cache_manager = get_cache_manager(make_hash(fn, target, get_env_vars(), _device_backend, **kwargs))
ERROR: File "/usr/local/lib/python3.10/dist-packages/triton/runtime/cache.py", line 152, in get_cache_manager
ERROR: return __cache_cls(key)
ERROR: File "/usr/local/lib/python3.10/dist-packages/triton/runtime/cache.py", line 66, in __init__
ERROR: os.makedirs(self.cache_dir, exist_ok=True)
ERROR: File "/usr/lib/python3.10/os.py", line 215, in makedirs
ERROR: makedirs(head, exist_ok=exist_ok)
ERROR: File "/usr/lib/python3.10/os.py", line 215, in makedirs
ERROR: makedirs(head, exist_ok=exist_ok)
ERROR: File "/usr/lib/python3.10/os.py", line 225, in makedirs
ERROR: mkdir(name, mode)
ERROR: PermissionError: [Errno 13] Permission denied: '/app/aphrodite-engine/.triton'
As a workaround I can pass --user=root
to the docker run args although this obviously this isn't a good practice. If anyone else uses this workaround I recommend the following aphrodite-engine options: always use --load-format safetensors
and never use --trust-remote-code
.
Looking at the contents of /app/aphrodite-engine/.triton
:
root@6c985414e457:~/.triton# find .
.
./cache
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttir
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/__grp___fwd_kernel.json
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttgir
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.llir
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ptx
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.json
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.cubin
./cache/e848d61dc399d578c7a9dd3c6ff8a1a1
./cache/e848d61dc399d578c7a9dd3c6ff8a1a1/_fwd_kernel.so
./cache/6e97c2a1f7a095255f6dd5de1807841d
./cache/6e97c2a1f7a095255f6dd5de1807841d/cuda_utils.so
./dump
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttir
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttgir
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.llir
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ptx
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.cubin
root@6c985414e457:~/.triton# more ./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.json
{"num_warps": 8, "num_ctas": 1, "num_stages": 1, "enable_warp_specialization": false, "enable_persistent": false, "constants": {"14": 1, "31": 1, "35": 1, "36": 128, "37": 128, "38": 1
28}, "debug": null, "target": {"capability": 86, "num_warps": 8, "enable_fp_fusion": true}, "AMDGCN_ENABLE_DUMP": false, "DISABLE_FAST_REDUCTION": false, "DISABLE_MMA_V3": false, "ENAB
LE_TMA": false, "LLVM_IR_ENABLE_DUMP": false, "MLIR_ENABLE_DUMP": false, "TRITON_DISABLE_LINE_INFO": false, "device_type": "cuda", "shared": 65538, "name": "_fwd_kernel_0d1d2d3d4d5d67d
8d9d10de11e12d1314c15de16de17de18de19de20de21de22de23de24de25de26de27de28de29de30e31c32de33de34de35c", "clusterDims": [1, 1, 1]}
That's a weird one - the process should have rw access to the /app/aphrodite-engine
directory. I guess we'll need to explicitly modify the perms for the directory in the Dockerfile. Thanks for the report.
Cool - I'll send over a PR