日期2023-02-11后oneflow包导致，程序在运行过程中出现错误导致程序卡死。
ccssu opened this issue · 1 comments
ccssu commented
环境信息
第一次出现的该错误的commit
程序运行过程中出现错误
错误栈信息
环境信息
- 机器 oneflow27-root
- 编译oneflow不同版本
第一次出现的该错误的commit
commit 号： c3a7e8c8c98538cc061263c86e8036db422d5295 对应 pr : https://github.com/Oneflow-Inc/oneflow/pull/9851
commit c3a7e8c8c98538cc061263c86e8036db422d5295 # is bad
Author: daquexian <daquexian566@gmail.com>
Date: Sat Feb 11 17:00:21 2023 +0800
Refine vm::TensorStorage (#9851)
从 DTR 里抽出来的一部分修改
1. 把 vm::TensorStorage 挪到专门的文件 oneflow/core/eager/tensor_storage.* 里
2. 用组合（TensorStorage +
true/false）代替继承（InsideVmTensorStorage/OutsideVmTensorStorage），减少继承层级
---------
Signed-off-by: daquexian <daquexian566@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>程序运行过程中出现错误
python classify/train.py --model /home/fengwen/one-yolov5/runs/export/yolov5n.of --data /data/dataset/fengwen/classify/imagenette160 --project tes
t_nightly_v1_2_0 --name exp_cls --epochs 90 --img 224
wandb: Currently logged in as: wearmheart. Use `wandb login --relogin` to force relogin
classify/train: model=/home/fengwen/one-yolov5/runs/export/yolov5n.of, data=/data/dataset/fengwen/classify/imagenette160, epochs=90, batch_size=64
, imgsz=224, nosave=False, cache=None, device=, workers=8, project=test_nightly_v1_2_0, name=exp_cls, exist_ok=False, pretrained=True, optimizer=A
dam, lr0=0.001, decay=5e-05, label_smoothing=0.1, cutoff=None, dropout=None, verbose=False, seed=0, local_rank=-1, multi_tensor_optimizer=False
fatal: ambiguous argument 'upgrade_yolov5_064365d8683fd002e9ad789c1e91fa3d021b44f0..origin/master': unknown revision or path not in the working tr
ee.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Command 'git rev-list upgrade_yolov5_064365d8683fd002e9ad789c1e91fa3d021b44f0..origin/master --count' returned non-zero exit status 128.
YOLOv5 🚀 v1.2.0-26-gf94dc927 Python-3.8.13 oneflow-0.9.1+cu117.git.c3a7e8c8c9 CUDA:0 (NVIDIA GeForce RTX 3090, 24268MiB)
34/90 2.187G 0.511 : 39%|███▉ | 58/148 [00:01<00:02, 37.87it/s]Traceback (most recent call
last):
File "/data/dataset/fengwen/miniconda3/envs/python3.8/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
obj = _ForkingPickler.dumps(obj)
File "/data/dataset/fengwen/miniconda3/envs/python3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "/data/dataset/fengwen/package/oneflow/python/oneflow/multiprocessing/reductions.py", line 87, in reduce_tensor
shm = shared_memory.SharedMemory(create=True, size=tensor_data.nbytes)
File "/data/dataset/fengwen/package/oneflow/python/oneflow/multiprocessing/shared_memory/__init__.py", line 28, in __init__
self.shm_ = flow._oneflow_internal.multiprocessing.SharedMemory(
oneflow._oneflow_internal.exception.Exception: Resource temporarily unavailable
File "/data/dataset/fengwen/package/oneflow/oneflow/core/ipc/shared_memory.cpp", line 175, in Open
ShmSetUp(&shm_name, shm_size, create)
File "/data/dataset/fengwen/package/oneflow/oneflow/core/ipc/shared_memory.cpp", line 70, in ShmSetUp
Error Type: oneflow.ErrorProto.check_failed_errorccssu commented
错误栈信息
bt # 当前C调用栈
py-bt # 当前Py调用栈
py-list # 当前py代码位置
当前Py调用栈
bt 当前调用栈
Error Type: oneflow.ErrorProto.check_failed_error
Traceback (most recent call last):
File "/data/dataset/fengwen/miniconda3/envs/python3.8/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
obj = _ForkingPickler.dumps(obj)
File "/data/dataset/fengwen/miniconda3/envs/python3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
File "/data/dataset/fengwen/package/oneflow/python/oneflow/multiprocessing/reductions.py", line 87, in reduce_tensor
shm = shared_memory.SharedMemory(create=True, size=tensor_data.nbytes)
File "/data/dataset/fengwen/package/oneflow/python/oneflow/multiprocessing/shared_memory/__init__.py", line 28, in __init__
self.shm_ = flow._oneflow_internal.multiprocessing.SharedMemory(
oneflow._oneflow_internal.exception.Exception: Resource temporarily unavailable
File "/data/dataset/fengwen/package/oneflow/oneflow/core/ipc/shared_memory.cpp", line 175, in Open
ShmSetUp(&shm_name, shm_size, create)
File "/data/dataset/fengwen/package/oneflow/oneflow/core/ipc/shared_memory.cpp", line 70, in ShmSetUp
Error Type: oneflow.ErrorProto.check_failed_error
^C--Type <RET> for more, q to quit, c to continue without paging--
Thread 1 "python3" received signal SIGINT, Interrupt.
0x00007ffff7e4c678 in futex_abstimed_wait_cancelable (private=0, abstime=0x7fffffffbf00, clockid=0, expected=0, futex_word=0x55557d986b10)
at ../sysdeps/nptl/futex-internal.h:320
320 ../sysdeps/nptl/futex-internal.h: No such file or directory.
(gdb) py-bt
Undefined command: "py-bt". Try "help".
(gdb) bt
#0 0x00007ffff7e4c678 in futex_abstimed_wait_cancelable (private=0, abstime=0x7fffffffbf00, clockid=0, expected=0, futex_word=0x55557d986b10)
at ../sysdeps/nptl/futex-internal.h:320
#1 do_futex_wait (sem=sem@entry=0x55557d986b10, abstime=abstime@entry=0x7fffffffbf00, clockid=0) at sem_waitcommon.c:112
#2 0x00007ffff7e4c7a3 in __new_sem_wait_slow (sem=0x55557d986b10, abstime=0x7fffffffbf00, clockid=0) at sem_waitcommon.c:184
#3 0x000055555568650b in PyThread_acquire_lock_timed (lock=0x55557d986b10, microseconds=<optimized out>, intr_flag=1)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/thread_pthread.h:453
#4 0x00005555556b2d18 in acquire_timed (timeout=<optimized out>, lock=0x55557d986b10)
at /opt/conda/conda-bld/python-split_1648465063888/work/Modules/_threadmodule.c:63
#5 lock_PyThread_acquire_lock (self=0x7ffe33772cc0, args=<optimized out>, kwds=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Modules/_threadmodule.c:146
#6 0x000055555570dc03 in method_vectorcall_VARARGS_KEYWORDS (func=<optimized out>, args=0x7ffc24001408, nargsf=<optimized out>, kwnames=0x0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/descrobject.c:332
#7 0x000055555572ae60 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffc24001408, callable=0x7ffff7909c20)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#8 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#9 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3486
#10 0x0000555555720600 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffc24001270) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#11 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0,
kwargs=0x555578ee7348, kwcount=0, kwstep=1, defs=0x7ffff7735958, defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff78b8fb0, qualname=0x7ffff7738fb0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4298
#12 0x0000555555721b64 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x555578ee7338, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:436
#13 0x000055555572ae60 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x555578ee7338, callable=0x7ffff76a59d0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#14 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#15 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3486
#16 0x0000555555720600 in PyEval_EvalFrameEx (throwflag=0, f=0x555578ee7190) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#17 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>,
kwnames=0x7ffeac3dd088, kwargs=0x55557c19fe80, kwcount=1, kwstep=1, defs=0x7ffeac3e28d8, defcount=2, kwdefs=0x0, closure=0x0, name=0x7ffff7930d30,
qualname=0x7ffeac3e2530) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4298
#18 0x0000555555721bc4 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x55557c19fe78, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:436
#19 0x00005555556bab2e in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=0x55557c19fe78, callable=0x7ffeac3d8c10)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#20 method_vectorcall (method=<optimized out>, args=0x55557c19fe80, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/classobject.c:60
#21 0x000055555572beb0 in _PyObject_Vectorcall (kwnames=0x7ffeac3dd070, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffe4cdfcd00)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#22 call_function (kwnames=0x7ffeac3dd070, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
--Type <RET> for more, q to quit, c to continue without paging--
#23 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3515
#24 0x00005555557210ff in PyEval_EvalFrameEx (throwflag=0, f=0x55557c19fcb0) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#25 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0,
kwargs=0x55555803ccb0, kwcount=0, kwstep=1, defs=0x7ffeac350aa8, defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffeac3db7f0, qualname=0x7ffeac3da3f0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4298
#26 0x0000555555721bc4 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x55555803cca8, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:436
#27 0x000055555572ae60 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55555803cca8, callable=0x7ffeac3594c0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#28 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#29 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3486
#30 0x00005555557217e7 in PyEval_EvalFrameEx (throwflag=0, f=0x55555803cb20) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#31 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:284
#32 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:411
#33 0x000055555572ae60 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffe4d157d18, callable=0x7ffeac359550)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#34 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#35 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3486
#36 0x00005555557217e7 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffe4d157b80) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#37 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:284
#38 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:411
#39 0x000055555572ae60 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55557c344b68, callable=0x7ffeac3595e0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#40 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#41 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3486
#42 0x00005555557217e7 in PyEval_EvalFrameEx (throwflag=0, f=0x55557c3449e0) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#43 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:284
#44 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:411
#45 0x00005555556e8ad0 in _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7fffffffcd60, callable=0x7ffeac3590d0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#46 _PyObject_FastCall () at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:147
#47 _PyObject_FastCall_Prepend (nargs=1, args=0x0, obj=<optimized out>, callable=0x7ffeac3590d0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:850
--Type <RET> for more, q to quit, c to continue without paging--
#48 call_unbound (nargs=0, args=0x0, self=<optimized out>, func=0x7ffeac3590d0, unbound=1)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/typeobject.c:1453
#49 call_method (obj=<optimized out>, name=0x5555558bdf80 <PyId___next__.15195>, args=0x0, nargs=0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/typeobject.c:1485
#50 0x00005555556ebce4 in builtin_next (self=<optimized out>, args=0x55557e841e08, nargs=1)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/bltinmodule.c:1381
#51 0x000055555568f23d in cfunction_vectorcall_FASTCALL (func=0x7ffff78eea90, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/methodobject.c:426
#52 0x000055555572b0bb in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55557e841e08, callable=0x7ffff78eea90)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#53 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#54 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3500
#55 0x00005555556e950a in PyEval_EvalFrameEx (throwflag=<optimized out>, f=0x55557e841c80)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#56 gen_send_ex (gen=0x7ffe33836ba0, arg=<optimized out>, exc=<optimized out>, closing=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/genobject.c:222
#57 0x00005555556b5a53 in enum_next (en=0x7ffe33b1c980) at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/enumobject.c:162
#58 0x000055555572b474 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3202
#59 0x00005555556e950a in PyEval_EvalFrameEx (throwflag=<optimized out>, f=0x55557e7b6fc0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#60 gen_send_ex (gen=0x7ffe33836e40, arg=<optimized out>, exc=<optimized out>, closing=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/genobject.c:222
#61 0x000055555572b474 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3202
#62 0x00005555557210ff in PyEval_EvalFrameEx (throwflag=0, f=0x55555d915910) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#63 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0,
kwargs=0x55555cf913f0, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff76466b0, qualname=0x7ffff76466b0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4298
#64 0x0000555555721bc4 in _PyFunction_Vectorcall (func=<optimized out>, stack=0x55555cf913e0, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:436
#65 0x000055555572b0bb in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55555cf913e0, callable=0x7ffe6d769820)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#66 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#67 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3500
#68 0x00005555557217e7 in PyEval_EvalFrameEx (throwflag=0, f=0x55555cf91260) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#69 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:284
#70 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Objects/call.c:411
--Type <RET> for more, q to quit, c to continue without paging--
#71 0x000055555572b0bb in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff7860d50, callable=0x7ffe6d7698b0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Include/cpython/abstract.h:127
#72 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5555558f5620)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4963
#73 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:3500
#74 0x0000555555720600 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff7860be0) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:741
#75 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0,
kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4298
#76 0x0000555555721eb3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>,
globals=<optimized out>, _co=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:4327
#77 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/ceval.c:718
#78 0x0000555555796622 in run_eval_code_obj (co=0x7ffff77539d0, globals=0x7ffff78b0080, locals=0x7ffff78b0080)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/pythonrun.c:1166
#79 0x00005555557a71d2 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff78b0080, locals=0x7ffff78b0080, flags=<optimized out>,
arena=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/pythonrun.c:1188
#80 0x00005555557aa36b in pyrun_file (fp=0x5555558f3e80, filename=0x7ffff7894080, start=<optimized out>, globals=0x7ffff78b0080, locals=0x7ffff78b0080,
closeit=1, flags=0x7fffffffd8b8) at /opt/conda/conda-bld/python-split_1648465063888/work/Python/pythonrun.c:1085
#81 0x00005555557aa54f in pyrun_simple_file (flags=0x7fffffffd8b8, closeit=1, filename=0x7ffff7894080, fp=0x5555558f3e80)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/pythonrun.c:439
#82 PyRun_SimpleFileExFlags (fp=0x5555558f3e80, filename=<optimized out>, closeit=1, flags=0x7fffffffd8b8)
at /opt/conda/conda-bld/python-split_1648465063888/work/Python/pythonrun.c:472
#83 0x00005555557aaa29 in pymain_run_file (cf=0x7fffffffd8b8, config=0x5555558f4890) at /opt/conda/conda-bld/python-split_1648465063888/work/Modules/main.c:391
#84 pymain_run_python (exitcode=0x7fffffffd8b0) at /opt/conda/conda-bld/python-split_1648465063888/work/Modules/main.c:616
#85 Py_RunMain () at /opt/conda/conda-bld/python-split_1648465063888/work/Modules/main.c:695
#86 0x00005555557aac29 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /opt/conda/conda-bld/python-split_1648465063888/work/Modules/main.c:1127
#87 0x00007ffff7c6c083 in __libc_start_main (main=0x55555565bea0 <main>, argc=14, argv=0x7fffffffdab8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffdaa8) at ../csu/libc-start.c:308
#88 0x000055555574dad7 in _start ()
(gdb)
(gdb)