basic readme example failing with errors with latest torch nightlies
chauhang opened this issue · 2 comments
chauhang commented
Not to run the basic examples described in the readme -- it is failing with the latest nightlies torch==2.3.0.dev20240127+cu118
Output with errors:
TORCH_LOGS="+dynamo" python compile_depf.py
/opt/conda/envs/nllm3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/opt/conda/envs/nllm3/lib/python3.10/site-packages/depyf/explain/enable_debugging.py:153: UserWarning: You are trying to debug `torch.compile`. Please make sure the code runs multiple times to cover all the possible branches.
warnings.warn((
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing toy_example /home/ubuntu/compile/compile_depf.py:5
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] Stack (most recent call last):
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/home/ubuntu/compile/compile_depf.py", line 20, in <module>
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] main()
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/home/ubuntu/compile/compile_depf.py", line 14, in main
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] toy_example(torch.randn(10), torch.randn(10))
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 443, in _fn
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] return fn(*args, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 605, in catch_errors
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] return callback(frame, cache_entry, hooks, frame_state)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 748, in _convert_frame
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] result = inner_convert(frame, cache_entry, hooks, frame_state)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 390, in _convert_frame_assert
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] return _compile(
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/contextlib.py", line 79, in inner
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] return func(*args, **kwds)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 650, in _compile
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] guarded_code = compile_inner(code, one_graph, hooks, transform)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 248, in time_wrapper
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] r = func(*args, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 531, in compile_inner
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] out_code = transform_code_object(code, transform)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] transformations(instructions, code_options)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 155, in _fn
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] return fn(*args, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 478, in transform
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] tracer = InstructionTranslator(
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2032, in __init__
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] _step_logger()(
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/logging.py", line 55, in log
[2024-01-28 01:08:50,002] [0/0] torch._dynamo.symbolic_convert: [INFO] logger.log(level, "Step %s: %s", step, msg, **kwargs)
[2024-01-28 01:08:50,002] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-28 01:08:50,003] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/ubuntu/compile/compile_depf.py:5 in toy_example ()
[2024-01-28 01:08:50,003] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] @torch.compile
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/ubuntu/compile/compile_depf.py:7 in toy_example (toy_example)
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] x = a / (torch.abs(a) + 1)
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST a []
[2024-01-28 01:08:50,008] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [LazyVariableTracker()]
[2024-01-28 01:08:50,051] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR abs [LazyVariableTracker(), PythonModuleVariable(<module 'torch' from '/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/__init__.py'>)]
[2024-01-28 01:08:50,052] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST a [LazyVariableTracker(), TorchInGraphFunctionVariable(<built-in method abs of type object at 0x7f7f874c6000>)]
[2024-01-28 01:08:50,052] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [LazyVariableTracker(), TorchInGraphFunctionVariable(<built-in method abs of type object at 0x7f7f874c6000>), LazyVariableTracker()]
[2024-01-28 01:08:50,052] [0/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_a_ L['a']
[2024-01-28 01:08:50,053] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['a'] (10,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], constraint_sizes=[None], tensor_source=LocalSource(local_name='a', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
[2024-01-28 01:08:50,055] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), TensorVariable()]
[2024-01-28 01:08:50,055] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_ADD None [TensorVariable(), TensorVariable(), ConstantVariable(int)]
[2024-01-28 01:08:50,057] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_TRUE_DIVIDE None [TensorVariable(), TensorVariable()]
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/ubuntu/compile/compile_depf.py:8 in toy_example (toy_example)
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] if b.sum() < 0:
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST b []
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sum [LazyVariableTracker()]
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_b_ L['b']
[2024-01-28 01:08:50,058] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['b'] (10,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], constraint_sizes=[None], tensor_source=LocalSource(local_name='b', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
[2024-01-28 01:08:50,059] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), sum)]
[2024-01-28 01:08:50,060] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 0 [TensorVariable()]
[2024-01-28 01:08:50,060] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE COMPARE_OP < [TensorVariable(), ConstantVariable(int)]
[2024-01-28 01:08:50,061] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 38 [TensorVariable()]
[2024-01-28 01:08:50,062] [0/0] torch._dynamo.symbolic_convert: [DEBUG] generic_jump triggered compile
[2024-01-28 01:08:50,062] [0/0] torch._dynamo.output_graph: [DEBUG] COMPILING GRAPH due to GraphCompileReason(reason='generic_jump TensorVariable()', user_stack=[<FrameSummary file /home/ubuntu/compile/compile_depf.py, line 8 in toy_example>], graph_break=True)
Traceback (most recent call last):
File "/home/ubuntu/compile/compile_depf.py", line 20, in <module>
main()
File "/home/ubuntu/compile/compile_depf.py", line 14, in main
toy_example(torch.randn(10), torch.randn(10))
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 443, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 605, in catch_errors
return callback(frame, cache_entry, hooks, frame_state)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 748, in _convert_frame
result = inner_convert(frame, cache_entry, hooks, frame_state)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 390, in _convert_frame_assert
return _compile(
File "/opt/conda/envs/nllm3/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 677, in _compile
raise InternalTorchDynamoError(str(e)).with_traceback(
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 650, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 248, in time_wrapper
r = func(*args, **kwargs)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 531, in compile_inner
out_code = transform_code_object(code, transform)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
transformations(instructions, code_options)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 155, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 496, in transform
tracer.run()
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2125, in run
super().run()
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 787, in run
and self.step()
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 750, in step
getattr(self, inst.opname)(inst)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 379, in inner
self.output.compile_subgraph(
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 918, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/opt/conda/envs/nllm3/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1071, in compile_and_call_fx_graph
graph_code_log.debug("%s", lazy_format_graph_code(name, gm))
File "/opt/conda/envs/nllm3/lib/python3.10/site-packages/depyf/explain/patched_lazy_format_graph_code.py", line 34, in patched_lazy_format_graph_code
fn.__code__ = scope[fn.__name__].__code__
torch._dynamo.exc.InternalTorchDynamoError: _lazy_forward
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] Function Runtimes (s)
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] ------------------------------- --------------
[2024-01-28 01:08:50,099] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner 0
youkaichao commented
@chauhang Hey, thanks for your interest. Can you try the latest code? It should work now. The problem you mentioned should be caused by LazyGraphModule introduced several days ago.
youkaichao commented
Actually that PR pytorch/pytorch#117911 was just merged yesterday. So thank you @chauhang for pointing out the issue so timely :)