ZigZag Fatal error. System.AccessViolationException: Attempted to read or write protected memory.
vincehayek opened this issue · 8 comments
Hello,
I am using the ZigZag indicator exactly as described here
https://python.stockindicators.dev/indicators/ZigZag/#content
quotes = [Quote(d,o,h,l,c,v) for d,o,h,l,c,v in zip(df['date'], df['open'], df['high'], df['low'], df['close'],df['vol'])]
zigzag_objects = indicators.get_zig_zag(quotes, EndType.HIGH_LOW, change)
I am inputting a DataFrame and if it's too large (beyond 20K rows), I'm getting increasing chances of seeing the below error.
At 50K+ rows, I'm yet to see the soft to not crash.
This is the error from the Python terminal.
Fatal error. Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. at Python.Runtime.Runtime.PyObject_TYPE(Python.Runtime.BorrowedReference) at Python.Runtime.Runtime.PyObject_TYPE(Python.Runtime.BorrowedReference) at Python.Runtime.Runtime.NullGCHandles(System.Collections.Generic.IEnumerable1)
at Python.Runtime.Runtime.NullGCHandles(System.Collections.Generic.IEnumerable1<IntPtr>) at Python.Runtime.Runtime.TryCollectingGarbage(Int32, Boolean) at Python.Runtime.Runtime.TryCollectingGarbage(Int32, Boolean) at Python.Runtime.Runtime.Shutdown() at Python.Runtime.Runtime.Shutdown() at Python.Runtime.PythonEngine.Shutdown() at Python.Runtime.PythonEngine.Shutdown() at Python.Runtime.Loader.Shutdown(IntPtr, Int32) at Python.Runtime.Loader.Shutdown(IntPtr, Int32)
I'm using Python 3.10.7
It appears that you're exceeding the memory capacity of your computer; but it may also be something unusual with the way we're handling garbage collection. Could just be an inherent limitation of Pandas Dataframe.
@LeeDongGeon1996, have any thoughts here?
Thanks for your quick reply.
The input file I'm using is below 200Mo and while running the computer is below 20% usage of CPU and RAM.
So I don't think I'm exceeding system capacities.
As for garbage collection, I let Python handle it, I dont have anything specified.
I'll extract the file from the DB as csv, and prepare a stand alone example so I can share the code and input file with you.
I'll extract the file from the DB as csv, and prepare a stand-alone example to share the code and input file with you.
Your CSV quotes would be helpful for troubleshooting, thank you. Has this happened on other indicators or just this one?
Hurst Component would be another test to try.
I'm only using the ZigZag indicator
Here is the example
https://drive.google.com/file/d/1JbOgReaeYGl5-Z4D3GTyUvelJNBY3PmQ/view?usp=share_link
It's a 12 mega zip files containing the python code and the input CSV file containing several tickers (14 tickers, 1minute, 200 days history, 100 mega raw data).
The python file multithreads through the tickers and for each of them runs two different %change values.
It's a scenario where you want to run several change config to benchmark them.
In this file I'm not saving the result, so the file does nothing.
But for me it bugs, I get the Fatal Error.
I spent a few hours to isolate the problem.
- The change value is in play: I only get the error for higher values (here round 2.4)
- Multithreading seems to play a role. Sometimes a scenario that fails, can succeed if you reduce the number of treads.
But that should not happen.
I'm working a lot with multithreading with much more heavy tasks and input files, and never got this error.
I hope it helps!
Hi, @vincehayek . Thank you for providing samples.
I've tested your code several times and it works well. Every time, It took almost half an hour though.
(I used the same machine as the one for benchmark)
Are there any variables I should change for testing?
No it's the exact same one I'm using to reproduce.
The problem might come from my environment.
I will keep you updated if I discover anything.
Thanks for your time.
Keep the good work up!
Yeah, Thank you🙂
If you have any further questions, please reopen or create new issue any time.
Tip
For a working example using price quotes from a pandas.DataFrame, see our Replit of the Williams Fractal indicator. If you're having general trouble getting setup, see our QuickStart guide.
See also:
- #1165