This is a C++ analytical library that provides interface and functionality similar to packages/libraries in Python and R. For example, you could compare this to Pandas or R data.frame.
You could slice the data in many different ways. You could join, merge, group-by the data. You could run various statistical, summarization, financial, and ML algorithms on the data. You could add your custom algorithms easily. You could multi-column sort, custom pick and delete the data. And more …
DataFrame also includes a large collection of analytical algorithms in form of visitors -- see documentation. These are from basic stats such as Mean, Std Deviation, Return, … to more involved analysis such as Affinity Propagation, Polynomial Fit, Hurst Exponent … including a good collection of trading indicators -- See documentation for a complete list with code samples, and how you can add your custom algorithms.
I have followed a few principles in this library:
- Support any type either built-in or user defined without needing new code
- Never chase pointers ala
linked lists,std::any,pointer to base, ..., includingvirtual functions - Have all column data in continuous memory space. Also, be mindful of cache-line aliasing misses between multiple columns
- Never use more space than you need ala
unions,std::variant, ... - Avoid copying data as much as possible
- Use multi-threading but only when it makes sense
- Do not attempt to protect the user against
garbage in,garbage out
DateTime
DateTime class included in this library is a very cool and handy object to manipulate date/time with nanosecond precision and multi timezone capability.
There is a test program dataframe_performance that should give you a sense of how this library performs. As a comparison, there is also a Pandas pandas_performance script that does exactly the same thing.
dataframe_performance.cc uses DataFrame async interface and is compiled with gcc compiler with -O3 flag. pandas_performance.py is ran with Pandas 1.2 and Python 3.7. I ran both on my mac-book, doing the following:
- Generate ~1.6 billion timestamps (second resolution) and load them into the DataFrame/Pandas as index.
- Generate ~1.6 billion random numbers for 3 columns with normal, log normal, and exponential distributions and load them into the DataFrame/Pandas.
- Calculate the mean of each of the 3 columns.
Result:
MacBook> time python test/pandas_performance.py
All memory allocations are done. Calculating means ...
-4.318030613360519e-05, 1.6486147626420655, 1.0000042573786951
real 17m18.916s
user 4m47.113s
sys 5m31.901s
MacBook> time bin/Linux.GCC64/dataframe_performance
All memory allocations are done. Calculating means ...
1, 1.64879, 0.999994
real 4m15.786s
user 2m30.841s
sys 1m36.944sThe Interesting Part:
- Pandas script, I believe, is entirely implemented in Numpy which is in C.
- In case of Pandas, allocating memory + random number generation takes almost the same amount of time as calculating means.
- In case of DataFrame ~90% of the time is spent in allocating memory + random number generation.
- You load data once, but calculate statistics many times. So DataFrame, in general, is about ~21x faster than parts of Pandas that are implemented in Numpy. I leave parts of Pandas that are purely in Python to imagination.
- Pandas process image at its peak is ~105GB. C++ DataFrame process image at its peak is ~56GB.
DataFrame Test File
DataFrame Test File 2
Heterogeneous Vectors Test File
Date/Time Test File
mkdir [Debug | Release]
cd [Debug | Release]
cmake -DCMAKE_BUILD_TYPE=[Debug | Release] ..
make
make install
cd [Debug | Release]
make uninstallIf you are using Conan to manage your dependencies, add dataframe/x.y.z@ to your requires, where x.y.z is the release version you want to use. Conan will acquire DataFrame, build it from source in your computer, and provide CMake integration support for your projects. See the Conan docs for more information.
Sample conanfile.txt:
[requires]
dataframe/1.17.0@
[generators]
cmake