/static-frame

Immutable and grow-only Pandas-like DataFrames with a more explicit and consistent interface.

Primary LanguagePythonOtherNOASSERTION

https://raw.githubusercontent.com/InvestmentSystems/static-frame/master/doc/images/sf-logo-web_icon-small.png
https://img.shields.io/github/workflow/status/InvestmentSystems/static-frame/Test?label=test&logo=Github https://img.shields.io/github/workflow/status/InvestmentSystems/static-frame/TestForward?label=test-forward&logo=Github https://img.shields.io/github/workflow/status/InvestmentSystems/static-frame/Quality?label=quality&logo=Github

static-frame

A library of immutable and grow-only Pandas-like DataFrames with a more explicit and consistent interface. StaticFrame is suitable for applications in data science, data engineering, finance, scientific computing, and related fields where reducing opportunities for error by prohibiting in-place mutation is critical.

While many interfaces are similar to Pandas, StaticFrame deviates from Pandas in many ways: all data is immutable, and all indices are unique; the full range of NumPy data types is preserved, and date-time indices use discrete NumPy types; hierarchical indices are seamlessly integrated; and uniform approaches to element, row, and column iteration and function application are provided. Core StaticFrame depends only on NumPy: Pandas is not a dependency.

A wide variety of table storage and representation formats are supported, including input from and output to CSV, TSV, JSON, MessagePack, Excel XLSX, SQLite, HDF5, NumPy, Pandas, Arrow, and Parquet; additionally, output to xarray, HTML, RST, Markdown, and LaTeX is supported, as well as HTML representations in Jupyter notebooks.

StaticFrame features a family of multi-table containers: the Bus is lazily loaded container of tables, the Batch is a deferred processor of tables, and the Quilt is a virtual concatenation of tables. All permit operating on large collections of tables with minimal memory overhead, as well as writing too and reading from zipped bundles of pickles, Parquet, or delimited files, as well as XLSX workbooks, SQLite, and HDF5.

Code: https://github.com/InvestmentSystems/static-frame

Docs: http://static-frame.readthedocs.io

Packages: https://pypi.org/project/static-frame

Benchmarks: https://investmentsystems.github.io/static-frame-benchmark

Context: Ten Reasons to Use StaticFrame instead of Pandas

Why Immutable Data?

The following example, executed in a low-memory environment (using prlimit), shows how Pandas cannot re-label columns of a DataFrame or concatenate a DataFrame to itself without copying underlying data. By using immutable NumPy arrays, StaticFrame can perform these operations in the same low-memory environment. By reusing immutable arrays without copying, StaticFrame can achieve more efficient memory usage.

Colorful Types

Unexpected type coercions can expose errors or degrade performance. StaticFrame's container display provides full visibility into the types in a Frame, and provides a variety of ways to configure the presentation and color of those types.

Installation

Install StaticFrame via PIP:

pip install static-frame

Or, install StaticFrame via conda:

conda install -c conda-forge static-frame

To install full support of input and output routines via PIP:

pip install static-frame [extras]

Dependencies

Core StaticFrame requires the following:

  • Python >= 3.6
  • NumPy >= 1.16.5
  • automap >= 0.4.8

For extended input and output, the following packages are required:

  • pandas >= 0.23.4
  • xlsxwriter >= 1.1.2
  • openpyxl >= 3.0.0
  • xarray >= 0.13.0
  • tables >= 3.6.1
  • pyarrow >= 0.16.0

Quick-Start Guide

StaticFrame provides numerous methods for loading and creating data, either as a 1D Series or a 2D Frame. All creation routines are exposed as alternate constructors on the desired class, such as Frame.from_records(), Frame.from_csv() or Frame.from_pandas().

Note

For a concise overview of all StaticFrame interfaces, see API Overview.

For example, we can load JSON data from a URL using Frame.from_json_url(), and then use Frame.head() to reduce the displayed output to just the first five rows. (Passing explicit dtypes is only necessary on Windows.)

>>> import numpy as np
>>> import static_frame as sf
>>> frame = sf.Frame.from_json_url('https://jsonplaceholder.typicode.com/photos', dtypes=dict(albumId=np.int64, id=np.int64))
>>> frame.head()
<Frame>
<Index> albumId id      title                url                  thumbnailUrl         <<U12>
<Index>
0       1       1       accusamus beatae ... https://via.place... https://via.place...
1       1       2       reprehenderit est... https://via.place... https://via.place...
2       1       3       officia porro iur... https://via.place... https://via.place...
3       1       4       culpa odio esse r... https://via.place... https://via.place...
4       1       5       natus nisi omnis ... https://via.place... https://via.place...
<int64> <int64> <int64> <<U86>               <<U38>               <<U38>

Note

The Pandas CSV reader out-performs the NumPy-based reader in StaticFrame: thus, for now, using Frame.from_pandas(pd.read_csv(fp)) is recommended for loading large CSV files.

For more information on Frame constructors, see Frame: Constructor.

As with a NumPy array, the Frame exposes common attributes of shape and size.

>>> frame.shape
(5000, 5)
>>> frame.size
25000
>>> frame.nbytes
3320000

Unlike a NumPy array, a Frame stores heterogeneous types, where each column is a single type. StaticFrame preserves the full range of NumPy types, including fixed-size character strings. Character strings can be converted to Python objects or other types as needed with the Frame.astype interface, which exposes a __getitem__ style interface for selecting columns to convert. As with all similar functions, a new Frame is returned.

>>> frame.dtypes
<Series>
<Index>
albumId      int64
id           int64
title        <U86
url          <U38
thumbnailUrl <U38
<<U12>       <object>
>>> frame.astype['title':](object).dtypes
<Series>
<Index>
albumId      int64
id           int64
title        object
url          object
thumbnailUrl object
<<U12>       <object>

Utility functions common to Pandas users are available on Frame and Series, such as Series.unique(), Series.isna(), and Series.any().

>>> frame['albumId'].unique().tolist()
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
>>> frame['id'].isna().any()
False

Note

For more information on Frame utility functions, see Frame: Method.

StaticFrame interfaces for extracting data will be familiar to Pandas users, though with a number of interface refinements to remove redundancies and increase consistency. On a Frame, __getitem__ is (exclusively) a column selector; loc and iloc are (with one argument) row selectors or (with two arguments) row and column selectors.

For example we can select a single column with __getitem__:

>>> frame['albumId'].tail()
<Series: albumId>
<Index>
4995              100
4996              100
4997              100
4998              100
4999              100
<int64>           <int64>

Consistent with other __getitem__ style selectors, a slice or a list can be used to select columns:

>>> frame['id':'title'].head()
<Frame>
<Index> id      title                <<U12>
<Index>
0       1       accusamus beatae ...
1       2       reprehenderit est...
2       3       officia porro iur...
3       4       culpa odio esse r...
4       5       natus nisi omnis ...
<int64> <int64> <<U86>

The loc interface, with one argument, returns a Series for the row found at the given index label.

>>> frame.loc[4]
<Series: 4>
<Index>
albumId      1
id           5
title        natus nisi omnis ...
url          https://via.place...
thumbnailUrl https://via.place...
<<U12>       <object>

With two arguments, loc can select both rows and columns at the same time:

>>> frame.loc[4:8, ['albumId', 'title']]
<Frame>
<Index> albumId title                <<U12>
<Index>
4       1       natus nisi omnis ...
5       1       accusamus ea aliq...
6       1       officia delectus ...
7       1       aut porro officii...
8       1       qui eius qui aute...
<int64> <int64> <<U86>

Where the loc interface uses index and column labels, the iloc interface uses integer offsets from zero, just as if the Frame were a NumPy array. For example, we can select the last row with -1:

>>> frame.iloc[-1]
<Series: 4999>
<Index>
albumId        100
id             5000
title          error quasi sunt ...
url            https://via.place...
thumbnailUrl   https://via.place...
<<U12>         <object>

Or, using two arguments, we can select the first two columns of the last two rows:

>>> frame.iloc[-2:, 0:2]
<Frame>
<Index> albumId id      <<U12>
<Index>
4998    100     4999
4999    100     5000
<int64> <int64> <int64>

Just as with Pandas, expressions can be used in __getitem__, loc, and iloc statements to create more narrow selections. For example, we can select all "albumId" greater than or equal to 98.

>>> frame.loc[frame['albumId'] >= 98, ['albumId', 'title']].head()
<Frame>
<Index> albumId title                <<U12>
<Index>
4850    98      aut aut nulla vol...
4851    98      ducimus neque del...
4852    98      fugit officiis su...
4853    98      pariatur temporib...
4854    98      qui inventore inc...
<int64> <int64> <<U86>

However, unlike Pandas, __getitem__, loc, and iloc cannot be used for assignment or in-place mutation on a Frame or Series. Throughout StaticFrame, all underlying NumPy arrays, and all container attributes, are immutable. Making data and objects immutable reduces opportunities for coding errors and offers, in some situations, greater efficiency by avoiding defensive copies.

>>> frame.loc[4854, 'albumId']
98
>>> frame.loc[4854, 'albumId'] = 200
Traceback (most recent call last):
TypeError: 'InterfaceGetItem' object does not support item assignment
>>> frame.values[4854, 0] = 200
Traceback (most recent call last):
ValueError: assignment destination is read-only

Note

For more information on Frame selection interfaces, see Frame: Selector.

Instead of in-place assignment, an assign interface object (similar to the Frame.astype interface shown above) is provided to expose __getitem__, loc, and iloc interfaces that, when called with an argument, return a new object with the desired changes. These interfaces expose the full range of expressive assignment-like idioms found in Pandas and NumPy. Arguments can be single values, or Series and Frame objects, where assignment will align on the Index.

>>> frame_new = frame.assign.loc[4854, 'albumId'](200)
>>> frame_new.loc[4854, 'albumId']
200

This pattern of specialized interfaces is used throughout StaticFrame, such as with the Frame.mask and Frame.drop interfaces. For example, Frame.mask can be used to create a Boolean Frame that sets rows to True if their "id" is even:

>>> frame.mask.loc[frame['id'] % 2 == 0].head()
<Frame>
<Index> albumId id     title  url    thumbnailUrl <<U12>
<Index>
0       False   False  False  False  False
1       True    True   True   True   True
2       False   False  False  False  False
3       True    True   True   True   True
4       False   False  False  False  False
<int64> <bool>  <bool> <bool> <bool> <bool>

Or, using the Frame.drop interface, a new Frame can be created by dropping rows with even "id" values and dropping URL columns specified in a list:

>>> frame.drop.loc[frame['id'] % 2 == 0, ['thumbnailUrl', 'url']].head()
<Frame>
<Index> albumId id      title                <<U12>
<Index>
0       1       1       accusamus beatae ...
2       1       3       officia porro iur...
4       1       5       natus nisi omnis ...
6       1       7       officia delectus ...
8       1       9       qui eius qui aute...
<int64> <int64> <int64> <<U86>

Iteration of rows, columns, and elements, as well as function application on those values, is unified under a family of generator interfaces. These interfaces are distinguished by the form of the data iterated (Series, namedtuple, or array) and whether key-value pairs (e.g., Frame.iter_series_items()) or just values (e.g., Frame.iter_series()) are yielded. For example, we can iterate over each row of a Frame and yield a corresponding Series:

>>> next(iter(frame.iter_series(axis=1)))
<Series: 0>
<Index>
albumId      1
id           1
title        accusamus beatae ...
url          https://via.place...
thumbnailUrl https://via.place...
<<U12>       <object>

Or we can iterate over rows as named tuples, applying a function that matches a substring of the "title" or returns None, then drop those None records:

>>> frame.iter_tuple(axis=1).apply(lambda r: r.title if 'voluptatem' in r.title else None).dropna().head()
<Series>
<Index>
19       assumenda volupta...
27       non neque eligend...
29       odio enim volupta...
31       ad enim dignissim...
40       in voluptatem dol...
<int64>  <object>

Element iteration and function application works the same way as for rows or columns (though without an axis argument). For example, here each URL is processed with the same string transformation function:

>>> frame[['thumbnailUrl', 'url']].iter_element().apply(lambda c: c.replace('https://', '')).iloc[-4:]
<Frame>
<Index> thumbnailUrl         url                  <<U12>
<Index>
4996    via.placeholder.c... via.placeholder.c...
4997    via.placeholder.c... via.placeholder.c...
4998    via.placeholder.c... via.placeholder.c...
4999    via.placeholder.c... via.placeholder.c...
<int64> <<U30>               <<U30>

Group-by functionality is exposed in a similar manner with Frame.iter_group_items() and Frame.iter_group().

>>> next(iter(frame.iter_group('albumId', axis=0))).shape
(50, 5)

Function application to a group Frame can be used to produce a Series indexed by the group label. For example, a Series, indexed by "albumId", can be produced to show the number of unique titles found per album.

>>> frame.iter_group('albumId', axis=0).apply(lambda g: len(g['title'].unique()), dtype=np.int64).head()
<Series>
<Index>
1        50
2        50
3        50
4        50
5        50
<int64>  <int64>

Note

For more information on Frame iterators and tools for function application, see Frame: Iterator.

If performing calculations on a Frame that result in a Series with a compatible Index, a grow-only FrameGO can be used to add Series as new columns. This limited form of mutation, i.e., only the addition of columns, provides a convenient compromise between mutability and immutability. (Underlying NumPy array data always remains immutable.)

A FrameGO can be efficiently created from a Frame, as underling NumPy arrays do not have to be copied:

>>> frame_go = frame.to_frame_go()

We can obtain a track number within each album, assuming the records are sorted, by creating the following generator expression pipe-line. Using a Frame grouped by "albumId", zip together as pairs the Frame.index and a contiguous integer sequence via range(); chain all of those iterables, and then pass the resulting generator to Series.from_items(). (As much as possible, StaticFrame supports generators as arguments wherever an ordered sequence is expected.)

>>> from itertools import chain
>>> index_to_track = chain.from_iterable(zip(g.index, range(len(g))) for g in frame_go.iter_group('albumId'))
>>> frame_go['track'] = sf.Series.from_items(index_to_track, dtype=np.int64) + 1
>>> frame_go.iloc[45:55]
<FrameGO>
<IndexGO> albumId id      title                url                  thumbnailUrl         track   <<U12>
<Index>
45        1       46      quidem maiores in... https://via.place... https://via.place... 46
46        1       47      et soluta est        https://via.place... https://via.place... 47
47        1       48      ut esse id           https://via.place... https://via.place... 48
48        1       49      quasi quae est mo... https://via.place... https://via.place... 49
49        1       50      et inventore quae... https://via.place... https://via.place... 50
50        2       51      non sunt voluptat... https://via.place... https://via.place... 1
51        2       52      eveniet pariatur ... https://via.place... https://via.place... 2
52        2       53      soluta et harum a... https://via.place... https://via.place... 3
53        2       54      ut ex quibusdam d... https://via.place... https://via.place... 4
54        2       55      voluptatem conseq... https://via.place... https://via.place... 5
<int64>   <int64> <int64> <<U86>               <<U38>               <<U38>               <int64>

Unlike with Pandas, StaticFrame Index objects always enforce uniqueness (there is no "verify_integrity" option: integrity is never optional). Thus, an index can never be set from non-unique data:

>>> frame_go.set_index('albumId')
Traceback (most recent call last):
static_frame.core.exception.ErrorInitIndexNonUnique: labels (5000) have non-unique values (100)

For a data set such as the one used in this example, a hierarchical index, by "albumId" and "track", is practical. StaticFrame implements hierarchical indices as IndexHierarchy objects. The Frame.set_index_hierarchy() method, given columns in a Frame, can be used to create a hierarchical index:

>>> frame_h = frame_go.set_index_hierarchy(['albumId', 'track'], drop=True)
>>> frame_h.head()
<FrameGO>
<IndexGO>                                    id      title                url                  thumbnailUrl         <<U12>
<IndexHierarchy: ('albumId', 'tra...
1                                    1       1       accusamus beatae ... https://via.place... https://via.place...
1                                    2       2       reprehenderit est... https://via.place... https://via.place...
1                                    3       3       officia porro iur... https://via.place... https://via.place...
1                                    4       4       culpa odio esse r... https://via.place... https://via.place...
1                                    5       5       natus nisi omnis ... https://via.place... https://via.place...
<int64>                              <int64> <int64> <<U86>               <<U38>               <<U38>

Hierarchical indices permit specifying selectors, per axis, at each hierarchical level. To distinguish hierarchical levels from axis arguments in a loc expression, the HLoc wrapper, exposing a __getitem__ interface, can be used. For example, we can select, from all albums, the second and fifth track, and then only the "title" and "url" columns.

>>> frame_h.loc[sf.HLoc[:, [2,5]], ['title', 'url']].head()
<FrameGO>
<IndexGO>                                    title                url                  <<U12>
<IndexHierarchy: ('albumId', 'tra...
1                                    2       reprehenderit est... https://via.place...
1                                    5       natus nisi omnis ... https://via.place...
2                                    2       eveniet pariatur ... https://via.place...
2                                    5       voluptatem conseq... https://via.place...
3                                    2       eaque iste corpor... https://via.place...
<int64>                              <int64> <<U86>               <<U38>

Just as a hierarchical selection can reside in a loc expression with an HLoc wrapper, an integer index selection can reside in a loc expression with an ILoc wrapper. For example, the previous row selection is combined with the selection of the last column:

>>> frame_h.loc[sf.HLoc[:, [2,5]], sf.ILoc[-1]].head()
<Series: thumbnailUrl>
<IndexHierarchy: ('albumId', 'tra...
1                                    2       https://via.place...
1                                    5       https://via.place...
2                                    2       https://via.place...
2                                    5       https://via.place...
3                                    2       https://via.place...
<int64>                              <int64> <<U38>

Note

For more information on IndexHierarchy, see Index Hierarchy.

While StaticFrame offers many of the features of Pandas and similar data structures, exporting directly to NumPy arrays (via the .values attribute) or to Pandas is supported for functionality not found in StaticFrame or compatibility with other libraries. For example, a Frame can export to a Pandas DataFrame with Frame.to_pandas().

>>> df = frame_go.to_pandas()