Impossible to asignate datetime index
Opened this issue · 0 comments
Arctic Version
Last
Arctic version here
#### Arctic Store
Local
# VersionStore, TickStore, or ChunkStore
Platform and version
Fedora 37
Description of problem and/or code sample that reproduces the issue
files = [f for f in os.listdir('dbns') if f.endswith('.dbn')]
lib = db.get_library('HE')
for f in range(len(files)):
print(files[f],end='\r')
df = dbn.from_dbn('dbns/'+files[f]).to_df(True,True)
symbols = df['symbol'].unique().tolist()
for ticker in symbols:
temp = df[df['symbol'] == ticker]
temp['side'] = temp['side'].apply(lambda x: transforms.side_transform(x))
temp['action'] = temp['action'].apply(lambda x: transforms.action_transform(x))
out= temp.reset_index().select_dtypes(exclude=['object'])
print(out.info())
if lib.has_symbol(ticker):
lib.update(ticker,out)
else:
lib.write(ticker,out)
[2023-03-31 00:12:19.128] [arcticdb] [error] Could not normalize item of type: <class 'pandas.core.frame.DataFrame'> with any normalizer.You can set pickle_on_failure param to force pickling of this object instead.(Note: Pickling has worse performance and stricter memory limitations)
[2023-03-31 00:12:19.132] [arcticdb] [error] Error while normalizing symbol=HE:BF G9-J9-K9, data= ts_recv
idx
2019-01-06 20:00:07.624950223+00:00 2019-01-06 20:00:07.885202870+00:00
2019-01-06 20:00:07.624950223+00:00 2019-01-07 00:00:00+00:00
2019-01-07 12:00:00.428979457+00:00 2019-01-07 12:00:00.429096613+00:00
2019-01-07 12:00:01.043700621+00:00 2019-01-07 12:00:01.043822277+00:00
2019-01-07 12:00:01.143670097+00:00 2019-01-07 12:00:01.143790281+00:00
... ...
2019-01-18 18:58:47.737260751+00:00 2019-01-18 18:58:47.737411699+00:00
2019-01-18 18:59:36.353778773+00:00 2019-01-18 18:59:36.353919621+00:00
2019-01-18 19:01:59.605600119+00:00 2019-01-18 19:01:59.605718999+00:00
2019-01-18 19:05:00.072278335+00:00 2019-01-18 19:05:00.093682481+00:00
2019-01-20 20:00:22.600106337+00:00 2019-01-20 20:00:22.889622402+00:00
ts_event \
idx
2019-01-06 20:00:07.624950223+00:00 2019-01-06 20:00:07.624950223+00:00
2019-01-06 20:00:07.624950223+00:00 2019-01-06 20:00:07.624950223+00:00
2019-01-07 12:00:00.428979457+00:00 2019-01-07 12:00:00.428979457+00:00
2019-01-07 12:00:01.043700621+00:00 2019-01-07 12:00:01.043700621+00:00
2019-01-07 12:00:01.143670097+00:00 2019-01-07 12:00:01.143670097+00:00
... ...
2019-01-18 18:58:47.737260751+00:00 2019-01-18 18:58:47.737260751+00:00
2019-01-18 18:59:36.353778773+00:00 2019-01-18 18:59:36.353778773+00:00
2019-01-18 19:01:59.605600119+00:00 2019-01-18 19:01:59.605600119+00:00
2019-01-18 19:05:00.072278335+00:00 2019-01-18 19:05:00.072278335+00:00
2019-01-20 20:00:22.600106337+00:00 2019-01-20 20:00:22.600106337+00:00
ts_in_delta publisher_id product_id \
idx
2019-01-06 20:00:07.624950223+00:00 20634 1 7673
2019-01-06 20:00:07.624950223+00:00 0 1 7673
2019-01-07 12:00:00.428979457+00:00 16649 1 7673
2019-01-07 12:00:01.043700621+00:00 16927 1 7673
2019-01-07 12:00:01.143670097+00:00 16585 1 7673
... ... ... ...
2019-01-18 18:58:47.737260751+00:00 18692 1 7673
2019-01-18 18:59:36.353778773+00:00 17050 1 7673
2019-01-18 19:01:59.605600119+00:00 16469 1 7673
2019-01-18 19:05:00.072278335+00:00 39177 1 7673
2019-01-20 20:00:22.600106337+00:00 20148 1 7673
action side depth flags price ... \
idx ...
2019-01-06 20:00:07.624950223+00:00 1 2 0 128 0.7 ...
2019-01-06 20:00:07.624950223+00:00 1 2 0 168 0.0 ...
2019-01-07 12:00:00.428979457+00:00 1 2 9 128 -1.1 ...
2019-01-07 12:00:01.043700621+00:00 1 2 4 128 0.3 ...
2019-01-07 12:00:01.143670097+00:00 1 2 7 128 -0.3 ...
... ... ... ... ... ... ...
2019-01-18 18:58:47.737260751+00:00 4 2 6 128 0.0 ...
2019-01-18 18:59:36.353778773+00:00 1 2 0 128 0.5 ...
2019-01-18 19:01:59.605600119+00:00 1 2 6 128 0.1 ...
2019-01-18 19:05:00.072278335+00:00 3 1 0 128 1.5 ...
2019-01-20 20:00:22.600106337+00:00 1 1 0 128 2.9 ...
bid_sz_08 ask_sz_08 bid_oq_08 \
idx
2019-01-06 20:00:07.624950223+00:00 1 0 1
2019-01-06 20:00:07.624950223+00:00 1 0 1
2019-01-07 12:00:00.428979457+00:00 1 0 1
2019-01-07 12:00:01.043700621+00:00 1 0 1
2019-01-07 12:00:01.143670097+00:00 1 0 1
... ... ... ...
2019-01-18 18:58:47.737260751+00:00 1 0 1
2019-01-18 18:59:36.353778773+00:00 1 0 1
2019-01-18 19:01:59.605600119+00:00 1 0 1
2019-01-18 19:05:00.072278335+00:00 0 0 0
2019-01-20 20:00:22.600106337+00:00 0 0 0
ask_oq_08 bid_px_09 ask_px_09 \
idx
2019-01-06 20:00:07.624950223+00:00 0 -1.375000e+00 9.223372e+09
2019-01-06 20:00:07.624950223+00:00 0 -1.375000e+00 9.223372e+09
2019-01-07 12:00:00.428979457+00:00 0 -1.100000e+00 9.223372e+09
2019-01-07 12:00:01.043700621+00:00 0 -1.025000e+00 9.223372e+09
2019-01-07 12:00:01.143670097+00:00 0 -9.250000e-01 9.223372e+09
... ... ... ...
2019-01-18 18:58:47.737260751+00:00 0 -6.500000e-01 9.223372e+09
2019-01-18 18:59:36.353778773+00:00 0 -3.000000e-01 9.223372e+09
2019-01-18 19:01:59.605600119+00:00 0 -3.000000e-01 9.223372e+09
2019-01-18 19:05:00.072278335+00:00 0 9.223372e+09 9.223372e+09
2019-01-20 20:00:22.600106337+00:00 0 9.223372e+09 9.223372e+09
bid_sz_09 ask_sz_09 bid_oq_09 \
idx
2019-01-06 20:00:07.624950223+00:00 1 0 1
2019-01-06 20:00:07.624950223+00:00 1 0 1
2019-01-07 12:00:00.428979457+00:00 1 0 1
2019-01-07 12:00:01.043700621+00:00 1 0 1
2019-01-07 12:00:01.143670097+00:00 1 0 1
... ... ... ...
2019-01-18 18:58:47.737260751+00:00 1 0 1
2019-01-18 18:59:36.353778773+00:00 1 0 1
2019-01-18 19:01:59.605600119+00:00 1 0 1
2019-01-18 19:05:00.072278335+00:00 0 0 0
2019-01-20 20:00:22.600106337+00:00 0 0 0
ask_oq_09
idx
2019-01-06 20:00:07.624950223+00:00 0
2019-01-06 20:00:07.624950223+00:00 0
2019-01-07 12:00:00.428979457+00:00 0
2019-01-07 12:00:01.043700621+00:00 0
2019-01-07 12:00:01.143670097+00:00 0
... ...
2019-01-18 18:58:47.737260751+00:00 0
2019-01-18 18:59:36.353778773+00:00 0
2019-01-18 19:01:59.605600119+00:00 0
2019-01-18 19:05:00.072278335+00:00 0
2019-01-20 20:00:22.600106337+00:00 0
[525 rows x 72 columns], metadata=None, bad argument type for built-in operation
TypeError Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_store.py:333, in NativeVersionStore._try_normalize(self, symbol, dataframe, metadata, pickle_on_failure, dynamic_strings, coerce_columns, **kwargs)
331 else:
332 # TODO: just for pandas dataframes for now.
--> 333 item, norm_meta = self._normalizer.normalize(
334 dataframe,
335 pickle_on_failure=pickle_on_failure,
336 dynamic_strings=dynamic_strings,
337 coerce_columns=coerce_columns,
338 dynamic_schema=dynamic_schema,
339 **kwargs,
340 )
341 except ArcticNativeNotYetImplemented as ex:
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_normalization.py:1134, in CompositeNormalizer.normalize(self, item, string_max_len, pickle_on_failure, dynamic_strings, coerce_columns, **kwargs)
1133 try:
-> 1134 return self._normalize(
1135 item,
1136 string_max_len=string_max_len,
1137 dynamic_strings=dynamic_strings,
1138 coerce_columns=coerce_columns,
1139 **kwargs,
1140 )
1141 except Exception as ex:
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_normalization.py:1081, in CompositeNormalizer._normalize(self, item, string_max_len, dynamic_strings, coerce_columns, **kwargs)
1080 log.debug("Normalizer used: {}".format(normalizer))
-> 1081 return normalizer(
1082 item,
1083 string_max_len=string_max_len,
1084 dynamic_strings=dynamic_strings,
1085 coerce_columns=coerce_columns,
1086 **kwargs,
1087 )
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_normalization.py:755, in DataFrameNormalizer.normalize(self, item, string_max_len, dynamic_strings, coerce_columns, **kwargs)
753 raise ArcticNativeNotYetImplemented("MultiIndex column are not supported yet")
--> 755 index_names, ix_vals = self._index_to_records(
756 item, norm_meta.df.common, dynamic_strings, string_max_len=string_max_len
757 )
758 # The first branch of this if is faster, but does not work with null/duplicated column names
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_normalization.py:494, in _PandasNormalizer._index_to_records(self, df, pd_norm, dynamic_strings, string_max_len)
492 index_norm.is_not_range_index = not isinstance(index, RangeIndex)
--> 494 return _normalize_single_index(index, list(index.names), index_norm, dynamic_strings, string_max_len)
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_normalization.py:311, in _normalize_single_index(index, index_names, index_norm, dynamic_strings, string_max_len)
310 if index_tz is not None:
--> 311 index_norm.tz = index_tz
313 return index_names, ix_vals
TypeError: bad argument type for built-in operation
During handling of the above exception, another exception occurred:
ArcticNativeException Traceback (most recent call last)
Cell In[12], line 49
47 out = out.set_index('idx')
48 if lib.has_symbol(ticker):
---> 49 lib.update(ticker,out)
50 else:
51 lib.write(ticker,out)
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/library.py:670, in Library.update(self, symbol, data, metadata, upsert, date_range, prune_previous_versions)
603 def update(
604 self,
605 symbol: str,
(...)
610 prune_previous_versions=False,
611 ) -> VersionedItem:
612 """
613 Overwrites existing symbol data with the contents of data. The entire range between the first and last index
614 entry in data is replaced in its entirety with the contents of data, adding additional index entries if
(...)
668 2018-01-04 4
669 """
--> 670 return self._nvs.update(
671 symbol=symbol,
672 data=data,
673 metadata=metadata,
674 upsert=upsert,
675 date_range=date_range,
676 prune_previous_version=prune_previous_versions,
677 )
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_store.py:743, in NativeVersionStore.update(self, symbol, data, metadata, date_range, upsert, prune_previous_version, **kwargs)
739 data = restrict_data_to_date_range_only(data, start=start, end=end)
741 _handle_categorical_columns(symbol, data)
--> 743 udm, item, norm_meta = self._try_normalize(symbol, data, metadata, False, dynamic_strings, coerce_columns)
745 if isinstance(item, NPDDataFrame):
746 with _diff_long_stream_descriptor_mismatch(self):
File /opt/conda/lib/python3.10/site-packages/arcticdb/version_store/_store.py:346, in NativeVersionStore._try_normalize(self, symbol, dataframe, metadata, pickle_on_failure, dynamic_strings, coerce_columns, **kwargs)
344 except Exception as ex:
345 log.error("Error while normalizing symbol={}, data={}, metadata={}, {}", symbol, dataframe, metadata, ex)
--> 346 raise ArcticNativeException(str(ex))
348 if norm_meta is None:
349 raise ArcticNativeException("Cannot normalize input {}".format(symbol))
ArcticNativeException: bad argument type for built-in operation
- My data:
Column Non-Null Count Dtype
0 ts_recv 6 non-null datetime64[ns, UTC]
1 ts_event 6 non-null datetime64[ns, UTC]
2 ts_in_delta 6 non-null int32
3 publisher_id 6 non-null uint16
4 product_id 6 non-null uint32
5 action 6 non-null int64
6 side 6 non-null int64
7 depth 6 non-null uint8
8 flags 6 non-null int16
9 price 6 non-null float64
10 size 6 non-null uint32
11 sequence 6 non-null uint32
12 bid_px_00 6 non-null float64
13 ask_px_00 6 non-null float64
14 bid_sz_00 6 non-null uint32
15 ask_sz_00 6 non-null uint32
16 bid_oq_00 6 non-null uint32
17 ask_oq_00 6 non-null uint32
18 bid_px_01 6 non-null float64
19 ask_px_01 6 non-null float64
20 bid_sz_01 6 non-null uint32
21 ask_sz_01 6 non-null uint32
22 bid_oq_01 6 non-null uint32
23 ask_oq_01 6 non-null uint32
24 bid_px_02 6 non-null float64
25 ask_px_02 6 non-null float64
26 bid_sz_02 6 non-null uint32
27 ask_sz_02 6 non-null uint32
28 bid_oq_02 6 non-null uint32
29 ask_oq_02 6 non-null uint32
30 bid_px_03 6 non-null float64
31 ask_px_03 6 non-null float64
32 bid_sz_03 6 non-null uint32
33 ask_sz_03 6 non-null uint32
34 bid_oq_03 6 non-null uint32
35 ask_oq_03 6 non-null uint32
36 bid_px_04 6 non-null float64
37 ask_px_04 6 non-null float64
38 bid_sz_04 6 non-null uint32
39 ask_sz_04 6 non-null uint32
40 bid_oq_04 6 non-null uint32
41 ask_oq_04 6 non-null uint32
42 bid_px_05 6 non-null float64
43 ask_px_05 6 non-null float64
44 bid_sz_05 6 non-null uint32
45 ask_sz_05 6 non-null uint32
46 bid_oq_05 6 non-null uint32
47 ask_oq_05 6 non-null uint32
48 bid_px_06 6 non-null float64
49 ask_px_06 6 non-null float64
50 bid_sz_06 6 non-null uint32
51 ask_sz_06 6 non-null uint32
52 bid_oq_06 6 non-null uint32
53 ask_oq_06 6 non-null uint32
54 bid_px_07 6 non-null float64
55 ask_px_07 6 non-null float64
56 bid_sz_07 6 non-null uint32
57 ask_sz_07 6 non-null uint32
58 bid_oq_07 6 non-null uint32
59 ask_oq_07 6 non-null uint32
60 bid_px_08 6 non-null float64
61 ask_px_08 6 non-null float64
62 bid_sz_08 6 non-null uint32
63 ask_sz_08 6 non-null uint32
64 bid_oq_08 6 non-null uint32
65 ask_oq_08 6 non-null uint32
66 bid_px_09 6 non-null float64
67 ask_px_09 6 non-null float64
68 bid_sz_09 6 non-null uint32
69 ask_sz_09 6 non-null uint32
70 bid_oq_09 6 non-null uint32
71 ask_oq_09 6 non-null uint32
dtypes: datetime64ns, UTC, float64(21), int16(1), int32(1), int64(2), uint16(1), uint32(43), uint8(1)
memory usage: 2.4 KB
- Description : I Try to load dataframes into artic, and impossible to with timestamp index. (For later apply futures updates).
- I can do, if store in a column de date, and later after read, set index, but cant update easy.
Any solution to this fix? Thanks for all!