errors while replicating
vadimtk opened this issue · 1 comments
vadimtk commented
I am trying to replication the table from MySQL
CREATE TABLE `acct` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`stamp_inserted` datetime NOT NULL,
`ip_src` char(45) NOT NULL,
`ip_dst` char(45) NOT NULL,
`src_port` int(2) unsigned NOT NULL,
`dst_port` int(2) unsigned NOT NULL,
`in_bytes` int(10) unsigned NOT NULL,
`out_bytes` int(10) unsigned NOT NULL,
`src_host` varchar(1024) DEFAULT NULL,
`dst_host` varchar(1024) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1391121 DEFAULT CHARSET=latin1
to a ClickHouse table
CREATE TABLE default.acct ( id UInt32, stamp_inserted DateTime, ip_src String, ip_dst String, src_port UInt16, dst_port UInt16, in_bytes UInt32, out_bytes UInt32, src_host String, dst_host String) ENGINE = MergeTree PARTITION BY toMonday(stamp_inserted) ORDER BY (stamp_inserted, ip_dst) SETTINGS index_granularity = 8192
my script is
python3 -m clickhouse_mysql.main \
--src-resume \
--src-wait \
--nice-pause=1 \
--log-level=info \
--log-file=acct.log \
--src-host=127.0.0.1 \
--src-user=reader \
--src-password=qwerty \
--dst-host=127.0.0.1 \
--csvpool \
--csvpool-file-path-prefix=qwe_ \
--mempool-max-flush-interval=60 \
--mempool-max-events-num=10000
while it is running and the events are coming to MySQL, the script fails as:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/processwriter.py", line 30, in process
writer.insert(event_or_events)
File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/csvwriter.py", line 83, in insert
self.fieldnames = sorted(events[0].column_names())
File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/event/event.py", line 53, in column_names
return self.mysql_event.rows[0]['values'].keys()
IndexError: list index out of range
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/processwriter.py", line 30, in process
writer.insert(event_or_events)
File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/csvwriter.py", line 83, in insert
self.fieldnames = sorted(events[0].column_names())
File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/event/event.py", line 53, in column_names
return self.mysql_event.rows[0]['values'].keys()
IndexError: list index out of range
the log is
2017-12-25 13:15:31,936/1514236531.936720:INFO:Starting
2017-12-25 13:15:31,940/1514236531.940504:INFO:raw dbs list len=0
2017-12-25 13:15:31,941/1514236531.941264:INFO:dbs list len=0
2017-12-25 13:15:31,941/1514236531.941383:INFO:raw tables list len=0
2017-12-25 13:15:31,941/1514236531.941494:INFO:tables list len=0
2017-12-25 13:16:24,682/1514236584.682820:INFO:rot now:1514236584.682676 bktttl:1 bktitemsttl: 13 index:t1.acct reason:FLUSH bktsonbelt:2 bktsize:13 beltnum:1
2017-12-25 13:16:44,976/1514236604.976741:INFO:rot now:1514236604.976535 bktttl:2 bktitemsttl: 52 index:t1.acct reason:FLUSH bktsonbelt:2 bktsize:39 beltnum:1
2017-12-25 13:16:44,980/1514236604.980124:INFO:PERF - buckets_per_sec:0.049276 items_per_sec:1.921764 for last 20 sec
sunsingerus commented
New version is published. Please, take a look