actiontech/dtle

MySQL-MySQL incr stage: filter where cuase dest dtle panic

asiroliu opened this issue · 1 comments

asiroliu commented

Description

MySQL-MySQL incr stage: filter where cuase dest dtle panic

Steps to reproduce the issue

  1. create database and table
sql> CREATE DATABASE IF NOT EXISTS `action_db_1`;
shell> sysbench /usr/share/sysbench/oltp_common.lua --mysql-host=172.100.9.3 --mysql-port=3306 --mysql-user=test --mysql-password=test --create_secondary=off --report-interval=10 --time=0 --mysql-db=action_db_1 --tables=5 --table_size=5000 prepare
  1. create dtle job
{
  "job_id": "oneToOne_table_level_1_db_n_table",
  "is_password_encrypted": false,
  "task_step_name": "all",
  "failover": true,
  "retry": 2,
  "src_task": {
    "task_name": "src",
    "node_id": "b8216366-516f-5560-de18-96354a824c4a",
    "mysql_src_task_config": {
      "gtid": "",
      "binlog_relay": false
    },
    "drop_table_if_exists": true,
    "skip_create_db_table": false,
    "repl_chan_buffer_size": 120,
    "chunk_size": 2000,
    "group_max_size": 1,
    "group_timeout": 100,
    "connection_config": {
      "database_type": "MySQL",
      "host": "172.100.9.1",
      "port": 3306,
      "user": "test_src",
      "password": "test_src"
    },
    "replicate_do_db": [
      {
        "table_schema": "action_db",
        "tables": [
          {
            "table_name": "sbtest1",
            "where": "id <= 2500"
          },
          {
            "table_name": "sbtest2",
            "where": "id > 2500"
          },
          {
            "table_name": "sbtest3",
            "where": "id BETWEEN 1001 AND 3500"
          },
          {
            "table_name": "sbtest4",
            "where": "id IN (2492,2494,2496,2498,2500,2502,2504,2506,2508,2510)"
          },
          {
            "table_name": "sbtest5",
            "where": "c LIKE '1%'"
          }
        ]
      }
    ]
  },
  "dest_task": {
    "task_name": "dest",
    "node_id": "9a1847d9-61b2-5dff-593a-ac016b669d27",
    "mysql_dest_task_config": {
      "use_my_sql_dependency": false,
      "dependency_history_size": 2500,
      "parallel_workers": 32
    },
    "connection_config": {
      "database_type": "MySQL",
      "host": "172.100.9.2",
      "port": 3306,
      "user": "test_dest",
      "password": "test_dest"
    }
  }
}
  1. src MySQL execute DML sql
shell> sysbench /usr/share/sysbench/oltp_write_only.lua --mysql-host=172.100.9.3 --mysql-port=3306 --mysql-user=test --mysql-password=test --create_secondary=off --report-interval=10 --time=0 --mysql-db=action_db_1 --tables=5 --table_size=5000 --events=15000 --rand-type=uniform run
  1. check dest DTLE log
2022-07-11T12:10:32.517+0800 [DEBUG] client.driver_mgr.dtle: incr. incrBytesQueue enqueued: driver=dtle job=oneToOne_table_level_1_db_n_table-migration vacancy=32 @module=dtle.applier timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.517+0800 [DEBUG] client.driver_mgr.dtle: incr. after publish nats reply.: driver=dtle @module=dtle.applier job=oneToOne_table_level_1_db_n_table-migration timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.517+0800 [DEBUG] client.driver_mgr.dtle: reuse tableColumns: driver=dtle @module=dtle.applier job=oneToOne_table_level_1_db_n_table-migration schema=action_db table=sbtest1 timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.517+0800 [DEBUG] client.driver_mgr.dtle: reuse tableColumns: driver=dtle @module=dtle.applier job=oneToOne_table_level_1_db_n_table-migration schema=action_db table=sbtest1 timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.517+0800 [DEBUG] client.driver_mgr.dtle: writeset use key: driver=dtle columns=map[id:0] name=PRIMARY timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.517+0800 [DEBUG] client.driver_mgr.dtle: writeset use key: driver=dtle columns=map[id:0] name=PRIMARY timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.518+0800 [DEBUG] client.driver_mgr.dtle: writeset use key: driver=dtle columns=map[id:0] name=PRIMARY timestamp=2022-07-11T12:10:32.517+0800
2022-07-11T12:10:32.519+0800 [DEBUG] client.driver_mgr.dtle: panic: runtime error: index out of range [0] with length 0: driver=dtle
2022-07-11T12:10:32.519+0800 [DEBUG] client.driver_mgr.dtle: : driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: goroutine 455 [running]:: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: github.com/actiontech/dtle/driver/common.RowColumnIsNull(...): driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/common/columntypes.go:13: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: github.com/actiontech/dtle/driver/mysql.HashTx.func1(0x0, 0x0, 0x0): driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/mysql/applier_mts.go:166 +0x525: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: github.com/actiontech/dtle/driver/mysql.HashTx(0xc0037519e0, 0xc0035b0530, 0x2, 0x2): driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/mysql/applier_mts.go:178 +0x12c: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: github.com/actiontech/dtle/driver/mysql.(*WritesetManager).GatLastCommit(0xc0014d3d10, 0xc0037519e0, 0x7fa3a8df09a0, 0xc00076b800, 0x8): driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/mysql/applier_mts.go:201 +0x6b: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: github.com/actiontech/dtle/driver/mysql.(*ApplierIncr).handleEntry(0xc003733600, 0xc0037519e0, 0x0, 0x0): driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/mysql/applier_incr.go:348 +0xc74: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: github.com/actiontech/dtle/driver/mysql.(*ApplierIncr).heterogeneousReplay.func1(0xc003733600): driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/mysql/applier_incr.go:385 +0x125: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle: created by github.com/actiontech/dtle/driver/mysql.(*ApplierIncr).heterogeneousReplay: driver=dtle
2022-07-11T12:10:32.520+0800 [DEBUG] client.driver_mgr.dtle:    /universe/src/github.com/actiontech/dtle/driver/mysql/applier_incr.go:378 +0xd0: driver=dtle
2022-07-11T12:10:32.529+0800 [DEBUG] client.driver_mgr.stdio: received EOF, stopping recv loop: driver=dtle err="rpc error: code = Unavailable desc = transport is closing"
2022-07-11T12:10:32.529+0800 [ERROR] client.driver_mgr: error receiving stream from TaskStats driver RPC, closing stream: driver=dtle error="rpc error: code = Unavailable desc = transport is closing"
2022-07-11T12:10:32.529+0800 [DEBUG] client.driver_mgr: plugin process exited: driver=dtle path=/opt/dtle/usr/share/dtle/nomad-plugin/dtle pid=591 error="exit status 2"
2022-07-11T12:10:32.529+0800 [WARN]  client.driver_mgr: received fingerprint error from driver: driver=dtle error="plugin is shut down"
2022-07-11T12:10:32.529+0800 [WARN]  client.driver_mgr: failed to reattach to plugin, starting new instance: driver=dtle err="singleton plugin exited"
2022-07-11T12:10:32.529+0800 [DEBUG] client.alloc_runner.task_runner: driver plugin has shutdown; attempting to recover task: alloc_id=7c456fc1-7e79-93eb-5251-91fd985b8bd3 task=dest driver=dtle

Output of ./dtle version:**

9.9.9.9-master-3784044
asiroliu commented

verify version:
9.9.9.9-master-72a4c06