KylinOLAP/Kylin

Build cube faild in large table

Closed this issue · 1 comments

Yancey1989 commented

Dear all!
I can build the cube with small table success(2 table join, 10 and 1000rows),But when i increase the rows of the table (10 million, 100 million ),Kylin will throw some exception:

[CuratorFramework-0-EventThread]:[2015-01-07 10:20:17,831][INFO][org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] - State change: SUSPENDED
[CuratorFramework-0-EventThread]:[2015-01-07 10:20:43,740][INFO][org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] - State change: LOST
[CuratorFramework-0-EventThread]:[2015-01-07 10:20:43,740][WARN][org.apache.curator.ConnectionState.handleExpiredSession(ConnectionState.java:289)] - Session expired event received
[CuratorFramework-0-EventThread]:[2015-01-07 10:20:43,744][INFO][org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] - State change: RECONNECTED
[QuartzScheduler_Worker-11]:[2015-01-07 10:21:13,327][DEBUG][com.kylinolap.job.engine.JobFetcher.execute(JobFetcher.java:60)] - 0 pending jobs
[CuratorFramework-0-EventThread]:[2015-01-07 10:21:13,377][INFO][org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] - State change: SUSPENDED
[CuratorFramework-0]:[2015-01-07 10:21:38,932][ERROR][org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:201)] - Connection timed out for connection string (192.168.44.21:2181) and timeout (15000) / elapsed (25555)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
    at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
    at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
    at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[CuratorFramework-0]:[2015-01-07 10:21:56,955][ERROR][org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:201)] - Connection timed out for connection string (192.168.44.21:2181) and timeout (15000) / elapsed (43578)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
    at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
    at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
    at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
lukehan commented

how big your cluster is? typically, more bigger data, more nodes required to processing cube