ClickHouse/clickhouse-jdbc-bridge

Chinese table name and column name

delubee opened this issue · 6 comments

delubee commented

when the column name is Chinese I got this error:

二月 18, 2022 7:18:46 下午 ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource executeQuery
信息: Executing query(schema=[]):
select * from t2
二月 18, 2022 7:18:46 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle errorHandler
严重: Failed to respond
java.lang.IllegalArgumentException: Unknown column [å­æ®µ1]! Available columns are: [id],[字段1]
	at ru.yandex.clickhouse.jdbcbridge.core.DataTableReader.process(DataTableReader.java:128)
	at ru.yandex.clickhouse.jdbcbridge.impl.JdbcDataSource.writeQueryResult(JdbcDataSource.java:721)
	at ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource.executeQuery(NamedDataSource.java:552)
	at ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle.handleQuery(JdbcBridgeVerticle.java:487)
	at ru.yandex.clickhouse.jdbcbridge.internal.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at ru.yandex.clickhouse.jdbcbridge.internal.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at ru.yandex.clickhouse.jdbcbridge.internal.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

table t2 DDL :
create table t2
(
id int not null,
字段1 varchar(255)
)

and when the table name is Chinese I got this error:

二月 18, 2022 7:31:19 下午 ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource executeQuery
信息: Executing query(schema=[]):
select * from 表1
二月 18, 2022 7:31:19 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle errorHandler
严重: Failed to respond
ru.yandex.clickhouse.jdbcbridge.core.DataAccessException: Failed to access [mssql] due to: SQLState(S0001) VendorCode(102) “¡”附近有语法错误。
	at ru.yandex.clickhouse.jdbcbridge.impl.JdbcDataSource.writeQueryResult(JdbcDataSource.java:734)
	at ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource.executeQuery(NamedDataSource.java:552)
	at ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle.handleQuery(JdbcBridgeVerticle.java:487)
	at ru.yandex.clickhouse.jdbcbridge.internal.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at ru.yandex.clickhouse.jdbcbridge.internal.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at ru.yandex.clickhouse.jdbcbridge.internal.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: “¡”附近有语法错误。
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1624)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:868)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:768)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2979)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:248)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:223)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStatement.java:744)
	at ru.yandex.clickhouse.jdbcbridge.internal.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:95)
	at ru.yandex.clickhouse.jdbcbridge.internal.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
	at ru.yandex.clickhouse.jdbcbridge.impl.JdbcDataSource.writeQueryResult(JdbcDataSource.java:718)
	... 8 more

table 表1 DDL:
create table 表1
(
id int not null,
f1 varchar(255)
)

and when column and table names is ASCII ， it’s fine.

yingyingqiqi commented

I have just started to try JDBC bridge, and we also have an abnormal problem in Chinese fields. Do you have a repair plan?

zhicwu commented

I have just started to try JDBC bridge, and we also have an abnormal problem in Chinese fields. Do you have a repair plan?

I'll take a look tonight to see if this can be quickly fixed.

yingyingqiqi commented

The question arises here:：

clickhouse-jdbc-bridge/src/main/java/ru/yandex/clickhouse/jdbcbridge/core/DataTableReader.java

Lines 105 to 116 in f02b015

// now result columns
if (!matched) {
for (int j = 0; j < resultColumns.length; j++) {
ColumnDefinition result = resultColumns[j];
if (colName.equals(result.getName())) {
// colIndices[i] = new int[] { i, j + 1 };
colIndices[i] = new int[] { i, j };
matched = true;
break;
}
}
}

due to： ColumnDefinition[] requestColumn != ColumnDefinition[] resultColumns
Because the following code cannot parse Chinese:

clickhouse-jdbc-bridge/src/main/java/ru/yandex/clickhouse/jdbcbridge/core/QueryParser.java

Lines 317 to 331 in f02b015

static String unescape(String str) {
StringBuilder builder = new StringBuilder();
for (int i = 0, len = str.length(); i < len; i++) {
char ch = str.charAt(i);
if (ch == '%' && i + 2 < len) {
ch = (char) Integer.parseInt(str.substring(i + 1, i + 3), 16);
i = i + 2;
}
builder.append(ch);
}
return builder.toString();
}

I think it can be changed to: URLDecoder.decode(str, "utf-8");

zhicwu commented

Thanks @yingyingqiqi for investigating the issue. Actually above code has nothing to do with charset encoding because it's based on characters. However, there is issue extracting query parameters but I haven't checked if it's caused by Vertx or ClickHouse.

delubee commented

The question arises here:：

clickhouse-jdbc-bridge/src/main/java/ru/yandex/clickhouse/jdbcbridge/core/DataTableReader.java

Lines 105 to 116 in f02b015

// now result columns
if (!matched) {
for (int j = 0; j < resultColumns.length; j++) {
ColumnDefinition result = resultColumns[j];
if (colName.equals(result.getName())) {
// colIndices[i] = new int[] { i, j + 1 };
colIndices[i] = new int[] { i, j };
matched = true;
break;
}
}
}

due to： ColumnDefinition[] requestColumn != ColumnDefinition[] resultColumns
Because the following code cannot parse Chinese:

clickhouse-jdbc-bridge/src/main/java/ru/yandex/clickhouse/jdbcbridge/core/QueryParser.java

Lines 317 to 331 in f02b015

static String unescape(String str) {
StringBuilder builder = new StringBuilder();
for (int i = 0, len = str.length(); i < len; i++) {
char ch = str.charAt(i);
if (ch == '%' && i + 2 < len) {
ch = (char) Integer.parseInt(str.substring(i + 1, i + 3), 16);
i = i + 2;
}
builder.append(ch);
}
return builder.toString();
}

I think it can be changed to: URLDecoder.decode(str, "utf-8");

it works :

// columns = unescape(table.substring(len, index));
// table = unescape(table.substring(index + 1));
try {
columns = URLDecoder.decode(table.substring(len, index), "utf-8");
table = URLDecoder.decode(table.substring(index + 1), "utf-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}

  • 👍1
zhicwu commented

it works

Good work. Mind to submit a pull request?