The response metadata is not correct for the table name is omitted
lilinghai opened this issue · 8 comments
Reproduce
1. create table and insert data in mysql
create table t(a int, b int)
insert into t values(1,2)
2. use gohive client to execute
cursor.Exec(ctx,"select * from t as x left join t as y on x.a=y.b")
cursor.RowMap(ctx)
Expected Result
x.a: 1 x.b:2 y.a: nil y.b:nil
But Get
a:nil b.nil
Hello, I've tested this here: https://github.com/beltran/gohive/pull/121/files and tests are passing (https://travis-ci.org/github/beltran/gohive/builds/738294406) so I haven't been able to reproduce (I'm getting the expected result x.a: 1 x.b:2 y.a: nil y.b:nil
)
Could you verify you are indeed getting a:nil b.nil
, also gohive simple surfaces whatever is coming from hive so I'm not sure how this could be happening, what happening when you execute "select * from t as x left join t as y on x.a=y.b"
with other clients like beehive?
My reproduce steps:
- start spark thrift server
./sbin/start-thriftserver.sh - use go hive to execute sql
drop table if exists t
create table t(a int, b int)
insert into t values(1,2)
select * from t as x left join t as y on x.a=y.b
the result of RowMap
is :
map[a:<nil> b:<nil>]
3. The expected result is
- use beeline:
+----+----+-------+-------+--+
| a | b | a | b |
+----+----+-------+-------+--+
| 1 | 2 | NULL | NULL |
+----+----+-------+-------+--+
- use PyHive:
[(1, 2, None, None)]
Hello, thank you for the repro script, I see now what the problem is but I don't see a solution with RowMap to this but print a warning that two columns have the same name (spark returns the raw column but hive returns tableName + columnName).
Your best option is to use FetchOne
, you can specify interfaces if you don't want to write down all the variables and it's type like it's done here if it's more convenient.
Thanks for your reply. The combination cursor.Description()
and FetchOne
is a good option to achieve it.
The problem is that RowMap
returns a map from TableName.ColumnName
to the values so if TableName
is not available from cursor.Description
there's no way to build this map. Also since several columns share the same name we'd need more than one key for the two columns in the map.
I see PyHive doesn't run into this because they return an array, for example, in the script you provided it returns [(1, 2, None, None)]
When using FetchOne
, it can't get null
values. The reproduce script is as follows:
cursor.Exec(ctx,"drop table if exists t")
cursor.Exec(ctx,"create table t(a int,b double ,c char(10),d date)")
cursor.Exec(ctx,"insert into t values(null,null,null,null)")
cursor.Exec(ctx, "select * from t")
desc := cursor.Description()
columnValues := make([]interface{}, len(desc))
cursor.FetchOne(ctx, columnValues...)
fmt.Println(columnValues)
The result is:
[0 0 ]
But the expected result is:
[nil nil nil nil]
If you care about nil values there are some docs here https://github.com/beltran/gohive#null-values, but the bottom line is that you won't be able to use
columnValues := make([]interface{}, len(desc))
you'll have to define all the pointers and then pass them to FetchOne
:
var a *int32 = new(int32)
var b *float64 = new(float64)
var c *string = new(string)
var d *string = new(string)
cursor.FetchOne(ctx, a, b, c, d)