BIGINT column shows up as col_integer()

[bhaskarvk]

I am not sure if this is a bug or not yet. But I have a parquet table with columns of datatype BIGINT.
When querying using drill_connection(), they show up as col_integer(). I am not sure what the range of col_integer is, but if it's indeed 4 bytes then there will be loss of data when querying BIGINT columns where value exceeds integers min/max values.

[bhaskarvk]

OK there's some internal cleverness going on. Looks like the datatype is determined at query time based on the actual values being returned. e.g.

When the returned rows does contain a value > Max(Int) the column is of type double, but if the returned values are within Int limits then column type is integer.

>drill_query(dc,'select max(src_bytes) as max_src_bytes from dfs.data.`counts/1970/03/01`;')
  |======================================================================| 100%
Parsed with column specification:
cols(
  max_src_bytes = col_double()
)
# A tibble: 1 x 1
  max_src_bytes
*         <dbl>
1  681542898424

>drill_query(dc,'select src_bytes from dfs.lanl_parquet.`counts/1970` limit 5;')
  |======================================================================| 100%
Parsed with column specification:
cols(
  src_bytes = col_integer()
)
# A tibble: 5 x 1
  src_bytes
*     <int>
1       576
2       138
3       654
4 810104048
5   4176466

>

[hrbrmstr]

I've been thinking abt making the pass through to `readr::type_convert()` an option vs default. It hasn't bit me yet, but it'll be interesting to see if your work manages to break it. I'll gladly nuke it if so.

…

On Mon, Oct 23, 2017 at 12:35 PM, Bhaskar V. Karambelkar < ***@***.***> wrote: OK there's some internal cleverness going on, looks like the datatype is determined at query time based on the actual values being returned. e.g. When the returned rows does contain a value > Max(Int) the column is of type double, but if the returned values are within Int limits then column type is integer. >drill_query(dc,'select max(src_bytes) as max_src_bytes from dfs.data.`counts/1970/03/01`;') |======================================================================| 100% Parsed with column specification: cols( max_src_bytes = col_double() ) # A tibble: 1 x 1 max_src_bytes * <dbl> 1 681542898424 >drill_query(dc,'select * from dfs.lanl_parquet.`counts/1970` limit 5;') |======================================================================| 100% Parsed with column specification: cols( netflows = col_integer(), dstname = col_character(), srcname = col_character(), src_bytes = col_integer(), dst_pkts = col_integer(), src_pkts = col_integer(), bucketid = col_double(), dir1 = col_character(), dir0 = col_character(), dst_bytes = col_integer() ) # A tibble: 5 x 10 netflows dstname srcname src_bytes dst_pkts src_pkts * <int> <chr> <chr> <int> <int> <int> 1 1 Comp738970 Comp472884 576 0 6 2 1 Comp906667 Comp904805 138 0 132 3 1 Comp398468 Comp531457 654 0 7 4 6264 Comp584379 Comp107130 810104048 218699720 88907572 5 15080 EnterpriseAppServer Comp030334 4176466 45604 65743 # ... with 4 more variables: bucketid <dbl>, dir1 <chr>, dir0 <chr>, # dst_bytes <int> > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfHtmy3AmnrG0BxFrI-wkfAf77vGO3Dks5svMBVgaJpZM4QDIEs> .

[hrbrmstr]

ODBC driver interface to Drill handles this appropriately.