BQ Table Schema - Maintaining Original Case of Spark DF Schema

Question

BQ Table Schema - Maintaining Original Case of Spark DF Schema

Closed this issue 7 years ago · 8 comments

Hi Sam,

Is it possible to retain the original case for the BQ Table schema generated? Currently it makes all fields lowercase (customeraccountid) when the original is camel (customerAccountId) - this makes it quite hard for some of the business people to read. Is this possible at all to retain the original case?

Cheers
K

Answer 1 · 2017-05-23T11:05:54.000Z

Hey kurt You can probably send a pr to the schema converter It should be a small one and help you understand the code base

…

On Tue, 23 May 2017 at 11:52, kurtmaile ***@***.***> wrote: Hi Sam, Is it possible to retain the original case for the BQ Table schema generated? Currently it makes all fields lowercase (customeraccountid) when the original is camel (customerAccountId) - this makes it quite hard for some of the business people to read. Is this possible at all to retain the original case? Cheers K — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#23>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEHLmwFMDA9GgCI1GWO-SDvbRoEEz3LNks5r8rpjgaJpZM4Njg0U> .

Answer 2 · 2017-05-23T13:58:21.000Z

cool will try - just looking at the code nothing jumps out as where this is explicitly happening - assume its implicitly (json4s library?) happening somewhere I cant see - a pointer?

fieldToJson - would appear the where this would happen?

Cheers

Answer 3 · 2017-05-30T14:15:13.000Z

Hey Sam any hints as to why the schema casing is all lower case? Nothing obvious jumps out, I may be missing the missing the obvous too :)

Answer 4 · 2017-05-30T14:19:35.000Z

Hey @kurtmaile I believe its here https://github.com/samelamin/spark-bigquery/blob/master/src/main/scala/com/samelamin/spark/bigquery/converters/BigQueryAdapter.scala#L17

Give it a test locally after you do the pr to ensure it still saves to bq :)

Answer 5 · 2017-05-30T19:32:38.000Z

Oh cool thanks - what is the reason you initially made this lower case though so that I dont break things? Is there a BQ constraint to adhere to? Thanks for the tip ;)

Answer 6 · 2017-06-13T15:37:30.000Z

I think initially BQ didnt allow uppercase schemas but this has now been fixed, send a PR over when your free :)

Answer 7 · 2017-06-22T21:51:54.000Z

This has now been released on v 0.1.7

Answer 8 · 2017-06-23T09:55:10.000Z

Awesome thanks!

…

On Thu, 22 Jun 2017 at 22:52, Sam Elamin ***@***.***> wrote: This has now been released on v 0.1.7 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATIvhR6M-vvTkCo1vbEDb23ORISmfxrrks5sGuH6gaJpZM4Njg0U> .