mitchelllisle/sparkdantic

Unable to represent BIGINT

boblannon opened this issue · 4 comments

boblannon commented

This is such a great project idea, just what I needed. Unfortunately, since Python unified int and long(PEP-237), the language no longer makes a type distinction between 4-byte integers and 8-byte integers. That makes it impossible to add a field to SparkModel whose spark type is LongType. I'm not sure what the best way forward is.

The LongType has at least one very critical use case: Unix Timestamps. If timestamps are stored as 4-byte signed integers (which is what IntegerType is), then the maximum date they can represent is 2038-01-19T04:14:07, which is not as far away as it seems :)

dan1elt0m commented

Created PR: #337.
Once merged you can add LongType to Pydantic model using the type override option.

id: int = Field(spark_type=LongType)

boblannon commented

Awesome! Thanks 🚀

DaanRademaker commented

+1 We really need this to keep using sparkdantic (which we love by the way :) )

mitchelllisle commented

Thanks all! Haven't had much time to work on this library lately but huge thanks to @dan1elt0m for the contribution!

Have just released v0.21.0 with this change

Thanks!