audienceproject/spark-dynamodb

In python following readme example but unable to print out any data

ahsan-n opened this issue · 3 comments

version: spark-dynamodb_2.12:1.1.0
dynamoDf = spark.read.option("tableName", "mytable").option("region","eu-west-1") \ .format("dynamodb") \ .load() dynamoDf.show(100)

this gives me empty table

++ || ++ || || ++

but when i use load().count() it showed me the right count of total entries in table.

yeah, it happened to me as well, trying passing the schema.

@ahsan-n can you tell how you are importing the module? Getting error while importing. Was able to download it as mentioned in readme.

I have the same error as above, get empty table in return

Have given the schema and got it working

from pyspark.sql.types import *

fields = [StructField("email", StringType(), True)
          , StructField("first_name", StringType(), True)
          , StructField("last_name", StringType(), True)]

schema = StructType(fields)

# Load data from dynamo into spark dataframe
dynamoDf = spark.read.option("tableName","db_dev_users") \
                     .option("region",region) \
                     .schema(schema) \
                     .format("dynamodb") \
                     .load() 

Have been using this in the databricks environment, and installed the packages from Maven repo
version 1.1.2, and it works

Thanks @kaifee