Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The provided key element does not match the schema
Closed this issue · 2 comments
Hi Team,
While trying to write spark dataframe to dynamodb I am getting the below error message
Error:
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The provided key element does not match the schema (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: O1THUCBT474TKDSSON3POOD3DBVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
Note: 50 records are successfully inserted into dynamodb while writing after 50 records below error is coming. There are nearly 5.7million rows in my dataframe
DynamoDB table structure:
f_cartid -> primary key datatype:Number
f_cartmenuid -> sort key datatype:Number
Spark write code:
df.write.option("tableName", mytable).option("region",region).format("dynamodb").save()
Spark Dataframe schema structure:
fnDF.printSchema()
root
|-- f_cartid: long (nullable = true)
|-- f_cartmenuid: long (nullable = true)
|-- f_orderid: double (nullable = true)
|-- f_hotelid: double (nullable = true)
|-- items: struct (nullable = false)
| |-- f_cartmenuid: long (nullable = true)
| |-- menuDetails: struct (nullable = false)
| | |-- f_menuid: long (nullable = true)
| | |-- f_qty: long (nullable = true)
| | |-- f_menuprice: double (nullable = true)
|-- users: struct (nullable = false)
| |-- f_custid: long (nullable = true)
| |-- userDetails: struct (nullable = false)
| | |-- F_Name: string (nullable = true)
| | |-- F_EmailId: string (nullable = true)
| | |-- f_delpostcode: string (nullable = true)
| | |-- f_deliveryaddress: string (nullable = true)
|-- f_device_name: string (nullable = true)
|-- f_ipaddress: string (nullable = true)
|-- f_carttotalamt: double (nullable = true)
|-- f_wanted_asap: long (nullable = true)
|-- f_deliverabletime: string (nullable = true)
|-- f_paymentcharge: double (nullable = true)
|-- f_redempoint: double (nullable = true)
|-- f_dellocation: long (nullable = true)
|-- f_delcharge: double (nullable = true)
|-- f_couponcode: string (nullable = true)
|-- f_crdt: timestamp (nullable = true)
|-- f_moddt: timestamp (nullable = true)
|-- updated_at: date (nullable = true)
Could anyone guide me where I am missing, it would be really helpful.
Thanks in advance!!
Hi gopi-t2s
I notice in your schema that your hash and range key fields are nullable.
|-- f_cartid: long (nullable = true)
|-- f_cartmenuid: long (nullable = true)
Can you check if you do in fact have null values in your dataframe? That would expectedly cause an error.