Error in Join Marketing and Sales Data

[johnsontroye1]

I have pulled down this repo and have it working until the last step (Join Marketing and Sales Data). I have tried to get past this unsuccessfully. Here's the error logged in Gluerunner CloudWatch logs:

[ERROR] 2018-07-18T15:17:26.792Z 88fb4fc4-8a9d-11e8-bec7-f7119107e998 Glue job "JoinMarketingAndSalesData" run with Run Id "jr_bebcc..." failed. Last state: FAILED. Error message: AnalysisException: u'Path does not exist: hdfs://ip-172-31-74-135.ec2.internal:8020/user/root/aa.etl-output-path/tmp/sales;'

[moanany]

Hello, This seems to be an internal issue related to AWS Glue. Could you reliably reproduce this error on subsequent runs of the ETL state machine? If so, please open a support case.

…

On Wed, Jul 18, 2018 at 7:01 PM Troy Johnson ***@***.***> wrote: I have pulled down this repo and have it working until the last step (Join Marketing and Sales Data). I have tried to get past this unsuccessfully. Here's the error logged in Gluerunner CloudWatch logs: [ERROR] 2018-07-18T15:17:26.792Z 88fb4fc4-8a9d-11e8-bec7-f7119107e998 Glue job "JoinMarketingAndSalesData" run with Run Id "jr_bebcc..." failed. Last state: FAILED. Error message: AnalysisException: u'Path does not exist: hdfs://ip-172-31-74-135.ec2.internal:8020/user/root/aa.etl-output-path/tmp/sales;' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APzRrLDRCQcqdIZtWpIBtuo2ROSSJXSPks5uH2oFgaJpZM4VU976> .

[johnsontroye1]

Yes, i essentially cleaned out everything several times and reran to the same point of error. The only difference i see in the logs is different run id and ip address to the ec2. Can you please tell me where I go to open a support case for this? Thank you.

[moanany]

Sure, check out the instructions here: https://docs.aws.amazon.com/awssupport/latest/user/getting-started.html#case-management Also, I've just re-run the ETL state machine again just to be sure. The state machine completed successfully. This leaves us with either a possible internal issue with AWS Glue or a project configuration issue. Hope this helps.

…

On Wed, Jul 18, 2018 at 7:36 PM Troy Johnson ***@***.***> wrote: Yes, i essentially cleaned out everything several times and reran to the same point of error. The only difference i see in the logs is different run id and ip address to the ec2. Can you please tell me where I go to open a support case for this? Thank you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APzRrGpB72r7DSQkTC-IdLNw5UMKMDuxks5uH3IKgaJpZM4VU976> .

[johnsontroye1]

There were 5 .json files in the repo that needed config changes.

• cloudformation/gluerunner-lambda-params.json
• lambda/s3-deployment-descriptor.json
• cloudformation/glue-resources-params.json
• lambda/gluerunner/gluerunner-config.json
• cloudformation/step-functions-resources-params.json

Would you mind sending me your .json files so i can compare against what i have. Maybe i did mess up a configuration.

troy.johnson@changepoint.com

Thank you very much,

Troy

[moanany]

Hey Troy — I don’t mind at all .. I’m out of office until 8/6, so I’ll share as soon as I return

…

On Wed, Jul 18, 2018 at 8:51 PM Troy Johnson ***@***.***> wrote: There were 5 .json files in the repo that needed config changes. - cloudformation/gluerunner-lambda-params.json - lambda/s3-deployment-descriptor.json - cloudformation/glue-resources-params.json - lambda/gluerunner/gluerunner-config.json - cloudformation/step-functions-resources-params.json Would you mind sending me your .json files so i can compare against what i have. Maybe i did mess up a configuration. ***@***.*** Thank you very much, Troy — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APzRrI3c_8rqfPnrfM16Iqwez-roQS8oks5uH4OjgaJpZM4VU976> .

[shengdade]

It could be the reason that a wrong parameter set in glue-resources-params.json:

{
    "ParameterKey": "S3ETLOutputPath",
    "ParameterValue": "<NO-DEFAULT>"
}

Please make sure ParameterValue is indeed set to a S3 path, like:

s3://<bucket_name>/output

Not simply:

output

Because the later will actually write the result to HDFS local system! That's why the Join Marketing and Sales Data couldn't find the file.

[moanany]

Config parameters and docs were updated to simplify the configuration process and make it less error prone.