Trying to make work two parallel conversion jobs

[josoroma-zz]

Hi!

I am trying to make two parallel requests to an endpoint that kicks off a State Machine, this State Machine handles a couple of video conversions jobs. The thing is, first request is going fine but the second one is failing.

2020-06-04T08:53:59.815-06:00
2020-06-04T14:53:59.813Z	ab2ec8d3-e529-40a0-8fe0-3c94a202ee03	ERROR	Invoke Error	
{
    "errorType": "AccessDeniedException",
    "errorMessage": "Unable to determine service/operation name to be authorized",
    "code": "AccessDeniedException",
    "message": "Unable to determine service/operation name to be authorized",
    "time": "2020-06-04T14:53:59.812Z",
    "requestId": "21253c82-72fe-4349-b22f-7ec72473d084",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 73.48494025325797,
    "stack": [
        "AccessDeniedException: Unable to determine service/operation name to be authorized",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)"
    ]
}

Note: I am using two different users with two different videos to call the convert endpoint.

...
const mediaConvertClient = new AWS.MediaConvert({
  apiVersion: '2017-08-29',
});

export const main = async (event, context, callback) => {
  ...

  const {
    noteId,
    userId,
    startResponse: videoObj = '',
  } = event;

  const { sourceBucketName, convertVideoCloudfront } = process.env;
  const filename = `${videoObj.sourceKey}.${videoObj.videoSourceExtension}`;
  const inputKey = `s3://${sourceBucketName}/private/${userId}/${noteId}/video/${filename}`;

  const job = await convertVideoCreateJob({
    noteId,
    inputBucket: sourceBucketName,
    inputKey,
    outputKey: filename,
    outputFilename: videoObj.sourceKey,
  });

  const result = {
    status: 'created',
    urlHR: `https://${convertVideoCloudfront}/${noteId}/hr.${filename}`,
    urlLR: `https://${convertVideoCloudfront}/${noteId}/lr.${filename}`,
  };

  let createJobResponse = null;

  // Set Mediaconvert endpoint & queue
  const { Endpoints } = await mediaConvertClient
    .describeEndpoints({ MaxResults: 0 })
    .promise();

  mediaConvertClient.endpoint = Endpoints[0].Url;

  const uniqueDate = new Date();
  const milliseconds = uniqueDate.getTime();

  // https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/MediaConvert.html#createQueue-property
  const queue = await mediaConvertClient
    .createQueue({
      Name: `ConvertVideo${milliseconds}`,
      PricingPlan: 'ON_DEMAND', // ON_DEMAND | RESERVED
    })
    .promise();

  const jobQueueParams = { ...job, Queue: queue.Queue.Arn };

  try {
    createJobResponse = await mediaConvertClient
      .createJob(jobQueueParams)
      .promise();

    result.conversionLog = JSON.stringify(createJobResponse);
  } catch (error) {
    result.conversionStatus = 'failed';
    result.conversionLog = error.toString();

    callback(new HandleErrorOnJob({
      error,
      result,
    }));
  }

  // Poll for Job Completion
  let getJobResponse = null;

  const jobParams = {
    Id: _get(createJobResponse, 'Job.Id'),
  };

  // Since we use and endpoint to kick off a state machine via an synchronous request,
  // I am not really sure how much this poller code will be needed here or not.
  // https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/MediaConvert.html#getJob-property
  do {
    await new Promise((resolve) => setTimeout(resolve, 1500));

    getJobResponse = await mediaConvertClient
      .getJob(jobParams)
      .promise();
  } while (
    // SUBMITTED, PROGRESSING, COMPLETE, CANCELED, or ERROR.
    getJobResponse.Status === 'PROGRESSING'
  );

  const jobResponse = {
    queue,
    getJobResponse,
    result,
    endpoints: Endpoints,
  };

  return jobResponse;
};

Where convertVideoCreateJob returns:

  return {
    Queue: '',
    Role: convertVideoRole,
    Settings: {
      OutputGroups: outputGroups,
      AdAvailOffset: 0,
      Inputs: [{
        FileInput: inputKey,
        ...inputSettings,
      }],
      TimecodeConfig: {
        Source: 'EMBEDDED',
      },
    },
    UserMetadata: {
      outputKey,
    },
  };

Thank you!

[josoroma-zz]

😢

[tschoffelen]

Hi @josoroma, I'd maybe post this on the AWS support forum or ask somewhere else, I can't really help you with this kind of stuff, the code in this repo is just a quick POC that we've done for a client, I can't spend time doing support unfortunately.

[prgwar]

Hello @josoroma I also encountered with the same problem I believe the problem is with the getting endpoint url api there might be a limit on calling this api from the same IP with in span of min const { Endpoints } = await mediaConvertClient .describeEndpoints({ MaxResults: 0 }) .promise() I hard coded the endpoint url after that the issue was resolved. let me know if worked

[prgwar]

If the endpoint url gets updated the Job will fail

[prgwar]

More information can be found on https://docs.aws.amazon.com/general/latest/gr/mediaconvert.html

[apolle]

^ That the correct answer, to quote AWS:

Request rate for DescribeEndpoints
0.01667 TPS (Once per 60 seconds)

This endpoint is specific to your AWS account and won't change. 
Request this endpoint once, and then hardcode or cache it.

[tschoffelen]

Fixed by #7.