From the Competitive Programming Initiative. Every 1,000 submissions costs us a little over 5 cents to run.
To create a low-cost, reliable, fast, and consistent online judge that supports C++, Java, and Python.
Everything is done with AWS Lambda functions. The online judge itself is split into two functions:
- The main lambda function that the user calls. This lambda function calls the sandboxed code execution lambda function and returns the output to the user. It has full access to AWS resources, so it should not execute any user code.
- The sandboxed code execution lambda function that either takes source code and returns a compiled binary, or takes a compiled binary and the input and returns the output. For security, this function has minimal access to AWS resources. This function is never directly called by the user, and can only be called by the main lambda function.
The submit function supports two modes: Execution (designed for the USACO Guide IDE) and submission (designed for problem submission with USACO Guide groups, or problems with multiple test cases).
User passes in language, source code, and either input. The submit function uses the execution lambda to compile the code, then uses the execution lambda to run the code. Finally, the result of the code execution is returned synchronously.
User passes in language, source code, and problem ID. The submit function creates a new submission ID, adds it to the database, and returns the submission ID to the user. Then, it calls the execution lambda to compile the code, updating the submission status in the database accordingly. Finally, for each test case, it calls another execution lambda to execute the code, updating the submission submission status as each result comes in.
The user passes in the submission ID they received from the Problem Submission POST request, and the lambda function queries the database and returns the status of the submission.
This internal lambda accepts a base64-encoded executable, the input, and returns the output of the program. The lambda does not sandbox code execution due to lack of root privileges; the lambda itself is the sandbox. Therefore:
- the lambda should not be given the expected output
- the lambda should not have internet access (blocked by AWS VPC)
- the lambda should not have dangerous permissions (blocked by AWS IAM roles)
- the lambda should be able to handle malicious code (if the lambda crashes then AWS auto-spawns a new one)
With these restrictions in place, I think the most malicious code can do is screw around with future submissions that are executed on the same lambda. But hopefully nobody is malicious enough to do that.
This lambda is also responsible for code compilation. For languages that don't need compilation, like Python, all "compilation" entails is putting the Python file as well as the Python command into a base64-encoded ZIP file.
Note that since Java compilation often results in multiple output files, this lambda should accept base64-encoded ZIP files.
We use the AWS Serverless Application Model (SAM).
- If you get "Build Failed .... Path resolution for runtime: provided of binary: make was not successful", make sure
makeis installed.
sam build
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
You can change the event JSON file accordingly to test various parts of the execution lambda.
sam build --cached --parallel && sam local invoke ExecuteFunction -e execute/events/compile.json
- Remove
.aws-samfolder, so thatsam local invokewill default to the source folder - Run
npm run watchinsidesubmit/ - Run
sam local start-api - Make a request to
http://localhost:3000/executeor the appropriate API route
You don't have to rerun sam local start-api when your code changes. Note that the execute lambda this calls is the execution lambda that's currently deployed on AWS, not the local execution lambda.
Alternatively, you can run npm run watch followed by node dist/src/runLambdaLocally.js if you don't want to have to make a request to the API route.
npm run test
If snapshots need to be updated:
npm run test -- -u
Source: https://docs.aws.amazon.com/AmazonECR/latest/userguide/amazon_linux_container_image.html
docker run -it public.ecr.aws/amazonlinux/amazonlinux /bin/bash
npx prettier --write .
To push to development:
sam build
sam deploy --stack-name Stage
To push to production:
sam build
sam deploy --stack-name online-judge
The API can be accessed at https://ggzk2rm2ad.execute-api.us-west-1.amazonaws.com/Prod
The staging API can be accessed at https://kmazh7pzpg.execute-api.us-west-1.amazonaws.com/Prod
POST /execute: Code execution. Ex: USACO Guide IDE.POST /submissions(under development): Create a new problem submission. Ex: code submission on USACO Guide groupsGET /submissions/{submissionID}(under development): Get the status of the submission associated with the given submission ID
Until API docs can be properly written, you can refer to submit/src/types.ts to get an idea of the payload and the response types.
Note: REST API is used over HTTP API because of CORS issues with HTTP.
Warning: Not for public use; contact us if you'd like to use the API. The API may change without warning.
The default version of GCC is not new enough to support some of the -fsanitize=undefined features. We can use devtoolset-10 to get access to a newer version of GCC.
yum install -y yum-utils wget
yum-config-manager --add-repo http://mirror.centos.org/centos/7/sclo/x86_64/rh/
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libgfortran5-8.3.1-2.1.1.el7.x86_64.rpm
yum install libgfortran5-8.3.1-2.1.1.el7.x86_64.rpm -y
yum install -y devtoolset-10 --nogpgcheck
scl enable devtoolset-10 bash
You might be able to get away with just installing devtoolset-10-toolchain, but I haven't tried.
Note that the last line, scl enable devtoolset-10 bash, doesn't seem to work with AWS Lambda. Instead, the full path, /opt/rh/devtoolset-10/root/usr/bin/g++, is used.
Furthermore, some optional libraries need to be installed for flags like fsanitize to work. Optional libraries can be found with yum list available devtoolset-10-\*. I have no idea which are needed...
devtoolset-10-libubsan-devel.x86_64is needed forfsanitize=undefineddevtoolset-10-libasan-devel.x86_64is needed forfsanitize=address
Also see this SO post.
I encountered an issue where all compilations would fail because the /tmp directory was out of space. This turned out to be because AWS Lambda would generate core dump files anytime there is a runtime error. Solved with ulimit -c 0.
I got this once:
2021-12-05T00:32:08.371Z 89cac995-f38c-43d1-bfb5-16a76aa83f70 ERROR InvalidSignatureException: Signature expired: 20211205T002507Z is now earlier than 20211205T002708Z (20211205T003208Z - 5 min.)
at deserializeAws_restJson1InvokeCommandError (/var/task/node_modules/@aws-sdk/client-lambda/dist-cjs/protocols/Aws_restJson1.js:5156:41)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async /var/task/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:6:20
at async /var/task/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20
at async StandardRetryStrategy.retry (/var/task/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
at async /var/task/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
at async compile (/var/task/dist/src/helpers/compile.js:16:29)
at async Promise.all (index 1)
at async createSubmission (/var/task/dist/src/problemSubmission/createSubmission.js:24:47) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '6642a280-9ae4-4de9-98f7-049727ebddca',
extendedRequestId: undefined,
cfId: undefined,
attempts: 3,
totalRetryDelay: 285
}
}
May be able to be temporarily resolved by forcing lambdas to restart:
aws lambda update-function-configuration --function-name "online-judge-SubmitFunction" --description "upd"
aws lambda update-function-configuration --function-name "online-judge-ExecuteFunction" --description "upd"
See #10