Unusual container problem breaks the compilation on the API service - need to improve the healthcheck

[stranger80]

A weird issue rendered the zkSync plugin unusable during the night of 26th Feb.
AWS ECS service monitor records indicate a spike of CPU and RAM at ca 2:50 AM:

After the spike, the RAM usage on the container dropped, and compilation started giving weird errors:

WARNING: You are using a version of Node.js that is not supported, and it may work incorrectly, or not work at all. See https://hardhat.org/nodejs-versions
Solidity 0.8.20 is not fully supported yet. You can still use Hardhat, but some features, like stack traces, might not work correctly.

Learn more at https://hardhat.org/hardhat-runner/docs/reference/solidity-support

Warning: Unused function parameter. Remove or comment out the variable name to silence this warning.
  --> latest/a1ad60de6cb8f802ca30c54d8347c2cb57125fb72b95d7f9eeaff47728b6a21f/test/test.sol:11:39:
   |
11 |     constructor(uint256 _initialData, uint256 _someMoreData, string memory _oneMore) {
   |                                       ^^^^^^^^^^^^^^^^^^^^^


Warning: Unused function parameter. Remove or comment out the variable name to silence this warning.
  --> latest/a1ad60de6cb8f802ca30c54d8347c2cb57125fb72b95d7f9eeaff47728b6a21f/test/test.sol:11:62:
   |
11 |     constructor(uint256 _initialData, uint256 _someMoreData, string memory _oneMore) {
   |                                                              ^^^^^^^^^^^^^^^^^^^^^^


Warning: Unused function parameter. Remove or comment out the variable name to silence this warning.
  --> latest/a1ad60de6cb8f802ca30c54d8347c2cb57125fb72b95d7f9eeaff47728b6a21f/test/test.sol:28:34:
   |
28 |     function set2(uint256 _data, uint256 _data2) public {
   |                                  ^^^^^^^^^^^^^^


Error HH700: Artifact for contract "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT" not found. Did you mean "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT"?
HardhatError: HH700: Artifact for contract "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT" not found. Did you mean "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT"?
    at Artifacts._handleWrongArtifactForFullyQualifiedName (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:681:11)
    at Artifacts._getValidArtifactPathFromFullyQualifiedName (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:633:21)
    at async Artifacts._getArtifactPath (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:504:16)
    at async Promise.all (index 1763)
    at async Artifacts.removeObsoleteArtifacts (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:371:34)
    at async SimpleTaskDefinition.action (/hardhat_env/node_modules/hardhat/src/builtin-tasks/compile.ts:1399:3)
    at async Environment._runTaskDefinition (/hardhat_env/node_modules/hardhat/src/internal/core/runtime-environment.ts:330:14)
    at async OverriddenTaskDefinition._action (/hardhat_env/node_modules/@matterlabs/hardhat-zksync-solc/src/index.ts:286:16)
    at async Environment._runTaskDefinition (/hardhat_env/node_modules/hardhat/src/internal/core/runtime-environment.ts:330:14)
    at async Environment.run (/hardhat_env/node_modules/hardhat/src/internal/core/runtime-environment.ts:163:14)
WARNING: You are using a version of Node.js that is not supported, and it may work incorrectly, or not work at all. See https://hardhat.org/nodejs-versions
Solidity 0.8.20 is not fully supported yet. You can still use Hardhat, but some features, like stack traces, might not work correctly.

Learn more at https://hardhat.org/hardhat-runner/docs/reference/solidity-support

Error HH700: Artifact for contract "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT" not found. Did you mean "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT"?
HardhatError: HH700: Artifact for contract "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT" not found. Did you mean "latest/b808582c378270d3e11c1956ad989878a3680e309122ac9da797e486484a3bcd/passport/passport.sol:NFT"?
    at Artifacts._handleWrongArtifactForFullyQualifiedName (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:681:11)
    at Artifacts._getValidArtifactPathFromFullyQualifiedName (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:633:21)
    at async Artifacts._getArtifactPath (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:504:16)
    at async Promise.all (index 1763)
    at async Artifacts.removeObsoleteArtifacts (/hardhat_env/node_modules/hardhat/src/internal/artifacts.ts:371:34)
    at async SimpleTaskDefinition.action (/hardhat_env/node_modules/hardhat/src/builtin-tasks/compile.ts:1399:3)
    at async Environment._runTaskDefinition (/hardhat_env/node_modules/hardhat/src/internal/core/runtime-environment.ts:330:14)
    at async OverriddenTaskDefinition._action (/hardhat_env/node_modules/@matterlabs/hardhat-zksync-solc/src/index.ts:286:16)
    at async Environment._runTaskDefinition (/hardhat_env/node_modules/hardhat/src/internal/core/runtime-environment.ts:330:14)
    at async Environment.run (/hardhat_env/node_modules/hardhat/src/internal/core/runtime-environment.ts:163:14)

The problem was gone after redeploying the API backend service.

We need following actions:

1. Examine the ECS logs from given time range to see if they include any hints of the root cause. If root cause can be identified - address.
2. Improve the healthcheck (/health) to attempt a compilation of a sample .sol file. If fails - fail the healthcheck.