oracle/oci-java-sdk

Retries for operations that upload binary data without request-level retries do not retry in OCI Java SDK versions 3.0.0 to 3.31.0

mricken opened this issue · 1 comments

mricken commented

If you are using any of the OCI Java SDK synchronous clients that upload streams of data, e.g. ObjectStorageClient or DataSafeClient, and you do not define the RetryConfiguration at request level, your requests will not be automatically retried. However, there is no chance of silent data corruption.

Description

When using OCI Java SDK (versions >= 3.0.0) for operations that have retries enabled by default, and you do not define the RetryConfiguration at request level, and the request fails with a retry-able error, the OCI Java SDK should automatically retry the request. In this situation, the clients fail to reset the stream position for requests that upload streams. As a result, retries cannot be attempted, and the operation fails with a BmcException.

The upload of the stream is likely incomplete at this time and needs to be re-attempted. Fortunately, this failure is visible and therefore cannot lead to silent data corruption.

Affected requests

This happens only for synchronous clients in versions 3.0.0 or higher:

You are also affected if you use the Object Storage Upload Manager:

Operations that do not upload streams are not affected.

If the stream that is being uploaded is a ByteArrayInputStream, operations are not affected.

To summarize, you are affected if you

  1. use version 3.0.0 or higher
  2. and upload streams using the above operations,
  3. and do not set a retry configuration at request level

Workarounds

Fixing this issue is our highest priority, and we are actively working on it. In the meantime, here are some possible workarounds:

  • Set the RetryConfiguration at request level by following this example.
    • The easiest way to do this is to insert the default retry policy at request level: 
      final PutObjectRequest putObjectRequest =
        PutObjectRequest.builder()
            .retryConfiguration(
                com.oracle.bmc.retrier.RetryConfiguration.SDK_DEFAULT_RETRY_CONFIGURATION)
            // other request parameters
            .build();
  • Disable default retries by following any of the methods described here.
  • Buffer the data in memory yourself by first copying it into a ByteArrayInputSteam, and then use that ByteArrayInputStream in your upload operation.

**Update

We had previously stated that there may be a potential data corruption issue. Our initial fear was that the first attempt may fail, and subsequent retries fail to reset the stream and therefore do not upload the entire stream again, leading to missing data. During careful evaluation, we determined that data corruption does not occur.

Instead, the first retry fails with an exception:

Caused by: java.lang.RuntimeException: Stream {} does not support mark/reset, retries do not work

This is still far from ideal, and we are working on fixing the problem, but a hard, visible failure is preferable to silent data corruption.