exasol/bucketfs-java

SyncAwareBucket: improve strategy to verify file upload

Closed this issue · 0 comments

Currently SyncAwareBucket validates upload of files by searching for time stamp in the log file e.g. /exa/logs/cored/bucketfsd.*.log considering the following constraints

  • adjust for potentially different time zone settings
  • repeated upload to the same file path, overwriting older versions of the file
  • wait until COS has synchronized the file to all nodes of the cluster
  • accuracy of log file timestamps is only 1 second.

This requires some prerequisites to be met

  • switch off log rotation to ensure to have only a single log file
  • ensure sufficient permissions to write and read log files

Sample error message in case SyncAwareBucket fails to verify successful upload of a file:

Timeout waiting for object 'replace_me.txt' to be synchronized 
in bucket 'bfsdefault/default' after 2022-08-11T06:52:22.312Z.

To handle repeated upload to the same file path, overwriting older versions of the file:

  • exasol-testcontainers maintains a list of files uploaded in the current session to identify repeated uploads to the same file path.
  • In this case SyncAwareBucket needs to identify the latest upload.
  • As accuracy of log file timestamps is only 1 second SyncAwareBucket in edge cases waits for a second.

The current ticket proposes to change SyncAwareBucket:

  • to not look for timestamps anymore
  • but rather to remember the line number in the log file.

This could simplify the detection and remove some constraints and potential mistakes

  • time zone settings
  • accuracy of log file timestamps