SyncAwareBucket: improve strategy to verify file upload
Closed this issue · 0 comments
ckunki commented
Currently SyncAwareBucket
validates upload of files by searching for time stamp in the log file e.g. /exa/logs/cored/bucketfsd.*.log
considering the following constraints
- adjust for potentially different time zone settings
- repeated upload to the same file path, overwriting older versions of the file
- wait until COS has synchronized the file to all nodes of the cluster
- accuracy of log file timestamps is only 1 second.
This requires some prerequisites to be met
- switch off log rotation to ensure to have only a single log file
- ensure sufficient permissions to write and read log files
Sample error message in case SyncAwareBucket
fails to verify successful upload of a file:
Timeout waiting for object 'replace_me.txt' to be synchronized
in bucket 'bfsdefault/default' after 2022-08-11T06:52:22.312Z.
To handle repeated upload to the same file path, overwriting older versions of the file:
exasol-testcontainers
maintains a list of files uploaded in the current session to identify repeated uploads to the same file path.- In this case
SyncAwareBucket
needs to identify the latest upload. - As accuracy of log file timestamps is only 1 second
SyncAwareBucket
in edge cases waits for a second.
The current ticket proposes to change SyncAwareBucket
:
- to not look for timestamps anymore
- but rather to remember the line number in the log file.
This could simplify the detection and remove some constraints and potential mistakes
- time zone settings
- accuracy of log file timestamps