CLP Log Ingestion Notification Monitor.
- Create and enter the virtual environment:
python3.x -m venv v3x; . ./v3x/bin/activate - Update
pip:pip install --upgrade pip - Install the project in editable mode along with the development dependencies:
pip install -e .[dev]
To ingest files directly via the S3 API:
python start.py \
--seaweed-filer-endpoint $SEAWEEDFS_ENDPOINT \
--seaweed-s3-endpoint-url $SEAWEEDFS_S3_GATEWAY \
--filer-notification-path-prefix $NOTIFICATION_PATH_PREFIX \
--db-uri $DATABASE_URI \
--max-buffer-size 16777216 \
--min-refresh-frequency 5000 \
s3To ingest files via mounted directories in the local filesystem:
python start.py \
--seaweed-filer-endpoint $SEAWEEDFS_ENDPOINT \
--seaweed-s3-endpoint-url $SEAWEEDFS_S3_GATEWAY \
--filer-notification-path-prefix $NOTIFICATION_PATH_PREFIX \
--db-uri $DATABASE_URI \
--max-buffer-size 16777216 \
--min-refresh-frequency 5000 \
fs --seaweed-mnt-prefix $SEAWEED_MNT_PREFIXTo introduce job grouping by folder names under a specific path prefix, add the following option:
--group-by-paths-under-prefix $GROUP_BY_PATH_UNDER_PREFIXBefore submitting a pull request, run the following error-checking and formatting tools:
- mypy:
mypy src- mypy checks for typing errors. You should resolve all typing errors or if an
error cannot be resolved (e.g., it's due to a third-party library), you
should add a comment
# type: ignoreto silence the error.
- mypy checks for typing errors. You should resolve all typing errors or if an
error cannot be resolved (e.g., it's due to a third-party library), you
should add a comment
- docformatter:
docformatter -i src- This formats docstrings. You should review and add any changes to your PR.
- Black:
black src- This formats the code according to Black's code-style rules. You should review and add any changes to your PR.
- ruff:
ruff check --fix src- This performs linting according to PEPs. You should review and add any changes to your PR.
Note that docformatter should be run before black to give Black the last.