Modify SWODLR component logs to support Cloud Metrics dashboards
Closed this issue · 9 comments
5/1/24: Additional request from viviant (5/1/24): Capture cycle, pass and scene in the logs to support Cloud Metrics dashboards.
4/23/24:
We should modify the following components to support Cloud Metrics dashboards based on OPS team feedback.
SWODLR-APP Logs (service-swodlr-{venue}/service-swodlr-{venue}-app) and/or SWODLR-PROXY Logs (service-swodlr-{venue}/service-swodlr-{venue}-proxy)
- Capture API request, user_ip, EDL username.
Raster Create - Preflight (/aws/lambda/service-swodlr-{venue}-raster-create-preflight)
- pixc_grq : Name of granule (one log statement per granule)
- orbit_grq : Name of orbit file
- pixc_cmr : Name of granule (one log statement per granule)
- orbit_cmr : Name of orbit file
- to_ingest : Name of granule to ingest (one log statement per granule)
- to_delete : Name of granule to delete (one log statement per granule)
Raster Create - Wait For Complete (/aws/lambda/service-swodlr-{venue}-raster-create-wait_for_complete)
- sds.job_stage : Log the job stage (e.g., evaluate or create_raster)
Raster Create Submit Evaluate (/aws/lambda/service-swodlr-{venue}-raster-create-submit_evaluate)
- job_stage
- eval_product_id
- eval_job_id
- eval_job_status
Raster Create Submit Raster (/aws/lambda/service-swodlr-{venue}-raster-create-submit_raster)
- job_stage
- raster_product_id
- raster_job_id
- raster_job_status
Google Sheet field mappings. Yellow indicates modifications required.
It looks like errors are not being captured in the Cloud Metrics platform. According to the Cloud Metrics team this may be because the tracebacks captured by a field are too long.
It would be good to log the error message independent of the traceback and capture that as a field in the Cloud Metrics platform. I have updated the Google Sheet to reflect the need for error messages.
PR opened for raster-create changes - podaac/swodlr-raster-create#30
Changes requested for raster-create/ingest-to-sds regarding error logging would need to be handled in a separate ticket as it requires an overhaul of our error handling
Add additional request to capture cycle, pass and scene in the logs to support Cloud Metrics dashboards. Also updated in the original ticket description.
PR opened for raster-create changes - podaac/swodlr-raster-create#30
Changes requested for raster-create/ingest-to-sds regarding error logging would need to be handled in a separate ticket as it requires an overhaul of our error handling
@joshgarde Could you add the separate ticket number here for reference?
Add additional request to capture cycle, pass and scene in the logs to support Cloud Metrics dashboards. Also updated in the original ticket description.
CPS has been captured as part of the full input parameter dump in the SDS metrics log message. Lmk if that's insufficient
Add additional request to capture cycle, pass and scene in the logs to support Cloud Metrics dashboards. Also updated in the original ticket description.
CPS has been captured as part of the full input parameter dump in the SDS metrics log message. Lmk if that's insufficient
Could you provide an example of what full input parameter dump look like?
The log messages look like this
[INFO] 2024-05-02T02:26:32.322Z 71405e69-65f0-4c10-8dd2-31a485ab1f46 [product_id: dddc01df-ce80-4f7e-9882-d6323cb6b852, job_id: 95c91f57-1dd2-4b1e-86c7-9e48af6cac71] SDS metrics: {"metrics": {"time_queued": "2024-05-02T01:37:15.106528Z", "time_start": "2024-05-02T01:38:11.371278Z", "time_end": "2024-05-02T02:25:47.097290Z"}, "input": {"product_id": "dddc01df-ce80-4f7e-9882-d6323cb6b852", "output_granule_extent_flag": false, "output_sampling_grid_type": "UTM", "raster_resolution": 1000, "utm_zone_adjust": 0, "mgrs_band_adjust": 0, "cycle": 9, "pass": 39, "scene": 113}}
PR opened for raster-create changes - podaac/swodlr-raster-create#30
Changes requested for raster-create/ingest-to-sds regarding error logging would need to be handled in a separate ticket as it requires an overhaul of our error handling@joshgarde Could you add the separate ticket number here for reference?