couchbase/sync_gateway

Rotated syncgateway logs are not getting deleted after maxage

ArihantRk opened this issue · 2 comments

ArihantRk commented

Sync Gateway version

2.1.2

Operating system

RedHat 7.

Config file

"logging": {
"log_file_path": "/DGlogs/syncgw/",
"console": {
"enabled": false,
"log_level": "info",
"log_keys": ["*"],
"color_enabled": false
},
"error": {
"enabled": true,
"rotation": {
"max_size": 20,
"max_age": 180,
"localtime": true
}
},
"warn": {
"enabled": true,
"rotation": {
"max_size": 20,
"max_age": 90,
"localtime": true
}
},
"info": {
"enabled": true,
"rotation": {
"max_size": 20,
"max_age": 3,
"localtime": true
}
},
"debug": {
"enabled": false,
"rotation": {
"max_size": 20,
"max_age": 1,
"localtime": true
}
}
},

Log output

-rw-r--r-- 1 sync_gateway kodiakgroup 366516 Nov 16 13:25 sg_debug-2019-11-16T18-25-31.133.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 322270 Nov 16 13:25 sg_debug-2019-11-16T18-25-40.736.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 295673 Nov 16 13:25 sg_debug-2019-11-16T18-25-47.819.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 345870 Nov 16 13:25 sg_debug-2019-11-16T18-25-56.540.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 604878 Nov 16 13:26 sg_debug-2019-11-16T18-26-02.553.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 283074 Nov 16 13:26 sg_debug-2019-11-16T18-26-09.344.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 473214 Nov 16 13:26 sg_debug-2019-11-16T18-26-17.012.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 332150 Nov 16 13:26 sg_debug-2019-11-16T18-26-27.133.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 582715 Nov 16 13:26 sg_debug-2019-11-16T18-26-34.660.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 338708 Nov 16 13:26 sg_debug-2019-11-16T18-26-41.893.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 699671 Nov 16 13:26 sg_debug-2019-11-16T18-26-52.821.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 291969 Nov 16 13:26 sg_debug-2019-11-16T18-26-58.312.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 395527 Nov 16 13:27 sg_debug-2019-11-16T18-27-10.604.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 523765 Nov 16 13:27 sg_debug-2019-11-16T18-27-20.245.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 389764 Nov 16 13:27 sg_debug-2019-11-16T18-27-28.266.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 295705 Nov 16 13:27 sg_debug-2019-11-16T18-27-34.730.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 401512 Nov 16 13:27 sg_debug-2019-11-16T18-27-44.710.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 347873 Nov 16 13:27 sg_debug-2019-11-16T18-27-52.615.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 308303 Nov 16 13:27 sg_debug-2019-11-16T18-27-58.055.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 767027 Nov 16 13:28 sg_debug-2019-11-16T18-28-01.839.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 404102 Nov 16 13:28 sg_debug-2019-11-16T18-28-09.609.log.gz
-rw-r--r-- 1 sync_gateway kodiakgroup 344746 Nov 16 13:28 sg_debug-2019-11-16T18-28-17.530.log.gz

Current Date : Thu Dec 5 01:13:25 EST 2019

Expected behavior

Rotated syncgateway logs files should be deleted after max age

Actual behavior

Rotated syncgateway log files are still exists in log_file_path

Steps to reproduce

  1. Add above log file config details in syncgateway config and start the syncgateway service
  2. Verify rotated log files are getting deleted after maxage
bbrks commented

Hi @ArihantRk

It's because the debug output is disabled, and so log rotation handling (including cleanup) is not being run for that output.

You can force a log rotation to happen for all outputs at once, even if they're disabled by sending a SIGHUP signal to the Sync Gateway process like follows:

✔ /private/tmp/sglog-test-1
10:23 $ ls -l
.rw-r--r--  98k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-19.589.log.gz
.rw-r--r--  74k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-47.441.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-48.647.log.gz
.rw-r--r--  71k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-49.912.log.gz
.rw-r--r--  72k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-51.240.log.gz
.rw-r--r--  72k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-52.845.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-54.552.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-56.694.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-58.293.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:20 sg_debug-2019-12-03T10-20-59.705.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:21 sg_debug-2019-12-03T10-21-01.766.log.gz
.rw-r--r--  73k benbrooks  5 Dec 10:21 sg_debug-2019-12-03T10-21-03.766.log.gz
.rw-r--r--  74k benbrooks  5 Dec 10:21 sg_debug-2019-12-03T10-21-06.303.log.gz
.rw-r--r-- 726k benbrooks  5 Dec 10:21 sg_debug.log
.rw-r--r--  10k benbrooks  5 Dec 10:21 sg_stats.log

$ ps | grep sync_gateway
58551 ttys001    0:37.75 ./sync_gateway sg_config.json
$ kill -SIGHUP 58551

2019-12-05T10:24:31.261Z [INF] Handling signal: hangup
2019-12-05T10:24:31.261Z [INF] Rotating log files...

✔ /private/tmp/sglog-test-1
10:25 $ ls -l
.rw-r--r--  50k benbrooks  5 Dec 10:24 sg_debug-2019-12-05T10-24-31.261.log.gz
.rw-r--r--    0 benbrooks  5 Dec 10:24 sg_debug.log
.rw-r--r--    0 benbrooks  5 Dec 10:24 sg_error.log
.rw-r--r--    0 benbrooks  5 Dec 10:24 sg_info.log
.rw-r--r-- 1.5k benbrooks  5 Dec 10:24 sg_stats-2019-12-05T10-24-31.262.log.gz
.rw-r--r--    0 benbrooks  5 Dec 10:24 sg_stats.log
.rw-r--r--    0 benbrooks  5 Dec 10:24 sg_warn.log
ArihantRk commented

@bbrks
We have added this as ExecStartPost part of sycngateway start process in systemd.
Sometime we are getting below error log while sending SIGHUP to sync_gateway process.
2019-12-11T13:53:20.709-06:00 [INF] Handling SIGHUP signal.
2019-12-11T13:53:20.710-06:00 [INF] Rotating log files...
2019-12-11T13:53:20.710-06:00 [WRN] Error rotating : nil FileLogger -- rest.HandleSighup() at config.go:927