Using up disk space
Opened this issue · 0 comments
Following updating ruby to 2.7.7 (first step before updating to ruby 3) and ubuntu to 22.04, I am experiencing problems with my disk filling up, and have tracked this down to sneakers. Note that the ruby update does cause a deprecation warning for syntax deprecated in ruby 3 in our application that has caused our log files to be larger than usual - but I've adjusted logrotate for that, and those changes are keeping the sizes reasonable. It isn't our application logs that are filling up the disk space. It is something that clears the disk space when sneakers is restarted.
Here is what I know so far:
df shows 14G used
deployer@ip-x-x-x-x:/$ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 14G 16G 47% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 960K 1.6G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/nvme0n1p15 105M 6.1M 99M 6% /boot/efi
tmpfs 784M 4.0K 784M 1% /run/user/1000
du show 7.5G used
deployer@ip-x-x-x-x:/$ sudo du -sh
7.5G
After seeing this: https://serverfault.com/a/581521 in which apache caused a similar problem for someone, I tried restarting various services running. Restarting sneakers cleared 8.4G
deployer@ip-x-x-x-x:/$ sudo service sneakers restart
deployer@ip-x-x-x-x:/$ sudo service sneakers status
● sneakers.service - SYSV: Starts and Stops Sneakers message processor.
Loaded: loaded (/etc/init.d/sneakers; generated)
Active: active (exited) since Wed 2024-02-14 14:39:23 UTC; 40s ago
Docs: man:systemd-sysv-generator(8)
Process: 128372 ExecStart=/etc/init.d/sneakers start (code=exited, status=0/SUCCESS)
CPU: 37ms
deployer@ip-x-x-x-x systemd[1]: Starting SYSV: Starts and Stops Sneakers message processor....
deployer@ip-x-x-x-x sneakers[128372]: Starting sneakers message processor ..
deployer@ip-x-x-x-x su[128378]: pam_unix(su-l:session): session opened for user deployer(uid=1000) by (uid=0)
deployer@ip-x-x-x-x systemd[1]: Started SYSV: Starts and Stops Sneakers message processor..
deployer@ip-x-x-x-x: /$ sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 5.6G 24G 20% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 960K 1.6G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/nvme0n1p15 105M 6.1M 99M 6% /boot/efi
tmpfs 784M 4.0K 784M 1% /run/user/1000
So restarting sneakers cleared 8.4 GB of diskspace on my server. Prior to this, sneakers had been running for 2 days (when I had done an instance refresh for our servers).
We are running sneakers 2.7.0, with bunny 2.9.2.
sneakers.rb
require 'sneakers'
connection = Bunny.new(Settings.rabbit.to_h)
Sneakers.configure(
heartbeat: 5,
connection: connection,
vhost: '/',
exchange: 'sneakers',
exchange_type: :direct,
arguments: { :'x-queue-type' => 'quorum' },
daemonize: true,
start_worker_delay: 10,
pid_path: tmp/pids/sneakers.pid
log: log/sneakers.log
timeout_job_after: 30,
prefetch: 10,
threads: 10,
env: Rails.env
)
Sneakers.logger.level = Logger::INFO
Sneakers.error_reporters << proc { |exception, _worker, context_hash| Honeybadger.notify(exception, context_hash) }
settings.yml relevation sections:
rabbit:
host: <%= ENV['RABBITMQ_HOST'] %>
port: <%= ENV['RABBITMQ_PORT'] %>
user: <%= ENV['RABBITMQ_USERNAME'] %>
password: <%= ENV['RABBITMQ_PASSWORD'] %>
automatically_recover: true
tls: true
tls_cert: <%= ENV['TLS_CLIENT_CERT'] %>
tls_key: <%= ENV['TLS_CLIENT_KEY'] %>
tls_ca_certificate: <%= ENV['TLS_CA_CERT'] %>
verify_peer: false
fail_if_no_peer_cert: false
queue:
event: <%= "#{Rails.env}_events" %>
error: <%= "#{Rails.env}_errors" %>
I am not sure what in sneakers is using up the disk space - but something in it or controlled by it (such as bunny) is using space that is cleared on restarting the sneakers service.
If anyone has any insight or suggestions, I'd greatly appreciate it.