crowdsec-firewall-bouncer-iptables upgrade problem
merkleID opened this issue · 13 comments
What happened?
I manage 8 machines with crowdsec running, and on 5 out of 8 I got this problem:
sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
crowdsec-firewall-bouncer-iptables
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,656 kB of archives.
After this operation, 143 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://packagecloud.io/crowdsec/crowdsec/debian bullseye/main amd64 crowdsec-firewall-bouncer-iptables amd64 0.0.26 [3,656 kB]
Fetched 3,656 kB in 1s (4,881 kB/s)
(Reading database ... 97834 files and directories currently installed.)
Preparing to unpack .../crowdsec-firewall-bouncer-iptables_0.0.26_amd64.deb ...
Removed /etc/systemd/system/multi-user.target.wants/crowdsec-firewall-bouncer.service.
Unpacking crowdsec-firewall-bouncer-iptables (0.0.26) over (0.0.25) ...
Setting up crowdsec-firewall-bouncer-iptables (0.0.26) ...
Configuration file '/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** crowdsec-firewall-bouncer.yaml (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml ...
Installing new version of config file /etc/systemd/system/crowdsec-firewall-bouncer.service ...
cscli/crowdsec is present, generating API key
API Key successfully created
Created symlink /etc/systemd/system/multi-user.target.wants/crowdsec-firewall-bouncer.service → /etc/systemd/system/crowdsec-firewall-bouncer.service.
Job for crowdsec-firewall-bouncer.service failed because the control process exited with error code.
See "systemctl status crowdsec-firewall-bouncer.service" and "journalctl -xe" for details.
dpkg: error processing package crowdsec-firewall-bouncer-iptables (--configure):
installed crowdsec-firewall-bouncer-iptables package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
crowdsec-firewall-bouncer-iptables
E: Sub-process /usr/bin/dpkg returned an error code (1)
What did you expect to happen?
full upgrade
How can we reproduce it (as minimally and precisely as possible)?
it was a simple apt upgrade
Anything else we need to know?
No response
Crowdsec version
OS version
$cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux srv11222 5.15.102-1-pve crowdsecurity/crowdsec#1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z) x86_64 GNU/LinuxEnabled collections and parsers
$ cscli hub list -o raw
# paste output hereAcquisition config
On Windows:
C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
paste output here
Config show
$ cscli config show
Global:
- Configuration Folder : /etc/crowdsec
- Data Folder : /var/lib/crowdsec/data
- Hub Folder : /etc/crowdsec/hub
- Simulation File : /etc/crowdsec/simulation.yaml
- Log Folder : /var/log/
- Log level : info
- Log Media : file
Crowdsec:
- Acquisition File : /etc/crowdsec/acquis.yaml
- Parsers routines : 1
- Acquisition Folder : /etc/crowdsec/acquis.d
cscli:
- Output : human
- Hub Branch :
- Hub Folder : /etc/crowdsec/hub
Local API Server:
- Listen URL : 127.0.0.1:8080
- Profile File : /etc/crowdsec/profiles.yaml
- Trusted IPs:
- 127.0.0.1
- ::1
- Database:
- Type : sqlite
- Path : /var/lib/crowdsec/data/crowdsec.db
- Flush age : 7d
- Flush size : 5000Prometheus metrics
$ cscli metrics
FATA[08-05-2023 17:36:22] failed to fetch prometheus metrics : executing GET request for URL "http://127.0.0.1:6060/metrics" failed: Get "http://127.0.0.1:6060/metrics": dial tcp 127.0.0.1:6060: connect: connection refusedRelated custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
@merkleID: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
- Check Crowdsec Documentation to see if your issue can be self resolved.
- You can also join our Discord.
- Check Releases to make sure your agent is on the latest version.
Details
I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.
Moving issue over to relevant project
Hi @merkleID, thanks for the report
Could you run these (with sudo) and see what it says?
See "systemctl status crowdsec-firewall-bouncer.service" and "journalctl -xe" for details.
Hi, same issue here.
On my side the issue was that I have disabled LAPI.
The upgrade process want at all cost to generate an API Key on LAPI...
Once I have reactivate the LAPI on crowdsec I was able to finish the upgrade for the bouncer...
But I need to activate LAPI, upgrade the bouncer, deactivate again the LAPI...
Here are some logs of the upgrade process :
Setting up crowdsec-firewall-bouncer-iptables (0.0.26) ...
cscli/crowdsec is present, generating API key
INFO[09-05-2023 10:01:48] Patching yaml: '/etc/crowdsec/config.yaml' with '/etc/crowdsec/config.yaml.local'
FATA[09-05-2023 10:01:48] Local API is disabled, please run this command on the local API machine
failed to create API key
ERR: missing required variable value
dpkg: error processing package crowdsec-firewall-bouncer-iptables (--configure):
installed crowdsec-firewall-bouncer-iptables package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
crowdsec-firewall-bouncer-iptables
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
Note : the bouncer work fine if I do a systemctl restart crowdsec-firewall-bouncer...
But the upgrade process want to register on LAPI, even if I have the API key and API url configured in the bouncer config file (or in the .local file)...
[EDIT]
Oh, and I just discovered a new file in the bouncer directory "crowdsec-firewall-bouncer.yaml.id".
With the bouncer name generated on LAPI...
Will try to find another server with the issue, create this file with the name registered on my central LAPI...
[/EDIT]
[EDIT2]
More info, the process fail on some install where I have commented this part of config.yaml :
server:
log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
console_path: /etc/crowdsec/console.yaml
online_client: # Central API credentials (to push signals and receive bad IPs)
credentials_path: /etc/crowdsec/online_api_credentials.yaml
Adding -no-api in systemd don't cause any issue.
But at least whith this part not commented the process work fine, after creating a local bouncer... Even if the bouncer configuration with the .local is configured on another crowdsec instance on another server...
And the crowdsec-firewall-bouncer.id is created with the id of the local created bouncer, who has nothing to do with the bouncer id used on the "central LAPI"...
[/EDIT2]
Thanks @LtSich for the detailed explanation.
We are preparing a new release to deal with these cases, the .yaml.id file is used later when the package is uninstalled, it would unregister the bouncer if the configuration is removed too (i.e. apt purge) otherwise it's harmless. I can ping you to test a pre-release if you want.
By the way you said that when upgrading the package, if you already have an API_KEY it would generate a new one. I guess you accepted the new configuration file which reverts the generated key to ${API_KEY}. In this case the bouncer is registered a second time but it should work correctly.
The configuration file don't have any API key, but I have a .local file with everything needed.
I have try to manually add the api key and api url directly in the config file, but same result, the upgrade process want to register a local bouncer.
Note : I don't want that the upgrade process register a new bouncer... I have close to 50 bouncers registered on a "central" LAPI... Each one with a "real" name, I don't want to manage random name in my bouncers list :)
Of course we must take .yaml.local into account: if BACKEND or API_KEY are there, we won't touch the config.
Another thing: you said an issue was that LAPI was not running, but LAPI is not used to create an api key, because cscli writes it directly in the database.
As I say, the -no-api in the systemd file is not a problem.
But commenting (#) the server part in config.yaml break the upgrade process.
@merkleID: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
- Check Crowdsec Documentation to see if your issue can be self resolved.
- You can also join our Discord.
- Check Releases to make sure your agent is on the latest version.
Details
Hi,
here's the log
root@srv21934:/home/luca# journalctl -xe -u crowdsec-firewall-bouncer.service
May 10 15:19:50 srv21934 systemd[1]: crowdsec-firewall-bouncer.service: Scheduled restart job, restart counter is at 13340.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Automatic restarting of the unit crowdsec-firewall-bouncer.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
May 10 15:19:50 srv21934 systemd[1]: Stopped The firewall bouncer for CrowdSec.
░░ Subject: A stop job for unit crowdsec-firewall-bouncer.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit crowdsec-firewall-bouncer.service has finished.
░░
░░ The job identifier is 1699431 and the job result is done.
May 10 15:19:50 srv21934 systemd[1]: Starting The firewall bouncer for CrowdSec...
░░ Subject: A start job for unit crowdsec-firewall-bouncer.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit crowdsec-firewall-bouncer.service has begun execution.
░░
░░ The job identifier is 1699431.
May 10 15:19:50 srv21934 crowdsec-firewall-bouncer[883631]: time="2023-05-10T15:19:50+02:00" level=info msg="crowdsec-firewall-bouncer v0.0.26-debian-pragmatic-fa0e7d25b886aea927c1c3323ba854b5751fc3d1"
May 10 15:19:50 srv21934 crowdsec-firewall-bouncer[883637]: time="2023-05-10T15:19:50+02:00" level=info msg="crowdsec-firewall-bouncer v0.0.26-debian-pragmatic-fa0e7d25b886aea927c1c3323ba854b5751fc3d1"
May 10 15:19:52 srv21934 crowdsec-firewall-bouncer[883637]: time="10-05-2023 15:19:52" level=fatal msg="process terminated with error: stream api init failed"
May 10 15:19:52 srv21934 systemd[1]: crowdsec-firewall-bouncer.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit crowdsec-firewall-bouncer.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
May 10 15:19:52 srv21934 systemd[1]: crowdsec-firewall-bouncer.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit crowdsec-firewall-bouncer.service has entered the 'failed' state with result 'exit-code'.
May 10 15:19:52 srv21934 systemd[1]: Failed to start The firewall bouncer for CrowdSec.
░░ Subject: A start job for unit crowdsec-firewall-bouncer.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit crowdsec-firewall-bouncer.service has finished with a failure.
░░
░░ The job identifier is 1699431 and the job result is failed.
Thanks. In the linked PR the install script will avoid touching the config file if api_key is defined in .yaml.local. That should be ok for this case
0.0.27-rc1 is out, should fix that
Just updated Crowdsec, got this:
Setting up crowdsec-firewall-bouncer-iptables (0.0.28) ...
Configuration file '/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** crowdsec-firewall-bouncer.yaml (Y/I/N/O/D/Z) [default=N] ? d
--- /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml 2023-06-05 14:23:03.504134248 +0000
+++ /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.dpkg-new 2023-10-02 11:36:23.000000000 +0000
@@ -1,7 +1,5 @@
mode: iptables
-pid_dir: /var/run/
update_frequency: 10s
-daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
@@ -10,7 +8,7 @@
log_max_backups: 3
log_max_age: 30
api_url: http://127.0.0.1:8080/
-api_key: <my_current_api_key_here>
+api_key: ${API_KEY}
insecure_skip_verify: false
disable_ipv6: false
deny_action: DROP
@@ -55,6 +53,6 @@
anchor_name: ""
prometheus:
- enabled: true
+ enabled: false
listen_addr: 127.0.0.1
listen_port: 60601
Is this intended behavior? Can I safely let the system overwrite my current config file or should I keep it?
You can keep the old file, or replace it -- it will just create a new api_key when the service is run.
We had to disable metrics by default because on some systems they have a performance impact.
Note: we should find a clean way to move api_key to its own file.