rfxn/linux-malware-detect

Proxy/cache maldet.sigs.ver and other artefacts

brianjmurrell opened this issue · 2 comments

brianjmurrell commented

We seem to somewhat frequently hit issues downloading from the CDN. For example, just last night:

# tar -C /var/tmp/lmd_src --strip-components=1 -xf /var/tmp/maldetect-current.tar.gz
# pushd /var/tmp/lmd_src
# ./install.sh
/var/tmp/lmd_src ~
Created symlink from /etc/systemd/system/multi-user.target.wants/maldet.service to /usr/lib/systemd/system/maldet.service.
Linux Malware Detect v1.6.4
            (C) 2002-2019, R-fx Networks <proj@r-fx.org>
            (C) 2019, Ryan MacDonald <ryan@r-fx.org>
This program may be freely redistributed under the terms of the GNU GPL
 
installation completed to /usr/local/maldetect
config file: /usr/local/maldetect/conf.maldet
exec file: /usr/local/maldetect/maldet
exec link: /usr/local/sbin/maldet
exec link: /usr/local/sbin/lmd
cron.daily: /etc/cron.daily/maldet
maldet(11372): {sigup} performing signature update check...
maldet(11372): {sigup} local signature set is version 201907043616
maldet(11372): {sigup} could not download https://cdn.rfxn.com/downloads/maldet.sigs.ver, please try again later.
maldet(11372): {sigup} could not download signature data from server, please try again later.
 
# popd
~
# /usr/local/sbin/maldet --update-sigs
Linux Malware Detect v1.6.4
            (C) 2002-2019, R-fx Networks <proj@rfxn.com>
            (C) 2019, Ryan MacDonald <ryan@rfxn.com>
This program may be freely redistributed under the terms of the GNU GPL v2
 
maldet(11556): {sigup} performing signature update check...
maldet(11556): {sigup} local signature set is version 201907043616
maldet(11556): {sigup} could not download https://cdn.rfxn.com/downloads/maldet.sigs.ver, please try again later.
maldet(11556): {sigup} could not download signature data from server, please try again later.
Provisioning step had errors: Running the cleanup provisioner, if present...
Deleting output directory...

We need to be able to insulate ourselves from these kinds of external outages even if it means using a slightly (likely not even) outdated maldet.sigs.ver et al.) . While #363 is quite relevant once maldet is installed, how can the install.sh be told to fetch the artefacts it needs to download (i.e. maldet.sigs.ver, and whatever else might be downloaded) from a local server/service which we keep updated from the upstream https://cdn.rfxn.com/downloads?

This would also actually ease load on the upstream CDN as we would/could update our mirror much less frequently than we run maldet in our CI (which is probably hundreds of times per day).

brianjmurrell commented

So, I was able to work this out. Simply after unpacking the tarball, but before running install.sh one has to do:

sed -i -e '/^base_domain=/s/".*"/"$local_url"/' files/internals/internals.conf

Where one replaces $local_url with one's local mirror of the upstream files:

maldet.sigs.ver
maldet-sigpack.tgz
maldet-sigpack.tgz.md5
maldet-cleanv2.tgz
maldet-cleanv2.tgz.md5
maldet-clean.tgz.md5

which one has to of course set up a local mirror of.

It would be nice not to have to do this quite so hackily though. An argument to install.sh to specify a local URL would be nicer.

rfxn commented

This is a growing timing / capacity issue and has been resolved by moving the CDN into Cloudflare.