Investigate categorify.org filtering via API

[mrbluecoat]

Ref #2

[luigi1809]

I did not notice categorify.org had a limit of 200 requests per day. Even with cache, it should not be enough.

Good work https://mrbluecoat.blogspot.com/2019/11/url-filtering-services.html

So, webshrinker API would be a better option.

However, I noticed that DNSfilter.com uses webshrinker API. So we should better use this.

The problem with paid DNS filtering service is that we need a fixed IPv4. May be another option is to build its own DNS filter server using webshrinker API. Need to see how to make this

[mrbluecoat]

DNSFilter only offers DNSSEC and DNS-over-TLS on their Pro plan. Their FAQ says

We have a minimum charge of $50 per month on the Pro Plan which includes your first 25 users.

So it's not ideal for home use.

[luigi1809]

Cleanbrowsing is cheaper.

I am working on API option in webfilter-ng.

But the better option would be to have an opensource dns server that do filtering based on API

[luigi1809]

@mrbluecoat Please test new dns_categorify option

[luigi1809]

Feature is implemented. Feel free to re-open if needed

[mrbluecoat]

Thanks @luigi1809! I'll test soon and re-open if needed.

[luigi1809]

Thanks @luigi1809! I'll test soon and re-open if needed.

Please give me your feedback about the feature.

I will implement support for webshrinker as well

[mrbluecoat]

I got it compiled but honestly I'm still trying to figure out how to test it effectively. Good sites work and bad sites are blocked but I'm not sure if that's just the Clean Browsing DNS blocking working.

[mrbluecoat]

I'm using my knot-resolver setup described in #4 (comment) so I'm not sure if that's also a factor (Bind is a bit heavyweight for my tastes)

[mrbluecoat]

Also, I don't see anything in /var/log/webfilter-ng
or /var/cache/webfilter-ng so that also leads me to think I'm doing something wrong.

[mrbluecoat]

Can you add the "Lingerie" category to the block? That will make it easier to test with sites like victoriassecret

[mrbluecoat]

Okay, I don't think it's due to BIND. Here's my test with your instructions:

apt install -y bind9 bind9utils bind9-doc dnsutils
cd ~
wget https://github.com/luigi1809/webfilter-ng/archive/master.zip
unzip master.zip
cp webfilter-ng-master/bind/* /etc/bind/
wget --user=ftp --password=ftp ftp://ftp.rs.internic.net/domain/db.cache -O /etc/bind/db.root

I edited bind/db.rpz to add this entry:
duckduckgo.com CNAME safe.duckduckgo.com.

I edited named.conf.options with these changes

dnssec-enable yes;
dnssec-validation auto;

192.168.2.1 --> changed to my device IP: $(hostname -I | cut -d' ' -f1)

forwarders { 185.228.168.168; 185.228.169.168; };

I then restarted BIND and tested:

dig @127.0.0.1 google.com +short | grep forcesafesearch
dig @$(hostname -I | cut -d' ' -f1) google.com +short | grep forcesafesearch
dig @127.0.0.1 duckduckgo.com +short | grep safe
dig @127.0.0.1 sigok.verteiltesysteme.net +dnssec | grep NOERROR
dig @127.0.0.1 sigfail.verteiltesysteme.net +dnssec | grep SERVFAIL
dig @127.0.0.1 badexample.com | grep NXDOMAIN

So far so good -- I have BIND running correctly on localhost.

I then compiled and enabled webfilter-ng according to your instructions:

cd ~/webfilter-ng-master

apt install -y build-essential git dnsutils iptables iptables-persistent netfilter-persistent libnetfilter-queue-dev libnetfilter-queue1 libnfnetlink-dev libnfnetlink0 nghttp2 redis-server jq

make

cp -p /etc/nghttpx/nghttpx.conf /etc/nghttpx/nghttpx.conf.save

cat>/etc/nghttpx/nghttpx.conf <<\EOF
frontend=127.0.0.1,3000;no-tls
backend=categorify.org,443;;tls
errorlog-syslog=no
backend-keep-alive-timeout=300
insecure=yes
workers=1
EOF

systemctl restart nghttpx.service
systemctl enable nghttpx.service

systemctl restart redis-server.service
systemctl enable redis-server.service

make dns_categorify

make install

iptables -A FORWARD -i eth0 -o eth0 -p tcp -j ACCEPT
iptables -A OUTPUT -p udp -m udp --dport 443 -j DROP
iptables -A FORWARD -i eth0 -p tcp -m tcp -j NFQUEUE --queue-num 200
iptables-save > /etc/iptables/rules.v4
systemctl start netfilter-persistent.service
systemctl enable netfilter-persistent.service

iptables -t nat -A PREROUTING -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.1
iptables-save > /etc/iptables/rules.v4
systemctl restart netfilter-persistent.service

ip6tables -I FORWARD -i eth0 -j REJECT
ip6tables-save > /etc/iptables/rules.v6
systemctl restart netfilter-persistent.service

I tested curl -v -L http://badexample.com but it wasn't using the local DNS server so I had to force it via:

echo "nameserver 127.0.0.1" > /etc/resolv.conf
sed -i 's/#prepend\sdomain-name-servers.*/prepend domain-name-servers 127.0.0.1;/' /etc/dhcp/dhclient.conf
> /etc/resolvconf/resolv.conf.d/head
> /etc/resolvconf/resolv.conf.d/tail

Then it started working as expected:

curl -v -L http://google.com
curl -v -L http://badexample.com

But I still don't see anything in the logs:

ll /var/log/webfilter-ng
ls: cannot access '/var/log/webfilter-ng': No such file or directory

ll /var/cache/webfilter-ng/
total 0

Thoughts?

[luigi1809]

iptables -A FORWARD -i eth0 -o eth0 -p tcp -j ACCEPT
iptables -A OUTPUT -p udp -m udp --dport 443 -j DROP
iptables -A FORWARD -i eth0 -p tcp -m tcp -j NFQUEUE --queue-num 200

FORWARD is for filtering routing packet from one interface to another. You need OUTPUT

ip=$(getent ahostsv4 categorify.org | awk '{ print $1 }' | head -n1)
echo $ip
iptables -A OUTPUT -d $ip -j ACCEPT
iptables -A OUTPUT -p tcp -j NFQUEUE --queue-num 200

[mrbluecoat]

Thanks! Clearly iptables are not my strong suit ;)

[mrbluecoat]

For others that find this thread, https://www.forwardproxy.com/2019/01/using-nfqueue-to-build-your-own-firewall/ helped me understand the networking fundamentals