Many entries seem to have been removed
Closed this issue · 12 comments
We make use of your OISD small block list, and have heard reports from many of our customers that the blocking is not as optimal as it once was. The optimal blocking was perhaps up until around 2-3 weeks ago.
From what I have seen the list of URLs blocked has reduced by about 50%, was there a reason for this?
Do you have any reference to which lists you removed, and why you removed them if so?
Thanks in advance
Which OISD small list? - If using domains / hosts syntax, please switch to a syntax that automatically block subdomains by default.
blocking is not as optimal as it once was
Examples please. What did it block before that it's not blocking now?
Do you have any reference to which lists you removed, and why you removed them if so?
I don't think I've alively removed lists. Most probably a list (or lists) have stopped being included because they started including domain(s) that should never be blocked, or stopped being included because it got too old.
I'll do some diggin to see what's going on..
Edit: I think it's because a/some blocklist(s) no longer got included because they are too old (over 1 year since last edit).
However; "too old" is subjective, so I took that logic out. No longer resolving domains get tossed out anyways and false positives get fixed.
Should be fixed in next update.
We use this list: https://dbl.oisd.nl/small/
I cannot give exact examples, I am just relaying what I have been informed of, since our customers do not always provide exact examples we do not always have a lot to go by.
Thanks for replying, I will return with any responses we get.
I suggest you switch to either https://small.oisd.nl/domainswild or https://big.oisd.nl/domainswild as I will eventually drop support for domains / hosts -syntax's.
Please note the header;
All domains ánd their subdomains should be blocked
Entry: "*.example.com" should block access to "example.com" and "subdomain.example.com" but not "thiseexample.com"
How soon do you plan to "drop support for domains / hosts -syntax's"? 1 month, 3, 12? Just as a reference point.
There was an explanation as to how and why on Reddit (which I left), but that also didn't include a date to when. I was thinking next newyear. But since there's no open info about it atm, I should either do that or extend it even further.
Thank you for your help on this so far.
I just refreshed this morning and found 37657 lines in https://dbl.oisd.nl/small/ compared to 58283 on 8th May 2023
Somewhat different, is the list different to the one you are referring to when you say "Should be fixed in next update."?
I think I "fixed" it... Making it grow to over 50K again. Then yesterday I had some server issues, had to restore some backup files and this change was restored to the pre-fixed version.
Will address this as soon as I get home from work.
I think I "fixed" it... Making it grow to over 50K again. Then yesterday I had some server issues, had to restore some backup files and this change was restored to the pre-fixed version. Will address this as soon as I get home from work.
Thank you ever so much!
And now it's up to about 250K-ish.. lol
Maybe some dead domains to be pruned out, not sure. it'll be fine.
We'll see what it does the coming days. Even if it stays like this
Not really what I had in mind for being "oisd small" but then again .. this highlights the issue of the domains/hosts syntax as to why support will be dropped for them in the future.
It seems like the subdomains werent propely added anymore.
Thanks for your help on this!
I was using the "basic" list, which is now being referred to as the "small" list. Except, the amount of entries on the small list has doubled from what the basic list used to provide. What changed? I'd like fewer entries on the small list if possible. It's at ~222K at the time of me writing this, ~150K or lesser would be better.
Subdomains from all lists included in oisd big were added to small, making that grow to 225-ish? Now only subdomains from the lists included in oisd small are added .. making the current count; 57715
... I "fixed" it... Making it grow to over 50K again. ...
Pretty much that was the correct fix back then.