StevenBlack/hosts

[Funceble] Result of execution

Closed this issue · 11 comments

funilrys commented

Hello @StevenBlack ,

As you (may) know, I released and still continue to develop Funceble.

I wanted to help, so you can find here the result of my last execution against the following list.

As it's marked as "occasionally" for the update frequency,
I also published this message/issue to AdAway's repository .

adaway.github.io@AdAway ==> AdAway ❌

Have a nice day/night.

Cheers,

P.S: I plan to test all list of your "Sources of hosts data unified in this variant" section.

StevenBlack commented

Hi Nissar @funilrys thanks for this.

Help me understand, can you summarize what does Funceble does? Are you suggesting any action is required here? Thanks for clarifying.

funilrys commented

Hi @StevenBlack,
Funceble test if a domain or a list of domain is/are still active!

Here's a short summary:

  • ACTIVE
    • Date of expiration accessible per WHOIS-Server (whois command)
    • nslookup don't return "server can't find domain-name.me: NXDOMAIN"
  • INACTIVE
    • whois server don't return anything or date is unreadable
    • nslookup return "server can't find domain-name.me: NXDOMAIN"
  • INVALID
    • Domain extension has an invalid format or is unregistered into IANA Root Zone Database.

You simply can exclude/remove the domains under the INACTIVE directory of each list like @mitchellkrogza already did for his list 😄

[EDIT]

What does funceble do ?

It's a script that test if your host list are dead or not!

StevenBlack commented

@funilrys In your experience... I wonder: how is "inactive" prone to Type II errors – a "false negative"?

I'm guessing the whois test might not be great, since all whois servers practice some form of rate-limiting on requests.

funilrys commented

@StevenBlack That's why I test first whois then nslookup. I don't rest only on whois.

To make it simple
Whois true && nslookup true ===> active
Whois false && nslookup true ===> active
Whois true && nslookup false ===> inactive
whois false && nslookup false ==> inactive

Do you think my decisions/logics are incompatible with my idea of checking a domain ?

Thanks for pointing out the Type II errors, you're right...
I think I have to find a third way in order to be sure...

FadeMind commented

@funilrys @StevenBlack IMO good idea is MERGE this functionality directly in updateHostsFile.py code but main code is now in BASH: https://github.com/funilrys/funceble/blob/master/funceble

funilrys commented

@FadeMind I don't know yet if I'm gonna write a python version but like @mitchellkrogza like to say, I'm a bash basher 😄

funilrys commented

In between @FadeMind, It's not a good idea at all to merge it cause:

  • the file you're talking about gonna take too much time to execute if I implement it to it.
    • updateHostsFile.py don't read the domains one by one.
funilrys commented

Hello Steven @StevenBlack,

I may not have the experience you have as you previously said but I wanted to update the representation of the logic of funceble which is much clear now than the one I gave you in March.

I wanted a short expert opinion so can you tell me what you think about it?

Thank you.

StevenBlack commented

Hi Nissar @funilrys

This seems like a good start. It appears like a reasonable approach. But I think whois is the weak link in all this. Not much can be done about that. Maybe the flow should bypass whois if it times out or returns an error.

  • 👍1
mitchellkrogza commented

@StevenBlack having very good results with funceble on this repo - https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites and run it against very large lists with over 5900 entries and it seems to work like a charm. @funilrys been doing some great work on this and I am sure over time he will perfect it even more.

  • 👍2
funilrys commented

Thanks for the concrete answer Steven @StevenBlack.

Funceble already has an integrated time out function written for all call of whois which is set by default to 30 seconds but can be changed by the user with the -t argument. For example, for the big list like The Badd-Boyz-Hosts the time out under dead-hosts is set to 1 second instead of 30 seconds.

For the error part, I assume that if we don't match the expiration date there is an error but I may have to review this part...

Thanks for the support Mitchell @mitchellkrogza.

Be sure that funceble will continue to be developed in order to be more efficient and perfect!

Thanks again to both of you.

Have a nice day/night

Cheers,
Nissar