/hctl

Ultimate tool to manage (transform, merge, remove duplicates, validate) hostlists written in rust!

Primary LanguageRustBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

    __            __     __
   / /_   _____  / /_   / /
  / __ \ / ___/ / __/  / / 
 / / / // /__  / /_   / /  
/_/ /_/ \___/  \__/  /_/   
Repo Size Lines of code Last Commit Assets Downloads

About

Hostlists tools cli hctl is app written in rust to manage (merge, transform, clear from comments etc.) your hostlists from diffrent formats to selected by you!

Prerequisites, dependencies

openssl

  • *nix systems (linux, BSD, mac): should be already there, if not install it from your package repo like brew install openssl@3
  • windows: install from choco or vcpkg or whatever other way that won't welcome viruses.

Note that you can get rid of openssl dependency if you want, by building from source with modified dependencies, see building from source.

Downloads

As a package

Packaging status

OBS package build status

OBS package build status

Available:

  • Alpine Linux edge: apk add hctl --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing/ (from testing repository)
  • OpenSUSE Linux 15.6/Tumbleweed: opi hctl (using opi community repo helper)
  • Arch Linux (based): yay -Su hctl (using AUR helper)

Packaging progress issue

❗ BUILD REGRESSION

OpenBSD and DragonflyBSD doesn't have rust tier 1 support and their repo version is too old to compile on stable

PlatformDownloadsBuild Status
OS
Linux glibc
Linux musl
MacOS X
Windows
FreeBSD
NetBSD
Illumos
amd64 aarch64
app, sha256 app, sha256
app, sha256 app, sha256
app, sha256 app, sha256
app, sha256 app, sha256
app, sha256 NSU
app, sha256 NSU
app, sha256 NSU
amd64 aarch64
Build Status Build Status
Build Status Build Status
Build Status Build Status
Build Status Build Status
Build Status NSU
Build Status NSU
Build Status NSU
  • NSU - Not Set Up Yet, may (or not) appear in the future

Features

  • Merge hostlist
  • Remove duplicates
  • Remove commented lines (default behaviour)
  • Fetch lists directly from network (https)
  • YAML config
  • Parallel processing
  • Rejected lines may be saved with information about source list
  • Support for Windows encoded text files (line ending with \n\r)
  • Output to stdout to be used with unix pipe stream control (partial support)
  • Whitelists (enumerated, from external lists file/https)
  • Whitelists including subdomains
  • Record existance validation from DNS servers (unencrypted and over tls)
  • User-defined DNS and DOT Resolvers
  • Could use diffrent input types (like dnsmasq, bind) in yaml config
  • Interacive status indication
  • Settings yaml could be fetched from http(s)
  • Auto scheme recognition
  • YAML has full settings (remote sources, whitelist , args from cli)
  • Subdomains family recognition, save with subdomain order (like adaway does)
  • Fault-tolerance, better error handling
  • Deep rejected analysis
  • Quiet option
  • Progressbar on save

Jump here to see supported input and output formats

Jump here to feature-request issue.

# hctl -D --help
Manipulate on domains

Usage: hctl {domain|--domain|-D} [OPTIONS]

Options:
  -p, --path <path>          Path to file to be read [path without quotes]
  -o, --out <out>            Path to the out file [stdout/path without quotes]
  -z, --optimize <optimize>  Optimize for memory or speed, default: memory [memory/speed]
  -c, --config <config>      Path to config [without qoutes]
  -m, --mode <mode>          Process single or multiple lists [single/folder/config/url]
  -i, --intro <intro>        Whether append intro "Assembled From:..." to out file from config [true/false]
  -r, --rejected <rejected>  Whether save rejected to ./rejected.txt [true/false]
  -f, --format <format>      Type of out format [dnsmasq/loopback/empty/linewise/bind/snort/unbound/machosts/hostperm1/junkbuster/littlesnitch/pdnsd]
  -t, --validate <validate>  (Non-intrusive): Verbose validate your records in single file issuing dns-over-tls query in hardcoded servers [yes/no]
  -d, --dns <dns>            Validate your processed records issuing dns query in predefined servers, config mode allows custom ones [yes/no]
  -a, --pattern <pattern>    Input format of list [hosts/dnsmasq]
  -h, --help                 Print help

Magic behind the scene

There is heavy usage of rust iterators to enable lazy-reading

Processing is done by firstly removing addreses characteristic for host file like 127.0.0.1 and comments after the records. Next tool removes all whitespace characters and non-urls using regex. See rules.rs if you want to embed similar solution in your app.

License

BSD-3-clause