Features • Installation • Usage • Running httpx • Notes • Join Discord

httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.

Features

Simple and modular code base making it easy to contribute.
Fast And fully configurable flags to probe multiple elements.
Supports multiple HTTP based probings.
Smart auto fallback from https to http as default.
Supports hosts, URLs and CIDR as input.
Handles edge cases doing retries, backoffs etc for handling WAFs.

Supported probes:-

Probes	Default check	Probes	Default check
URL	true	IP	true
Title	true	CNAME	true
Status Code	true	Raw HTTP	false
Content Length	true	HTTP2	false
TLS Certificate	true	HTTP 1.1 Pipeline	false
CSP Header	true	Virtual host	false
Location Header	true	CDN	false
Web Server	true	Path	false
Web Socket	true	Ports	false
Response Time	true	Request method	false

Installation Instructions

httpx requires go1.17 to install successfully. Run the following command to get the repo -

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

Usage

httpx -h

This will display help for the tool. Here are all the switches it supports.

Usage:
  ./httpx [flags]

Flags:
INPUT:
   -l, -list string  Input file containing list of hosts to process
   -request string   File containing raw request

PROBES:
   -sc, -status-code     Display Status Code
   -td, -tech-detect     Display wappalyzer based technology detection
   -cl, -content-length  Display Content-Length
   -server, -web-server  Display Server header
   -ct, -content-type    Display Content-Type header
   -rt, -response-time   Display the response time
   -title                Display page title
   -location             Display Location header
   -method               Display Request method
   -websocket            Display server using websocket
   -ip                   Display Host IP
   -cname                Display Host cname
   -cdn                  Display if CDN in use
   -probe                Display probe status
   -nf, -no-fallback     Display both protocol (HTTPS and HTTP)

MATCHERS:
   -mc, -match-code string     Match response with given status code (-mc 200,302)
   -ml, -match-length string   Match response with given content length (-ml 100,102)
   -ms, -match-string string   Match response with given string
   -mr, -match-regex string    Match response with specific regex
   -er, -extract-regex string  Display response content with matched regex

FILTERS:
   -fc, -filter-code string    Filter response with given status code (-fc 403,401)
   -fl, -filter-length string  Filter response with given content length (-fl 23,33)
   -fs, -filter-string string  Filter response with specific string
   -fe, -filter-regex string   Filter response with specific regex

RATE-LIMIT:
   -t, -threads int      Number of threads (default 50)
   -rl, -rate-limit int  Maximum requests to send per second (default 150)

MISCELLANEOUS:
   -tls-grab            Perform TLS(SSL) data grabbing
   -tls-probe           Send HTTP probes on the extracted TLS domains
   -csp-probe           Send HTTP probes on the extracted CSP domains
   -pipeline            HTTP1.1 Pipeline probe
   -http2               HTTP2 probe
   -vhost               VHOST Probe
   -p, -ports string[]  Port to scan (nmap syntax: eg 1,2-10,11)
   -path string         File or comma separated paths to request
   -paths string        File or comma separated paths to request (deprecated)

OUTPUT:
   -o, -output string                File to write output
   -sr, -store-response              Store HTTP responses
   -srd, -store-response-dir string  Custom directory to store HTTP responses (default "output")
   -json                             Output in JSONL(ines) format
   -irr, -include-response           Include HTTP request/response in JSON output (-json only)
   -include-chain                    Include redirect HTTP Chain in JSON output (-json only)
   -store-chain                      Include HTTP redirect chain in responses (-sr only)
   -csv                              Output in CSV format

CONFIGURATIONS:
   -rsts, -response-size-to-save int  Max response size to save in bytes (default 2147483647)
   -rstr, -response-size-to-read int  Max response size to read in bytes (default 2147483647)
   -allow string[]                    Allowed list of IP/CIDR's to process (file or comma separated)
   -deny string[]                     Denied list of IP/CIDR's to process (file or comma separated)
   -random-agent                      Enable Random User-Agent to use (default true)
   -H, -header string[]               Custom Header to send with request
   -http-proxy, -proxy string         HTTP Proxy, eg http://127.0.0.1:8080
   -unsafe                            Send raw requests skipping golang normalization
   -resume                            Resume scan using resume.cfg
   -nc, -no-color                     Disable color in output
   -nfs, -no-fallback-scheme          Probe with input protocol scheme
   -fr, -follow-redirects             Follow HTTP redirects
   -fhr, -follow-host-redirects       Follow redirects on the same host
   -maxr, -max-redirects int          Max number of redirects to follow per host (default 10)
   -vhost-input                       Get a list of vhosts as input
   -x string                          Request methods to use, use 'all' to probe all HTTP methods
   -body string                       Post body to include in HTTP request
   -s, -stream                        Stream mode - start elaborating input targets without sorting
   -sd, -skip-dedupe                  Disable dedupe input items (only used with stream mode)

DEBUG:
   -silent   Silent mode
   -verbose  Verbose mode
   -version  Display version
   -debug    Debug mode
   -stats    Display scan statistic

OPTIMIZATIONS:
   -retries int                 Number of retries
   -timeout int                 Timeout in seconds (default 5)
   -maxhr, -max-host-error int  Max error count per host before skipping remaining path/s (default 30)
   -ec, -exclude-cdn            Skip full port scans for CDNs (only checks for 80,443)

Running httpX

Running httpx with stdin

This will run the tool against all the hosts and subdomains in hosts.txt and returns URLs running HTTP webserver.

cat hosts.txt | httpx 

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   / 
 / / / / /_/ /_/ /_/ /   |  
/_/ /_/\__/\__/ .___/_/|_|   v1.1.1  
             /_/            

    projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.

https://mta-sts.managed.hackerone.com
https://mta-sts.hackerone.com
https://mta-sts.forwarding.hackerone.com
https://docs.hackerone.com
https://www.hackerone.com
https://resources.hackerone.com
https://api.hackerone.com
https://support.hackerone.com

Running httpx with file input

This will run the tool with the probe flag against all of the hosts in hosts.txt and return URLs with probed status.

httpx -list hosts.txt -silent -probe

http://ns.hackerone.com [FAILED]
https://docs.hackerone.com [SUCCESS]
https://mta-sts.hackerone.com [SUCCESS]
https://mta-sts.managed.hackerone.com [SUCCESS]
http://email.hackerone.com [FAILED]
https://mta-sts.forwarding.hackerone.com [SUCCESS]
http://links.hackerone.com [FAILED]
https://api.hackerone.com [SUCCESS]
https://www.hackerone.com [SUCCESS]
http://events.hackerone.com [FAILED]
https://support.hackerone.com [SUCCESS]
https://gslink.hackerone.com [SUCCESS]
http://o1.email.hackerone.com [FAILED]
http://info.hackerone.com [FAILED]
https://resources.hackerone.com [SUCCESS]
http://o2.email.hackerone.com [FAILED]
http://o3.email.hackerone.com [FAILED]
http://go.hackerone.com [FAILED]
http://a.ns.hackerone.com [FAILED]
http://b.ns.hackerone.com [FAILED]

Running httpx with CIDR input

echo 173.0.84.0/24 | httpx -silent

https://173.0.84.29
https://173.0.84.43
https://173.0.84.31
https://173.0.84.44
https://173.0.84.12
https://173.0.84.4
https://173.0.84.36
https://173.0.84.45
https://173.0.84.14
https://173.0.84.25
https://173.0.84.46
https://173.0.84.24
https://173.0.84.32
https://173.0.84.9
https://173.0.84.13
https://173.0.84.6
https://173.0.84.16
https://173.0.84.34

Running httpx with subfinder

subfinder -d hackerone.com -silent| httpx -title -tech-detect -status-code

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.1

    projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
https://mta-sts.managed.hackerone.com [404] [Page not found · GitHub Pages] [Varnish,GitHub Pages,Ruby on Rails]
https://mta-sts.hackerone.com [404] [Page not found · GitHub Pages] [Varnish,GitHub Pages,Ruby on Rails]
https://mta-sts.forwarding.hackerone.com [404] [Page not found · GitHub Pages] [GitHub Pages,Ruby on Rails,Varnish]
https://docs.hackerone.com [200] [HackerOne Platform Documentation] [Ruby on Rails,jsDelivr,Gatsby,React,webpack,Varnish,GitHub Pages]
https://support.hackerone.com [301,302,301,200] [HackerOne] [Cloudflare,Ruby on Rails,Ruby]
https://resources.hackerone.com [301,301,404] [Sorry, no Folders found.]

Running httpx with docker

cat sub_domains.txt | docker run -i projectdiscovery/httpx

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.2

      projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
https://mta-sts.forwarding.hackerone.com
https://mta-sts.hackerone.com
https://mta-sts.managed.hackerone.com
https://www.hackerone.com
https://api.hackerone.com
https://gslink.hackerone.com
https://resources.hackerone.com
https://docs.hackerone.com
https://support.hackerone.com

📋 Notes

As default, httpx checks for HTTPS probe and fall-back to HTTP only if HTTPS is not reachable.
For printing both HTTP/HTTPS results, no-fallback flag can be used.
Custom scheme for ports can be defined, for example -ports http:443,http:80,https:8443
vhost, http2, pipeline, ports, csp-probe, tls-probe and path are unique flag with different probes.
Unique flags should be used for specific use cases instead of running them as default with other flags.
When using json flag, all the information (default probes) included in the JSON output.

Acknowledgement

httpx is made with 🖤 by the projectdiscovery team. Community contributions have made the project what it is. See the Thanks.md file for more details. Do also check out these similar awesome projects that may fit in your workflow:

Probing feature is inspired by @tomnomnom/httprobe work ❤️

Cgboal/httpx