metegenez/scalable-port-scanner

Scalable Port Scanner

In this application, a port scanner was developed that can scan computers in a network in a multithreaded and distributed manner. Basically, elements that could create bottlenecks on the application were made multithreaded and mapped efficiently.

System Design

The requirements of the designed system are that Workers can process different JOB requests on different machines at the same time and index the outputs of these requests in a DB. This means scalable services. Although this scaling is manual, it requires a message queue and services that periodically pool that queue.

For this reason, Redis was preferred as a message queue and added to the system. In this way, it is ensured that transactions are queued and consumed by different service replicas.

Setup

Test

To test application locally, we need virtual machines and Docker Desktop (Windows only).

• Create a Linux/Windows machine with bridge adapter to make VM get IP address from host network.

• If Host (where the Web Application is running) is Windows, disable the Security Firewall. Probably there is a least-priviledged approach here, but for simplicity disable it to ping Host from VM.

Setup

• Clone the repository. / Download the code.

• Prepare web application.

  • Start the web application.

  docker-compose up

  • PUT the mapping in Elasticsearch DB. (There can be done with config files anyway.) Index name is default "scans".

  curl --location --request PUT 'localhost:9200/scans' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "mappings": {
      "dynamic": false,
      "properties": {
        "start_timestamp": {
          "type": "date"
        },
        "finished_timestamp": {
          "type": "date"
        },
        "job_id": {
          "type": "keyword"
        },
        "scanned_ip": {
          "type": "keyword"
        },
        "status": {
          "type": "keyword"
        },
        "scanned_ports": {
          "type": "keyword"
        },
         "ip_info": {
          "type": "flattened"
        },
        "ip_port_info": {
          "type": "flattened"
        }
      }
    }
  }'

  • Then you can visit localhost:3000.

• Prepare Workers

  • Change the config.ini file in "./workers" subdirectory. Put your host IP address accordingly.
  • Install requirements. (You can use venv also.)

  cd workers/
  pip install -r requirements.txt

  • Run it.

  python main.py

All done.

Troubleshot:

• If any Redis intance is running in your host computer, worker and dockerized backend may end up connection to different Redis intances.

In this application, a port scanner was developed that can scan computers in a network in a multithreaded and distributed manner. Basically, elements that could create bottlenecks on the application were made multithreaded and mapped efficiently.

Workers

Workers are designed to be work in a distributed manner. There 3 main steps to scan given IP range and port range. We enabled that each job is executed in different machine, or next to a job finishes. This means, we do not divide the jobs between nodes. This can cause an suboptimal load balancing if jobs are different in size and job count is low.

Ping IP Range

It calls system "ping" command, returns OS information if ping succeeds or False otherwise. The main functionality is Python 3.5+, Windows and Linux compatible - OS detection is implemented only in Windows currently-. Making this "ping" function threaded is handled in same class. Most of the methods are static, still we used a non-static function to cover all.

OS Detection

You can quickly detect whether a system is running with Linux, or Windows or any other OS by looking at the TTL value from the output of the ping command. You don't need any extra applications to detect a remote system's OS. This is implemented only in Windows.

Connect Port Range

This part consumes the online IP knowdledge of previous block. This part takes online IP addresses and port ranges, takes product of two list as tuples as

[
('192.168.1.1', 80), 
('192.168.1.1', 8080), 
('192.168.1.2', 80), 
('192.168.1.2', 8080), 
...]

and so on. Then, it tries to connect them multihreaded.

Service Detection

Service detection is based on

socket.getservbyport()

If it cannot detect, we put most known default port services into account and read it from a file.