shipt/specter

Specter: An Open Source Geographic Visualization for NGINX Access Logs

Specter

Specter is an attack map style visualization that parses NGINX access logs. Specter then displays the source ip's location, the NGINX's ip's location, and the http status on a world map. Additionally allows filtering the traffic based on services.

To read more about, how to run, and how to develop Specter, check out our wiki.

Table of Contents

• Getting Started
  • Prerequisites
    • Third Party
    • Golang
    • Frontend
  • Installing
    • Running Specter in Your Development Environment
  • Running the Tests
    • What We Test
    • Coding Style
• Deployment
• Attributions
• Contributing
• Versioning
• Authors
• License

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

• Third Party

  • This project uses a MaxMind database to get geo data from IP addresses. You must download the GeoLite2 City database and save it somewhere Specter can reach it.

• Golang

  • Go version 1.11+
  • For go dependencies this project uses dep, so dep ensure will install all needed dependencies.

• Frontend (web/public)

  • npm must be installed

# Move into web/public
cd web/public
# Just once 
npm install
# Watch for changes during development
npm run watch
# When ready to build/deploy
npm run build

Installing

1. Clone this repo to your GOPATH.
2. From your local repository, run dep ensure.
3. From your local repository, cd web/public and run npm install.

Running Specter in Your Development Environment

1. From your local repository, start the Specter webserver:

   go run ./cmd/specter/main.go -db={{Where your GeoLite2-City database is}} -mbat={{Your MapBox Access Token}}

2. In a new console from your local repository, start the start the Specter data server:

   go run cmd/specter-data/main.go -log ./scripts/access.log

3. (Optional) From your local repository, start the start the load test:

   scripts/load.py

Running the Tests

Your code editor should be setup to run tests on save. However, to run the tests manually, you can run go test ./... from the local repository directory to run all tests. To run just one test, run go test ./dir/package.

• What We Test

  • The unit tests in this repo test our code, not code brought in through packages.

    • For example in the dataServer package, we do not test the tailFile function since it only implements imported code.
    • However, in the same package, we do test the processLog function since it contains code that is untested elsewhere.

• Coding Style

  • Just use the default revive configuration.

Building and Running Your Own Images

1. Ensure you have docker installed and working as expected.
2. Build Specter.Dockerfile:

   docker build -f Specter.Dockerfile .
   

3. Build the Specter-Data.Dockerfile:

   docker build -f Specter-Data.Dockerfile .
   

4. Start the Specter docker image:

   docker run -e DB=./db/GeoLite2-City.mmdb -e MBAT=<<YOUR_MAPBOX_API_TOKEN>> -v <<FOLDER_WHERE_YOUR_GEOLITE2_MMDB_EXISTS>>:/go/src/github.com/shipt/specter/db -p 1323:1323 <<YOUR_IMAGE_FROM_STEP_2>>
   

5. Start the Specter-Data docker image:

   docker run <<YOUR_IMAGE_FROM_STEP_3>>
   

Note: You will probably want to set some other ENV vars in order to get Specter working for you. The ENV vars can be found here.

Deployment

1. Download the appropriate version from the Releases page.
2. Deploy the Specter package to where you plan on running the Specter Webserver.
   • Start Specter with the -db={{Where your GeoLite2-City database is}} -mbat={{Your MapBox Access Token}} flags.
3. Deploy the Specter-Data package to all NGINX servers you wish to monitor.
   • Start Specter-Data with the -log ./scripts/access.log flag and any other applicable flags.

Attributions

Read the Attributions here.

Contributing

Please read our CONTRIBUTING.md for details on our community guidelines and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

To update versions, run the provided python script and follow the prompts.

Maintainers

To find out who our Maintainers are, check out: MAINTAINERS.md

License

This project is licensed under the MIT License - see the LICENSE.md file for details.