Geolocation

This service provides an API for searching the geolocation of a given IP address.

How the app works

The application has two features, an IP location importer and a HTTP API for searching IP locations.

The IP location data importer

In order to provide the locations through an API the service expects the data to be imported somehow. There are two way to do so, the first one is by calling the importer service directly via rails console. It's not recommended as there is a rake task that provides a simpler interface.

# docker compose run -it app bin/rails c

Locations::Importer
  .new(validator: Locations::Validator, persistence: IpLocation)
  .import_from_csv_file('public/data_dump.csv')

The following command can be used to execute the rake task that import the data from the csv file. By default it will import the data from public/data_dump.csv.

docker compose run app bin/rails data:import_from_csv_file

The task will print out some informations regarding the execution time and statistics about the rake task execution.

Importing data
validating...
persisting...
done.
===================================================
Time Elapsed: 107.83850705300574 seconds
        Validation: 26.08323292498244 seconds
        Persistence: 81.75525469798595 seconds
Records processed: 949844
        Accepted: 866449
        Discarted: 83395
===================================================
[Rake task] completed in 108.16110226599267 seconds

Both of them accepts a CSV file path as input and expects its content to be in the following format in order to perform the data load operation properly:

ip_address,country_code,country,city,latitude,longitude,mystery_value
200.106.141.15,SI,Nepal,DuBuquemouth,-84.87503094689836,7.206435933364332,7823011346
160.103.7.140,CZ,Nicaragua,New Neva,-68.31023296602508,-37.62435199624531,7301823115
70.95.73.73,TL,Saudi Arabia,Gradymouth,-49.16675918861615,-86.05920084416894,2559997162
,PY,Falkland Islands (Malvinas),,75.41685191518815,-144.6943217219469,0
125.159.20.54,LI,Guyana,Port Karson,-78.2274228596799,-163.26218895343357,1337885276

The HTTP API for searching IP location.

The requests can be done locally by running the following:

curl 'http://localhost:3000/api/v1/ip_locations?ip_address=125.159.20.54' | python -m json.tool

HTTP/1.1 200 OK

In the case of the location be found, the expected response follows this format:

{
    "ip_location": {
        "ip_address": "125.159.20.54",
        "city": "Port Karson",
        "country": "Guyana",
        "country_code": "LI",
        "latitude": -78.2274228596799,
        "longitude": -163.26218895343357,
        "mystery_value": 1337885276
    }
}

However, often things goes wrong, in those cases the service tries to catch two failure scenarios.

HTTP/1.1 404 Not Found

If the IP location hasn't been imported to the service yet

curl 'http://localhost:3000/api/v1/ip_locations?ip_address=125.159.20.5' | python -m json.tool

The following response is expected:

{
    "error": {
        "message": "No location found for `ip_address`: 125.159.20.5."
    }
}

HTTP/1.1 422 Unprocessable Entity

In the case the ip_address parameter contains an invalid value

curl 'http://localhost:3000/api/v1/ip_locations?ip_address=125.159205' | python -m json.tool

The following response is expected instead:

{
    "error": {
        "message": "`ip_address`: 125.159205 is an invalid IP Address."
    }
}

The absence of the ip_address is considered an invalid value

curl 'http://localhost:3000/api/v1/ip_locations' | python -m json.tool

And wil result in:

{
    "error": {
        "message": "`ip_address`: nil is an invalid IP Address."
    }
}

Tradeoffs

Importing with CSV.foreach

Ruby's Standard library offers a gem called CSV which provides a method called foreach capable of open a stream from the file in order to process each line individually instead of load the whole file to memory before hand.

Validation separately from ActiverRecord

As the CSV file's source isn't reliable, it's necessary to validate the data before persisting it. ActiveRecord offers a rich API for validation, but its validation rules are applied to model instances, which requires the importer to instantiate each record in order to validate it, making the import operation prohibitively slow. Hence, it implements a simple validator object that relies on CSV lines format instead of model instances, eliminating the need for object instantiation.

Persistence using insert_all

The number of ip locations in the provided file is huge, so that each location being persisted one by one would cause the application to do a lot of round trips to the database, which would take too long for persisting all the records. The decision here was to persist them all together. The implementation relies on ActiverRecord API that provides a very handy method called insert_all that builds a single SQL statement in order to insert all the given records. It was tried to persit the data in batches, executing a few amount of insert_all calls, but there was no improvement in the performance. Maybe if the collection were bigger, the batch approach would be viable, but for now the solution adopted was simpler to implement. Another alternative would be creating a a thread for each batch and execute the insert_all call from them. It also was tried, but the the problem was that the threads executed database calls concurrently, and the constraints of the database table seems to have become the bottleneck, as it had to assure uniqueness of primary key, or something else that wasn't fully understood.

How to setup the app

System dependencies docker

It's enough to have the docker and docker compose installed on local machine to run this application.

Execute the following command in order to run the Rails server:

docker compose up

Database creation

The setup of the database can done by running:

docker compose exec -it app bin/rails db:create db:migrate

How to run the test suite

Simple like this:

docker compose run app 'bundle exec rspec'

devjoaoGustavo/geolocation