/modular-bna

Primary LanguageShell

modular-bna

The objective of this project is to modernize the current version of the BNA by making it more modular, easier to update and easier to maintain in general.

Requirements

The modular BNA has a new client/server design which requires the process sending the queries to have all the tools installed locally.

Here are the tools required by the project:

  • docker
  • docker compose
  • gdal-bin
  • osm2pgrouting
  • osm2pgsql
  • osmctools
  • osmium-tool
  • postgis
  • postgresql-client

Validation

To validate the results, we compare the scores generated by the original BNA with the scores generated by this version. If the difference is less or equal to +/-10 points (i.e. +/-0.1%), the results are considered valid.

Example

This is the validation file for the city of Provincetown, MA, USA:

score_id modular original delta
people 19.19 19.17 2
opportunity_employment 8.27 8.26 1
opportunity_k12_education 8.33 8.31 2
opportunity_technical_vocational_college 0.0 0.0 0
opportunity_higher_education 0.0 0.0 0
opportunity 8.3 8.29 2
core_services_doctors 0.0 0.0 0
core_services_dentists 0.0 0.0 0
core_services_hospitals 5.18 5.18 0
core_services_pharmacies 0.0 0.0 0
core_services_grocery 1.69 1.69 0
core_services_social_services 0.0 0.0 0
core_services 3.24 3.24 0
retail 0.0 0.0 0
recreation_parks 7.13 7.13 0
recreation_trails 0.0 0.0 0
recreation_community_centers 0.0 0.0 0
recreation 7.13 7.13 0
transit 0.0 0.0 0
overall_score 8.94 8.93 1

All the values in the delta column are less than 10, therefore the results are considered correct.

Tests

The tests are written in Python 3 using PyTest and a set of plugins.

Test flow

For each city defined in the test file, the test workflow will be as follow:

  • Use the brokenspoke-analyzer to prepare the input files required to analyze the city.
  • Use the brokenspoke-analyzer to run the analysis, using the input files previous prepared.
  • Run the modular BNA, using the input files previous prepared.
  • Compare the results, and ensure a difference less than the predefined delta for ALL fields.
  • Generate an HTML report of the test execution (duration, retries, output, etc.)

The test machine

The machine used to run the tests is an Intel NUC NUC6i7KYK with 8 vCores, 32GB RAM and 512 GB SSD.

Test markers

To keep the tests organized and be able to run them in meaningful groups, a set of test markers was created. For those not familiar with PyTest, you can refer to the official page for the full explanation, other simply consider them as labels.

Duration marker: T-shirt sizes

The test runs were measured on our test machine, and categorized per their durations as follow:

  • XS: runs under 5 min
  • S: runs under 15 min
  • M: runs under 60 min
  • L: runs under 180 min (1/8 day)
  • XL: runs under 360 min (1/4 day)
  • XXL: runs under 720 min (1/2 day)

Since a full test is composed of 2 BNA runs, a test run takes about twice as long as a running a BNA analysis.

Keep in mind that these durations are simply here to give an idea of the processing time required to validate a city. They may drastically change depending on the machine being used.

Geolocation markers

Tests are also organized by countries, for instance usa or spain.

Example

Here is an example for the city of Provincetown, MA in the US:

@pytest.mark.usa
@pytest.mark.xs
@pytest.mark.asyncio
async def test_provincetown_ma():
    """Compare the results for the city of Provincetown, MA."""
    await compare("usa", "massachusetts", "provincetown", "555535")

We can see the city as being located in the usa and marked as xs, meaning the tests completed in less than 5 minutes on our test machine.

Creating a test

Testing a city means only 2 things:

  • adding the appropriate markers
  • and calling the compare function with the right arguments

Running the modular BNA manually

Source the .env file:

export $(cat .env | xargs)

Start and stop a clean docker-compose:

docker-compose up
docker-compose rm -sfv
docker volume rm modular-bna_postgres

Use the brokenspoke-analyzer to fetch the data to import into the BNA:

bna -v prepare massachusetts provincetown --output-dir tests/samples/provincetown-massachusetts/data

Source the analysis variables:

export $(cat tests/scripts/provincetown-massachusetts.sh | xargs)

Run the analysis script (optionally in debug mode):

bash -x tests/scripts/run-analysis.sh

To clean up the variables before running a new analysis (optional, just to be safe):

unset BNA_CITY BNA_CITY BNA_FULL_STATE BNA_CITY_FIPS BNA_COUNTRY BNA_SHORT_STATE BNA_STATE_FIPS

Run the analysis with the original bna:

bna analyze massachusetts \
  tests/samples/provincetown-massachusetts/data/provincetown-massachusetts.shp \
  tests/samples/provincetown-massachusetts/data/provincetown-massachusetts.osm \
  --output-dir tests/samples/provincetown-massachusetts/outputs