jeancochrane/mellow-bike-map

Data import bug

Closed this issue · 8 comments

beamalsky commented

I cloned this repo, ran docker-compose build, and then hit this error on the next step in the README:

Beas-MBP:mellow-bike-map beamalsky$ docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make
Creating network "mellow-bike-map_default" with the default driver
Creating volume "mellow-bike-map_mellow-bike-map-db-data" with default driver
Creating volume "mellow-bike-map_mellow-bike-map-node-modules" with default driver
Creating mellow-bike-map-postgres ... done
Building make
Step 1/4 : FROM mdillon/postgis:11
 ---> b2a8fefe8a63
Step 2/4 : RUN apt-get update && apt-get install -y --no-install-recommends     build-essential gcc make cmake gdal-bin postgresql-server-dev-11     expat libexpat1-dev libboost-dev libboost-graph-dev libboost-program-options-dev libpqxx-dev     wget ca-certificates
 ---> Using cache
 ---> ce2f9c425429
Step 3/4 : RUN wget -O pgrouting-3.1.0.tar.gz https://github.com/pgRouting/pgrouting/archive/v3.1.0.tar.gz &&     tar xvfz pgrouting-3.1.0.tar.gz &&     cd pgrouting-3.1.0 &&     mkdir build &&     cd build &&     cmake .. &&     make &&     make install &&     cd / && rm -Rf pgrouting-3.1.0*
 ---> Using cache
 ---> 8700ee10c6bc
Step 4/4 : RUN wget -O osm2pgrouting-2.3.6.tar.gz https://github.com/pgRouting/osm2pgrouting/archive/v2.3.6.tar.gz &&     tar xvfz osm2pgrouting-2.3.6.tar.gz &&     cd osm2pgrouting-2.3.6 &&     cmake -H. -Bbuild &&     cd build &&     make &&     make install &&     cd / && rm -Rf osm2pgrouting-2.3.6*
 ---> Using cache
 ---> e708efbda884

Successfully built e708efbda884
Successfully tagged mellow-bike-map-make:latest
WARNING: Image for service make was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
python manage.py loaddata app/mbm/fixtures/mellowroute.json && touch db/import/mellowroute.table
python: can't open file 'manage.py': [Errno 2] No such file or directory
Makefile:5: recipe for target 'db/import/mellowroute.table' failed
make: *** [db/import/mellowroute.table] Error 2
Beas-MBP:mellow-bike-map beamalsky$ docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make
Starting mellow-bike-map-postgres ... done
python manage.py loaddata app/mbm/fixtures/mellowroute.json && touch db/import/mellowroute.table
python: can't open file 'manage.py': [Errno 2] No such file or directory
Makefile:5: recipe for target 'db/import/mellowroute.table' failed
make: *** [db/import/mellowroute.table] Error 2

Do you know what's up here @jeancochrane? Looks like it's having trouble with the Django management command.

jeancochrane commented

There's a couple problems with the db/import/mellowroute.table target:

  1. It requires the dependencies in app/Dockerfile (notably Django), but the Makefile uses db/Dockerfile
  2. We need to run management commands from the app/ directory, not from the root directory

I think fixing both these problems will require refactoring the data import pipeline to use one shared Dockerfile that has both dependencies and adjusting the target to run management commands from the right directory. I can do this but probably not till later tonight.

In the meantime, one work around would be to adjust the command that's listed in the README to skip the db/import/mellowroute.table target:

docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table

Then import the MellowRoute fixture manually using the app container, instead of using the broken target:

docker-compose run --rm app ./manage.py loaddata mellowroute
beamalsky commented

I'll do this workaround, thank you! If you didn't want to do the refactor, FWIW I think it'd be fine to just have an extra step in the README.

beamalsky commented

Unfortunately I'm now getting this error:

Beas-MBP:mellow-bike-map beamalsky$ docker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table
Starting mellow-bike-map-postgres ... done
osm2pgrouting -f db/raw/chicago.osm -c /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml --prefix chicago_ --addnodes --tags --clean \
              -d mbm -U postgres -h postgres -W postgres &&
/bin/sh: 2: Syntax error: end of file unexpected
Makefile:8: recipe for target 'db/import/chicago.table' failed
make: *** [db/import/chicago.table] Error 2
jeancochrane commented

@beamalsky Good catch! I just pushed up 1d4539c which should fix that syntax error.

  • 👍1
beamalsky commented

I'm now seeing this:

Beas-MBP:mellow-bike-map beamalskydocker-compose -f docker-compose.yml -f db/docker-compose.yml run --rm make db/import/chicago.table
Starting mellow-bike-map-postgres ... done
wget --no-use-server-timestamps -O db/raw/chicago.osm https://overpass-api.de/api/map?bbox=-87.7488,41.7170,-87.5157,42.0003
--2020-09-07 02:51:43--  https://overpass-api.de/api/map?bbox=-87.7488,41.7170,-87.5157,42.0003
Resolving overpass-api.de (overpass-api.de)... 178.63.11.215, 178.63.48.217, 2a01:4f8:120:6464::2, ...
Connecting to overpass-api.de (overpass-api.de)|178.63.11.215|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/osm3s+xml]
Saving to: ‘db/raw/chicago.osm’

db/raw/chicago.osm      [             <=>    ] 927.82M  5.03MB/s    in 3m 24s

2020-09-07 02:55:14 (4.56 MB/s) - ‘db/raw/chicago.osm’ saved [972891961]

osm2pgrouting -f db/raw/chicago.osm -c /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml --prefix chicago_ --addnodes --tags --clean \
              -d mbm -U postgres -h postgres -W postgres && \
PGPASSWORD=postgres psql -U postgres -h postgres -d mbm -c " \
	UPDATE chicago_ways SET one_way = 2, oneway = 'NO', reverse_cost = cost \
	FROM osm_ways \
	WHERE osm_ways.osm_id = chicago_ways.osm_id \
	AND osm_ways.tags @> 'oneway:bicycle => no'" && \
touch db/import/chicago.table
Execution starts at: Mon Sep  7 02:55:14 2020

***************************************************
           COMMAND LINE CONFIGURATION             *
***************************************************
Filename = db/raw/chicago.osm
Configuration file = /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml
host = postgres
port = 5432
dbname = mbm
username = postgres
schema=
prefix = chicago_
suffix =
Drop tables
Don't create indexes
Add OSM nodes
***************************************************
Testing database connection: mbm
database connection successful: mbm
Connecting to the database
connection success

Dropping tables...
TABLE: chicago_ways dropped ... OK.
TABLE: chicago_ways_vertices_pgr dropped ... OK.
TABLE: chicago_pointsofinterest dropped ... OK.
TABLE: configuration dropped ... OK.
TABLE: osm_nodes dropped ... OK.
TABLE: osm_ways dropped ... OK.
TABLE: osm_relations dropped ... OK.

Creating tables...
TABLE: chicago_ways_vertices_pgr created ... OK.
TABLE: chicago_ways created ... OK.
TABLE: chicago_pointsofinterest created ... OK.
TABLE: configuration created ... OK.
TABLE: osm_nodes created ... OK.
TABLE: osm_ways created ... OK.
TABLE: osm_relations created ... OK.
Opening configuration file: /usr/local/share/osm2pgrouting/mapconfig_for_bicycles.xml
    Parsing configuration

Exporting configuration ...
  - Done
Counting lines ...
  - Done
Opening data file: db/raw/chicago.osm	total lines: 13185664
    Parsing data

Current osm_nodes:Killed760000
Makefile:8: recipe for target 'db/import/chicago.table' failed
make: *** [db/import/chicago.table] Error 137
beamalsky commented

Oh looks like this is a memory error—let me try upping my docker system resources!

jeancochrane commented
beamalsky commented

I upped my memory limit from 2 to 6 GiB, so far so good