mediagis/nominatim-docker

Crash while importing europe data

te-deum opened this issue · 4 comments

te-deum commented

Describe the bug
I run a France import succefully and now I re-run from scratch to import Europe.
2 times with same problem on post-process :

psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally before or while processing the request.

I would like to import Europe. Is it possible to re-run without re-creating database and all datas (5 to 6 days) ?

Screenshots & Logs

nominatim  | 2024-02-01 03:50:08: Post-process tables
nominatim  | Traceback (most recent call last):
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/sql_preprocessor.py", line 119, in run_parallel_sql_file
nominatim  |........................................................................................................................................................................................................................................
nominatim  |
nominatim  |     pool.next_free_worker().perform(part)
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 201, in next_free_worker
nominatim  |     return next(self.free_workers)
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 209, in _yield_free_worker
nominatim  |     if thread.is_done():
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 159, in is_done
nominatim  |     if self.conn.poll() == psycopg2.extensions.POLL_OK:
nominatim  | psycopg2.OperationalError: server closed the connection unexpectedly
nominatim  |    This probably means the server terminated abnormally
nominatim  |    before or while processing the request.
nominatim  |
nominatim  |
nominatim  | During handling of the above exception, another exception occurred:
nominatim  |
nominatim  | Traceback (most recent call last):
nominatim  |   File "/usr/local/bin/nominatim", line 12, in <module>
nominatim  |     exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 225, in nominatim
nominatim  |     return get_set_parser().run(**kwargs)
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
nominatim  |     return args.command.run(args)
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 138, in run
nominatim  |     database_import.create_search_indices(conn, args.config,
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 252, in create_search_indices
nominatim  |     sql.run_parallel_sql_file(config.get_libpq_dsn(),
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/sql_preprocessor.py", line 117, in run_parallel_sql_file
nominatim  |     with WorkerPool(dsn, num_threads) as pool:
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 235, in __exit__
nominatim  |     self.finish_all()
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 185, in finish_all
nominatim  |     thread.wait()
nominatim  |   File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 128, in wait
nominatim  |     wait_select(self.conn)
nominatim  |   File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 762, in wait_select
nominatim  |     state = conn.poll()
nominatim  | psycopg2.OperationalError: server closed the connection unexpectedly
nominatim  |    This probably means the server terminated abnormally
nominatim  |    before or while processing the request.

Desktop / Server (please complete the following information):

  • OS & Version: Debian 12
  • HDD : 1,2 TB
  • RAM : 32 GB
  • Docker Version: Docker version 24.0.7, build afdd53b
  • Nominatim Version: 4.3.2

Additional context
My docker-compose.yml :

version: "3"
services:
    nominatim:
        container_name: nominatim
        image: mediagis/nominatim:4.3
        ports:
            - "8080:8080"
        environment:
            PBF_URL: https://download.geofabrik.de/europe-latest.osm.pbf
            REPLICATION_URL: http://download.geofabrik.de/europe-updates/
            REPLICATION_UPDATE_INTERVAL: 60
            NOMINATIM_PASSWORD: "MyUniquePassword"
            IMPORT_WIKIPEDIA: "true"
            IMPORT_GB_POSTCODES: "true"
            IMPORT_TIGER_ADDRESSES: "true"
            THREADS: 8
            shm-size: 4G
        volumes:
            - type: bind
              source: /data/db
              target: /var/lib/postgresql/14/main
            - type: bind
              source: /data/flatnode
              target: /nominatim/flatnode
        shm_size: 1gb
        logging:
            options:
                max-size: "20m"
                max-file: "3"
mtmail commented

You should check the postgresql server logfiles for hints. Maybe the server crashed due to hardware constraints. Then rerunning with the same hardware and settings will only make it crash again. Europe data is much bigger than France obviously.

The last line ran from init.sh was sudo -E -u nominatim nominatim import --osm-file $OSMFILE --threads $THREADS

nominatim import can restart the post processing step, see nominatim import -h. I think you need sudo -E -u nominatim nominatim import --continue db-postprocess and then everything from init.sh manually.

        IMPORT_TIGER_ADDRESSES: "true"

You probably don't need this for Europe. TIGER data is US streets and house numbers.

leonardehrenfried commented

I also believe it has to do with not enough ram or disk. I've added a tag and you can check other issues with the same one: https://github.com/mediagis/nominatim-docker/issues?q=label%3A%22resource+problem%22+

They might contain a clue about your problem.

te-deum commented

Thank you for your replies 👍

You should check the postgresql server logfiles for hints. Maybe the server crashed due to hardware constraints. Then rerunning with the same hardware and settings will only make it crash again. Europe data is much bigger than France obviously.

I change my RAM from 32GB to 128GB to finish the process.

The last line ran from init.sh was sudo -E -u nominatim nominatim import --osm-file $OSMFILE --threads $THREADS

nominatim import can restart the post processing step, see nominatim import -h. I think you need sudo -E -u nominatim nominatim import --continue db-postprocess and then everything from init.sh manually.

Great, but I don't know how to start the container without re-launching import process. Any tips is welcome :)

        IMPORT_TIGER_ADDRESSES: "true"

You probably don't need this for Europe. TIGER data is US streets and house numbers.

I will change my docker-compose.yml according to this information.

leonardehrenfried commented

So it looks like you solved your problem, so I will close this. Please re-open if I was wrong.