No space left on device durig initial import
mayeulk opened this issue · 7 comments
Trying to import africa continent from geofabrik:
SQL Error: pq: could not extend file "base/18186/18416.2": No space left on device in query COPY "import"."osm_buildings"
imposm stops with an error. BUt apparently the updater continues to work with diffs.
Full logs:
Detect SRID: 4326,
Clip: no,
Qgis style: yes,
OSM PBF file: /home/settings/country.pbf,
Mapping: /home/settings/country.pbf,
No custom SQL files *.sql detected in /home/settings,
QGIS Style file: /home/settings/qgis_style.sql,
No *.shp detected in /home/settings, so no clipping.,
The checkup is OK.,
The database is empty. Let's import the PBF : /home/settings/country.pbf,
imposm import -diff -deployproduction -overwritecache -cachedir /home/cache -srid 4326 -dbschema-production public -dbschema-import import -dbschema-backup backup -diffdir /home/settings -mapping /home/settings/mapping.yml -read /home/settings/country.pbf -write -connection postgis://docker:docker@db/gis,
[Aug 23 10:28:09] Imposm
[Aug 23 10:28:09] Reading OSM data
[Aug 23 10:28:09] [INFO] [reader] reading /home/settings/country.pbf with data till 2018-08-22 20:15:02 +0000 UTC,
[Aug 23 10:28:09] Reading OSM data
[Aug 23 10:28:10] [ 0s] C: 0/s 0/s (896000) N: 0/s 0/s (4220) W: 0/s 0/s (0) R: 0/s 0/s (0)
[Aug 23 10:28:10] [ 1s] C: 2903000/s 944000/s (1368000) N: 10800/s 1800/s (5142) W: 0/s 0/s (0) R: 0/s 0/s (0)
[...]
100.0%)
[Aug 23 10:37:46] [ 6m10s] C: 0/s 0/s ( 0.0%) N: 0/s 0/s ( 0.0%) W: 70100/s 98000/s (42.6%) R: 2950/s 0/s (100.0%)
[Aug 23 10:37:47] [ 6m10s] C: 0/s 0/s ( 0.0%) N: 0/s 0/s ( 0.0%) W: 70100/s 53900/s (42.6%) R: 2950/s 0/s (100.0%)
[Aug 23 10:37:47] [ 6m11s] C: 0/s 0/s ( 0.0%) N: 0/s 0/s ( 0.0%) W: 70000/s 0/s (42.6%) R: 2950/s 0/s (100.0%)
[Aug 23 10:37:47] [PostGIS] SQL Error: pq: could not extend file "base/18186/18416.2": No space left on device in query COPY "import"."osm_buildings" ("osm_id", "geometry", "name", "type") FROM STDIN ([445246995 0103000020E61000000100000011000000548B96C7C13B4340A0EC74DDE1F21B40548BEACDC13B4340A0EC34DDE5F21B40548BAE09C23B4340A0ECD466E9F21B40548B0E5EC23B4340A0EC148EEBF21B40548B86C7C23B4340A0EC7496ECF21B40548B8234C33B4340A0ECD463ECF21B40548B5A98C33B4340A0EC34F6EAF21B40548B4AE4C33B4340A0ECD485E8F21B40548B2E0FC43B4340A0EC7461E5F21B40548B9610C43B4340A0EC94F9E1F21B40548B06ECC33B4340A0EC14B9DEF21B40548BE6A2C33B434080ECF426DCF21B40548BC23FC33B434080EC349DDAF21B40548B12D2C23B434080EC943DDAF21B40548BBA57C23B434080ECF472DBF21B40548BCEF8C13B434080ECD426DEF21B40548B96C7C13B4340A0EC74DDE1F21B40 yes])[Aug 23 10:37:47] [ 6m11s] C: 0/s 0/s ( 0.0%) N: 0/s 0/s ( 0.0%) W: 70000/s 0/s (42.6%) R: 2950/s 0/s (100.0%)
50An error occured in imposm with the original file.
I tried to change the default value for volume size, but I failed. I tried to modify the compose file but could not find the right syntax (for a "size: " parameter). I also tried various ways to change the dm.basesize and dm.loopdatasize in /etc/default/docker (on debian stretch).
Some of my tests included:
DOCKER_OPTS="--storage-opt dm.loopdatasize=500G dm.basesize=500G"
DOCKER_STORAGE_OPTIONS="dm.loopdatasize=500G dm.basesize=500G"
DOCKER_STORAGE_OPTIONS="--storage-opt dm.basesize=500G"
Some links:
https://stackoverflow.com/questions/30994760/how-to-increase-docker-container-default-size
#16343 (comment)
You mean the disk space is limited in the docker volume? But not your physical hard drive?
Yes. Apparently one volume takes almost all the size:
In osmupdate container console:
df -h
Filesystem Size Used Avail Use% Mounted on
overlay 20G 15G 4.0G 79% /
tmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/root 20G 15G 4.0G 79% /home/cache
/dev/md4 1.8T 2.5G 1.7T 1% /home/settings
shm 64M 0 64M 0% /dev/shm
tmpfs 16G 0 16G 0% /sys/firmware
The other volumes seem to be 20G or 16G by default, which is too small
/dev/md4 is 1.7T while the physical capacity is 2TB.
For country datasets it is usually not a problem. For a continent, it gets filled during import.
I never tried to import a continent so I never met this issue.
Dec 21 15:31:30] SQL Error: pq: could not extend file "base/16384/297049": No space left on device in query CREATE TABLE "import"."osm_highway_linestring_gen1" AS (SELECT "osm_id",
ST_SimplifyPreserveTopology("geometry", 38.218514) as "geometry",
I think this is more a hardware issue that the functionality of the image etc. With a country like Indonesia, it has almost 26million buildings which make the DB big. I thing the suggestion to solve this problem would be to optimize your mapping file to import only the stuff that you really need to work with. Because if you import all roads and buildings of the continent imagine the size it will grow to ?. Feel free to open this @Gustry if you feel this is a limitation of this image