Add natural earth data

Question

Add natural earth data

Opened this issue 4 years ago · 14 comments

https://www.naturalearthdata.com/downloads/10m-cultural-vectors/

Answer 1 · 2022-01-11T12:24:05.000Z

Hey @henrykironde , i want to work on this issue , can you guide me how to proceed for this issue? i already downloaded the dataset but i'm not getting how to add this dataset/files.

Answer 2 · 2022-01-11T19:39:53.000Z

@Nageshbansal thank you for the interest in working on this issue. Since these are vector files, once you have download the dataset, run retriever autocreate -dt vector -d path-to-the-directory-containing-the- data-files.
This will give you a template that you can work with. Let me know how that goes

Answer 3 · 2022-01-11T20:19:16.000Z

@henrykironde i tried this , and i'm getting some error in some files so working on that
Thanks for letting me know

Answer 4 · 2022-01-12T19:39:14.000Z

@henrykironde installed GDAL successfully , but still getting this error
"Skipped file: ne_10m_admin_0_antarctic_claims.shp name 'ogr' is not defined
Remove the file from the folder and try again"

Answer 5 · 2022-01-12T19:44:53.000Z

Try runing imports from python and report back what you get.

➜  rdataretriever git:(main) ✗ python
Python 3.9.6 (default, Jun 29 2021, 05:25:02) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> from osgeo import ogr
>>> from osgeo import gdal
>>>

Answer 6 · 2022-01-12T19:49:54.000Z

Import error i.e osgeo not found.
I have installed GDAL by using OSGeo4W installer

Answer 7 · 2022-01-13T11:32:16.000Z

@henrykironde successfully created template of data.

Answer 8 · 2022-01-13T11:59:30.000Z

Good, now you will populate the fields based on the appropriate meta data.
For example from reading about, it takes me to the GitHub page where the files are hosted.https://github.com/nvkelso/natural-earth-vector/blob/master/LICENSE.md

From the repo I see they have a license file that states
"non-exclusive license", I would use that in the license field.
And so on.

When working with raster data, you need to download Qgis. This tools can help you look at the various data and even get more info from the properties.

Answer 9 · 2022-01-13T12:00:40.000Z

henrykironde commented 3 years ago

Answer 10 · 2022-01-13T12:01:33.000Z

henrykironde commented 3 years ago

Answer 11 · 2022-01-13T12:02:07.000Z

Got it, will try to do it asap,
Thanks for this information
:)

Answer 12 · 2022-01-13T12:04:20.000Z

You can drag and drop the .shp file into Qgis

Answer 13 · 2022-01-18T10:53:18.000Z

@henrykironde, I have added the dataset, tested it by using PostGIS in qgis, out of 81 tables, 75 are able to add to the database, rest of 6 is throwing error .for e.g NOTICE: table "ne_10m_admin_0_countries_bdg" does not exist, skipping and when i tried to debug it i found out that this file ne_10m_admin_0_countries_bdg exist but it's not able to be read

Create a PR on the Retriever-recipes repo, I will look at it.

Answer 14 · 2022-01-18T13:17:40.000Z

okay I will create a PR, and there is one thing I want to know that by using "extract_all": "True", it extracts shapefiles
in /.retriever/raw_data/cultural-10m/10m-cultural/ folder due to which retriever is not able to get shapefiles
can you tell me is there any way to extract files in /.retriever/raw_data/cultural-10m/ itself.

Also,
I think 6 tables are left out because their schema holds some fields which have null or zero values and according
to SQL syntax, they can't be null . for e.g. size attribute of FCLASS_US have value 0 in fields of the schema of
ne_10m_admin_0_countries_bdg.

                  {
                       "name": "FCLASS_US",
                        "precision": 0,
                        "size": 0,
                        "type": "String"
                    },