[BUG] Error creating IP2Geo data source based on geolite2-city

Question

[BUG] Error creating IP2Geo data source based on geolite2-city

Closed this issue 3 months ago · 5 comments

Describe the bug
I am not able to create a IP2Geo data source based on geolite2-city

"state": "CREATE_FAILED"

[2023-12-07T14:13:14,271][ERROR][o.o.g.i.a.PutDatasourceTransportAction] [os01] Failed to create datasource for city-datasource

To Reproduce
DevTool:
I run following Commands with the output true

PUT /_plugins/geospatial/ip2geo/datasource/city-datasource
{
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
"update_interval_in_days": 3
}

PUT /_plugins/geospatial/ip2geo/datasource/county-datasource
{
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-country/manifest.json",
"update_interval_in_days": 3
}

PUT /_plugins/geospatial/ip2geo/datasource/asn-datasource
{
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-asn/manifest.json",
"update_interval_in_days": 3
}

After this commands i check the status

GET /_plugins/geospatial/ip2geo/datasource

{
"datasources": [
{
"name": "county-datasource",
"state": "AVAILABLE",
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-country/manifest.json",
"update_interval_in_days": 3,
"next_update_at_in_epoch_millis": 1702216774131,
"database": {
"provider": "maxmind",
"sha256_hash": "vDWU9ftCwtd0Ms+jwE1Hxcpq+U3u3HtV72ozsohyHwo=",
"updated_at_in_epoch_millis": 1701813614000,
"valid_for_in_days": 30,
"fields": [
"continent_name",
"country_name",
"country_iso_code"
]
},
"update_stats": {
"last_succeeded_at_in_epoch_millis": 1701957653967,
"last_processing_time_in_millis": 79682
}
},
{
"name": "city-datasource",
"state": "CREATE_FAILED",
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
"update_interval_in_days": 3,
"next_update_at_in_epoch_millis": 1702217124379,
"database": {},
"update_stats": {
"last_failed_at_in_epoch_millis": 1701958394272
}
},
{
"name": "asn-datasource",
"state": "AVAILABLE",
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-asn/manifest.json",
"update_interval_in_days": 3,
"next_update_at_in_epoch_millis": 1702219857696,
"database": {
"provider": "maxmind",
"sha256_hash": "zrGVJeArAtSu1vZK/8JJVMyRSETRO++lyjfJ7CHtnqs=",
"updated_at_in_epoch_millis": 1701795609000,
"valid_for_in_days": 30,
"fields": [
"asn",
"organization_name",
"network"
]
},
"update_stats": {
"last_succeeded_at_in_epoch_millis": 1701960719302,
"last_processing_time_in_millis": 61461
}
}
]
}

LOGS:

[2023-12-07T14:00:54,276][INFO ][o.o.g.i.j.DatasourceUpdateService] [os01] GeoIP database creation succeeded for county-datasource and took PT1M19.682591428S seconds
[2023-12-07T14:13:14,271][ERROR][o.o.g.i.a.PutDatasourceTransportAction] [os01] Failed to create datasource for city-datasource
[2023-12-07T14:51:59,441][INFO ][o.o.g.i.j.DatasourceUpdateService] [os01] GeoIP database creation succeeded for asn-datasource and took PT1M1.461367381S seconds

Expected behavior
"state": "AVAILABLE"

Plugins
name component version
os02 opensearch-alerting 2.11.1.0
os02 opensearch-anomaly-detection 2.11.1.0
os02 opensearch-asynchronous-search 2.11.1.0
os02 opensearch-cross-cluster-replication 2.11.1.0
os02 opensearch-custom-codecs 2.11.1.0
os02 opensearch-geospatial 2.11.1.0
os02 opensearch-index-management 2.11.1.0
os02 opensearch-job-scheduler 2.11.1.0
os02 opensearch-knn 2.11.1.0
os02 opensearch-ml 2.11.1.0
os02 opensearch-neural-search 2.11.1.0
os02 opensearch-notifications 2.11.1.0
os02 opensearch-notifications-core 2.11.1.0
os02 opensearch-observability 2.11.1.0
os02 opensearch-performance-analyzer 2.11.1.0
os02 opensearch-reports-scheduler 2.11.1.0
os02 opensearch-security 2.11.1.0
os02 opensearch-security-analytics 2.11.1.0
os02 opensearch-sql 2.11.1.0
os03 opensearch-alerting 2.11.1.0
os03 opensearch-anomaly-detection 2.11.1.0
os03 opensearch-asynchronous-search 2.11.1.0
os03 opensearch-cross-cluster-replication 2.11.1.0
os03 opensearch-custom-codecs 2.11.1.0
os03 opensearch-geospatial 2.11.1.0
os03 opensearch-index-management 2.11.1.0
os03 opensearch-job-scheduler 2.11.1.0
os03 opensearch-knn 2.11.1.0
os03 opensearch-ml 2.11.1.0
os03 opensearch-neural-search 2.11.1.0
os03 opensearch-notifications 2.11.1.0
os03 opensearch-notifications-core 2.11.1.0
os03 opensearch-observability 2.11.1.0
os03 opensearch-performance-analyzer 2.11.1.0
os03 opensearch-reports-scheduler 2.11.1.0
os03 opensearch-security 2.11.1.0
os03 opensearch-security-analytics 2.11.1.0
os03 opensearch-sql 2.11.1.0
os01 opensearch-alerting 2.11.1.0
os01 opensearch-anomaly-detection 2.11.1.0
os01 opensearch-asynchronous-search 2.11.1.0
os01 opensearch-cross-cluster-replication 2.11.1.0
os01 opensearch-custom-codecs 2.11.1.0
os01 opensearch-geospatial 2.11.1.0
os01 opensearch-index-management 2.11.1.0
os01 opensearch-job-scheduler 2.11.1.0
os01 opensearch-knn 2.11.1.0
os01 opensearch-ml 2.11.1.0
os01 opensearch-neural-search 2.11.1.0
os01 opensearch-notifications 2.11.1.0
os01 opensearch-notifications-core 2.11.1.0
os01 opensearch-observability 2.11.1.0
os01 opensearch-performance-analyzer 2.11.1.0
os01 opensearch-reports-scheduler 2.11.1.0
os01 opensearch-security 2.11.1.0
os01 opensearch-security-analytics 2.11.1.0
os01 opensearch-sql 2.11.1.0

Host/Environment (please complete the following information):

OS: UBUNTU 23.04
Opensearch: v 2.11.1

Answer 1 · 2023-12-07T19:00:17.000Z

Thanks for the issue report. Did you find any timeout error in the log?
Could you try to increase plugins.geospatial.ip2geo.timeout value to 60s?

Answer 2 · 2023-12-08T09:34:56.000Z

I try to set the value in opensearch.yml and get following error

plugins.geospatial.ip2geo.timeout: 60

ntation for the next major version.
[2023-12-08T09:10:32,456][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [os01] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: failed to parse setting [plugins.geospatial.ip2geo.timeout] with value [60] as a time value: unit is missing or unrecognized
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.11.1.jar:2.11.1]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.11.1.jar:2.11.1]
Caused by: java.lang.IllegalArgumentException: failed to parse setting [plugins.geospatial.ip2geo.timeout] with value [60] as a time value: unit is missing or unrecognized
at org.opensearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:406) ~[opensearch-common-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.Setting.lambda$minTimeValueParser$45(Setting.java:2674) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.Setting$MinTimeValueParser.apply(Setting.java:2556) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.Setting$MinTimeValueParser.apply(Setting.java:2518) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.Setting.get(Setting.java:483) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.Setting.get(Setting.java:477) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:644) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.common.settings.SettingsModule.(SettingsModule.java:178) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.node.Node.(Node.java:578) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.node.Node.(Node.java:407) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.11.1.jar:2.11.1]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.1.jar:2.11.1]
... 6 more
uncaught exception in thread [main]
java.lang.IllegalArgumentException: failed to parse setting [plugins.geospatial.ip2geo.timeout] with value [60] as a time value: unit is missing or unrecognized
at org.opensearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:406)
at org.opensearch.common.settings.Setting.lambda$minTimeValueParser$45(Setting.java:2674)
at org.opensearch.common.settings.Setting$MinTimeValueParser.apply(Setting.java:2556)
at org.opensearch.common.settings.Setting$MinTimeValueParser.apply(Setting.java:2518)
at org.opensearch.common.settings.Setting.get(Setting.java:483)
at org.opensearch.common.settings.Setting.get(Setting.java:477)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:644)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519)
at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489)
at org.opensearch.common.settings.SettingsModule.(SettingsModule.java:178)
at org.opensearch.node.Node.(Node.java:578)
at org.opensearch.node.Node.(Node.java:407)
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

Answer 3 · 2023-12-08T10:04:27.000Z

I fixed it ;-)

PUT _cluster/settings
{
"transient" : {
"plugins.geospatial.ip2geo.timeout" : "60s"
}
}

PUT /_plugins/geospatial/ip2geo/datasource/city-datasource
{
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
"update_interval_in_days": 3
}

GET /_plugins/geospatial/ip2geo/datasource/city-datasource

{
"datasources": [
{
"name": "city-datasource",
"state": "AVAILABLE",
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
"update_interval_in_days": 3,
"next_update_at_in_epoch_millis": 1702288412704,
"database": {
"provider": "maxmind",
"sha256_hash": "oJlZSzI1ZoFaNWhfc3/HA6HC9zgvF3kn9ie42jhVDr0=",
"updated_at_in_epoch_millis": 1701813614000,
"valid_for_in_days": 30,
"fields": [
"country_iso_code",
"country_name",
"continent_name",
"region_iso_code",
"region_name",
"city_name",
"time_zone",
"location"
]
},
"update_stats": {
"last_succeeded_at_in_epoch_millis": 1702029691549,
"last_processing_time_in_millis": 478397
}
}
]
}

Answer 4 · 2023-12-12T11:17:09.000Z

How can i use a ip2geo pipeline during ingestion?

Answer 5 · 2024-01-08T19:52:55.000Z

How can i use a ip2geo pipeline during ingestion?

Take a look at this doc. https://opensearch.org/docs/latest/ingest-pipelines/processors/ip2geo/#using-the-processor