Final 'Cities' data pull takes too long, and breaks (probably times itself out?) :(

Question

Final 'Cities' data pull takes too long, and breaks (probably times itself out?) :(

pareshpandit opened this issue 2 years ago · 1 comments

Hi: Thank you for this wonderful project, to begin with. :)

I have been trying, for a couple of days now, to integrate 'django-cities-light' in one of my Django (4.0) projects which I run in an virtualenv on a Fedora 36 (local machine - development) installation, with Python 3.10.4. Also to note here, that the project is using a remote PostGreSQL DB, hosted with ElephantSQL (on an AWS HK server). Hope this is enough pertinent information.

Once I start the app (which is registered in INSTALLED_APPS for the project), with

python manage.py cities_light

-- it starts downloading fields and migrates them, albeit at a slow rate. But if am patient, I am able to successfully integrate everything one by one, down from Country > Region/states > Subregions, and their hierarchy. However, the last (and largest) bit of Cities, seems to never end, or populate anything.

(Side note: I am not filtering any data out - need all countries > states > cities ).

Yesterday night, I also tried

./manage.py cities_light --force-import-all

, and kept it running through the night, to give it enough time. I guess it timed itself out or something, and I woke up to a broken sync (once again), and 0 Cities entries in the DB Table.

Could there not be a way to initialise with a complete [+ somehwat latest] dump, and then initialise the django_cities app in your Django project, so that it first updates the DB with this "about-latest" data dump ; and then, when given the '-import-latest' command (or something like that) on the CLI, will reconcile with the geonames.org environment for updates/changes or additions, and then only deal with it to that extent? (#suggestion)

[On the side: I have also now created a new project with no venv, and a standard local serverless SQLite Django DB, to see if that helps. ]

For now, this is what showed up on my terminal. Kindly check it out, and let me know, if there's something I could do to fix it, and get Cities populated quickly. (Many thanks in advance.)

...
...
...

(env) ┌─[✗]─[paresh@192]─[[../../my project path on Fedora36/]
TS _PRP]
└──╼ $./manage.py cities_light --force-import-all
Assuming local download is up to date for http://download.g
eonames.org/export/dump/countryInfo.txt
Assuming local download is up to date for http://download.g
eonames.org/export/dump/admin1CodesASCII.txt
Assuming local download is up to date for http://download.g
eonames.org/export/dump/admin2Codes.txt
Traceback (most recent call last):
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/fields/related_de
scriptors.py", line 187, in get
rel_obj = self.field.get_cached_value(instance)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/fields/mixins.py"
, line 15, in get_cached_value
return instance._state.fields_cache[cache_name]
KeyError: 'country'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/backends/utils.py", line
89, in _execute
return self.cursor.execute(sql, params)
psycopg2.OperationalError: terminating connection due to ad
ministrator command
SSL connection has been closed unexpectedly

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "[../../my project path on Fedora36/]/./manage.p
y", line 22, in
main()
File "[../../my project path on Fedora36/]/./manage.p
y", line 18, in main
execute_from_command_line(sys.argv)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/core/management/init.py
", line 446, in execute_from_command_line
utility.execute()
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/core/management/init.py
", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/core/management/base.py", l
ine 414, in run_from_argv
self.execute(*args, **cmd_options)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/core/management/base.py", l
ine 460, in execute
output = self.handle(*args, **options)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/cities_light/management/commands/c
ities_light.py", line 217, in handle
self.subregion_import(items)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/cities_light/management/commands/c
ities_light.py", line 471, in subregion_import
self.save(
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/cities_light/management/commands/c
ities_light.py", line 697, in save
model.save(
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/base.py", line 80
6, in save
self.save_base(
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/base.py", line 84
1, in save_base
pre_save.send(
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/dispatch/dispatcher.py", li
ne 176, in send
return [
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/dispatch/dispatcher.py", li
ne 177, in
(receiver, receiver(signal=self, sender=sender, **named
))
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/cities_light/receivers.py", line 2
5, in set_display_name
instance.display_name = instance.get_display_name()
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/cities_light/abstract_models.py",
line 164, in get_display_name
return '%s, %s' % (self.name, self.country.name)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/fields/related_de
scriptors.py", line 205, in get
rel_obj = self.get_object(instance)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/fields/related_de
scriptors.py", line 168, in get_object
return qs.get(self.field.get_reverse_related_filter(ins
tance))
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/query.py", line 4
92, in get
num = len(clone)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/query.py", line 3
02, in len
self._fetch_all()
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/query.py", line 1
507, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/query.py", line 5
7, in iter
results = compiler.execute_sql(
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/models/sql/compiler.py",
line 1361, in execute_sql
cursor.execute(sql, params)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/backends/utils.py", line
103, in execute
return super().execute(sql, params)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/backends/utils.py", line
67, in execute
return self._execute_with_wrappers(
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/backends/utils.py", line
80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/backends/utils.py", line
84, in execute
with self.db.wrap_database_errors:
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/utils.py", line 91, in _
exit
raise dj_exc_value.with_traceback(traceback) from exc_v
alue
File "[../../my project path on Fedora36/]/env/lib64/
python3.10/site-packages/django/db/backends/utils.py", line
89, in _execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: terminating connection du
e to administrator command
SSL connection has been closed unexpectedly

Many thanks in advance, for your help and guidance! :)

Answer 1 · 2022-05-27T05:20:10.000Z

Another update in the interim:

My experiment with setting up a New Django Project with local SQLite DB, and no VENV also didn´t bear fruit. There too, all is fine until we are at the stage where we are pulling-in Cities.

It failed, and blurted out the following :

┌─[paresh@fedora]─[/MyProjectDIRpath/citiesproj]
└──╼ $python manage.py cities_light
Traceback (most recent call last):
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/run/media/paresh/DATA/django_dev/django-cities-light/citiesproj/manage.py", line 22, in
main()
File "/run/media/paresh/DATA/django_dev/django-cities-light/citiesproj/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/paresh/.local/lib/python3.10/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/home/paresh/.local/lib/python3.10/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/paresh/.local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/paresh/.local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/paresh/.local/lib/python3.10/site-packages/cities_light/management/commands/cities_light.py", line 217, in handle
self.subregion_import(items)
File "/home/paresh/.local/lib/python3.10/site-packages/cities_light/management/commands/cities_light.py", line 471, in subregion_import
self.save(
File "/home/paresh/.local/lib/python3.10/site-packages/cities_light/management/commands/cities_light.py", line 697, in save
model.save(
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/base.py", line 726, in save
self.save_base(using=using, force_insert=force_insert,
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/base.py", line 763, in save_base
updated = self._save_table(
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/base.py", line 868, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/base.py", line 906, in _do_insert
return manager._insert(
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 98, in execute
return super().execute(sql, params)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
with self.db.wrap_database_errors:
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/utils.py", line 90, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/home/paresh/.local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

I have now set it to --force-import-all, to see if that would help, if at all....

$ ./manage.py cities_light --force-import-all
Assuming local download is up to date for http://download.geonames.org/export/dump/countryInfo.txt
Assuming local download is up to date for http://download.geonames.org/export/dump/admin1CodesASCII.txt
Assuming local download is up to date for http://download.geonames.org/export/dump/admin2Codes.txt