Migrations hang when "default=" is set on a new field, while USING HASH.

Question

Migrations hang when "default=" is set on a new field, while USING HASH.

gnat opened this issue 2 years ago · 10 comments

What works.

First time table creation, and most other migrations work fine.

What does not work.

Migrations hang when default= is set on a new field, while USING HASH in the table.

message = models.TextField(default='') # Hangs migration.
message2 = models.TextField(null=True) # Works.

Issue Details

This issue is mentioned here: https://github.com/django/django/blob/7e3c9c3205e9646261cea5e5a7af7ec0e806690a/docs/ref/migration-operations.txt#L163

I believe, when the migration has to perform a full rewrite of the table, Django cannot understand crdb_internal_id_shard_16 (or how it fits into the model) when USING HASH is specified.

Possible resolution

Is there a way to ignore or hide fields which CockroachDB generates automatically when using hash sharded index?

If we can get Django to simply ignore anything of crdb_internal_* we could be okay.

Related: 78049 #264

Answer 1 · 2022-08-25T13:11:18.000Z

Same results on both:

django 4.1
django-cockroachdb 4.1
psycopg2
cockroachdb 22.1.6

And latest development versions

django (django/django#15687)
django-cockroachdb (timgraham@2fcfa7c)
psycopg 3.0.16
cockroachdb 22.1.6

Answer 2 · 2022-08-25T13:43:20.000Z

Have you identified the offending query? I imagine it's just taking really a long time rather than hanging indefinitely? Are you saying the query is unneeded and can be skipped?

Answer 3 · 2022-08-25T14:31:10.000Z

Full walk through / report.

Simple 3 node cluster. Completely fresh from cockroach init.

Added to django_cockroachdb/base.py:

#BigAutoField='DEFAULT unique_rowid()',
BigAutoField='USING HASH DEFAULT unique_rowid()',

Initial model.

class Listing(models.Model):
	title = models.CharField(max_length=150)
	price = models.IntegerField()

First migration.

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying products.0001_initial... OK
  Applying sessions.0001_initial... OK

SQL after initial migration:

root@:26257/test_django> show create products_listing;
     table_name    |                                                        create_statement
-------------------+---------------------------------------------------------------------------------------------------------------------------------
  products_listing | CREATE TABLE public.products_listing (
                   |     crdb_internal_id_shard_16 INT8 NOT VISIBLE NOT NULL AS (mod(fnv32(crdb_internal.datums_to_bytes(id)), 16:::INT8)) VIRTUAL,
                   |     id INT8 NOT NULL DEFAULT unique_rowid(),
                   |     title VARCHAR(150) NOT NULL,
                   |     price INT8 NOT NULL,
                   |     CONSTRAINT products_listing_pkey PRIMARY KEY (id ASC) USING HASH WITH (bucket_count=16)
                   | )
(1 row)


Time: 35ms total (execution 34ms / network 0ms)

Adding some data:

INSERT INTO products_listing (title, price) VALUES ('Test Product', '55');
INSERT INTO products_listing (title, price) VALUES ('Test Product B', '60');

root@:26257/test_django>  select * from products_listing;
          id         |     title      | price
---------------------+----------------+--------
  790911495589101569 | Test Product   |    55
  790911510319955969 | Test Product B |    60
(2 rows)


Time: 4ms total (execution 4ms / network 0ms)

Updated model.

# Create your models here.
class Listing(models.Model):
	title = models.CharField(max_length=150)
	price = models.IntegerField()
	description = models.TextField(default='')

Second migration.

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
  Applying products.0002_listing_description...

⚠️ Hangs indefinitely here! ⚠️

The query that indefinitely is in REVERTING, RETRYING:

ALTER TABLE test_django.public.products_listing ADD COLUMN description STRING NOT NULL DEFAULT ''

Answer 4 · 2022-08-25T14:59:24.000Z

Report Part 2.

When I SIGTERM cockroach.

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
  Applying products.0002_listing_description...Traceback (most recent call last):
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
  File "/home/gnat/.local/lib/python3.10/site-packages/psycopg/cursor.py", line 560, in execute
    raise ex.with_traceback(None)
psycopg.OperationalError: consuming input failed: EOF detected

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gnat/Desktop/test_django/manage.py", line 22, in <module>
    main()
  File "/home/gnat/Desktop/test_django/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/base.py", line 96, in wrapper
    res = handle_func(*args, **kwargs)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 349, in handle
    post_migrate_state = executor.migrate(
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/migration.py", line 130, in apply
    operation.database_forwards(
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/migrations/operations/fields.py", line 108, in database_forwards
    schema_editor.add_field(
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 680, in add_field
    self.execute(sql, params)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/postgresql/schema.py", line 53, in execute
    return super().execute(sql, None)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 199, in execute
    cursor.execute(sql, params)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 103, in execute
    return super().execute(sql, params)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/gnat/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
  File "/home/gnat/.local/lib/python3.10/site-packages/psycopg/cursor.py", line 560, in execute
    raise ex.with_traceback(None)
django.db.utils.OperationalError: consuming input failed: EOF detected

Fresh database. Exact same setup, but instead `null=True`

# Create your models here.
class Listing(models.Model):
	title = models.CharField(max_length=150)
	price = models.IntegerField()
	description = models.TextField(null=True)

Migration successful! 👍

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
  Applying products.0002_listing_description... OK

New migration, trying `default=` again.

# Create your models here.
class Listing(models.Model):
	title = models.CharField(max_length=150)
	price = models.IntegerField()
	description = models.TextField(null=True)
	description2 = models.TextField(default='a')

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, products, sessions
Running migrations:
  Applying products.0003_listing_description2...

⚠️ 😢 Hangs indefinitely here... again! ⚠️

Query, again indefinitely REVERTING, RETRYING:

ALTER TABLE test_django2.public.products_listing ADD COLUMN description2 STRING NOT NULL DEFAULT 'a'

Answer 5 · 2022-08-25T15:49:48.000Z

Can you reproduce the problem outside of Django, i.e. by executing the SQL statements that Django generates using cockroachdb sql? It's not clear to me that this isn't a problem with CockroachDB itself.

Answer 6 · 2022-08-25T16:24:03.000Z

Oh wow, you're correct @timgraham It's a Cockroach DB issue! This is a bad bug. 😮

Answer 7 · 2022-08-25T16:30:09.000Z

Thanks for this report! Could you share the way of reproducing the bug using only cockroach sql? And file an issue at https://github.com/cockroachdb/cockroach (or post here, and I can file it for you).

Answer 8 · 2022-08-25T17:03:16.000Z

@rafiss cockroachdb/cockroach#86889

Answer 9 · 2022-08-25T17:37:33.000Z

Closing, this bug has been triaged.

Answer 10 · 2022-08-25T17:49:37.000Z

Great, thanks. Closing as "not completed" since it's not applicable to this repo.