has_total flag on FieldTable causes table creation and data import to fail

[xybrnet]

If has_total=False is set on a FieldTable, Wazimap fails to create the underlying database table and to import data if the table exists.

Removing the flag before creating/importing, and setting it again once done was used as a workaround.

[longhotsummer]

This happens because the FieldTable is trying to establish table columns by using the permutations of the fields in the table, but there are no rows in the table to use. Normally, this is silently ignored because the table is considered to only have a total column. On tables with has_total=False, it's not silent any more.

I see two possibilities:

1. keep raising an exception but improve the description to explain that the table must be filled with data.
2. stop raising an exception and simply have a table without columns.

My concern with 1 is that empty tables shouldn't be considered "bad". It's frustrating to block a call to python manage.py migrate just because you've declared a FieldTable but haven't imported data yet. The underlying problem isn't actually a badly declared table, it's just that it's empty.

I suggest we go with 2 and remove the exception. I'd appreciate thoughts from @cliftonmcintosh and @damian.

[longhotsummer]

I've tested option 2 (removing the exception) and Wazimap is happy with it, it just shows up as an empty table.

[cliftonmcintosh]

@longhotsummer

Thanks for the response. We usually run migrations before loading a table with data.

At what point do the other columns get created in a migration? For example, if declare

FieldTable(['lighting fuel'], universe='Population', table_per_level=False)

then migrations create columns geo_code, geo_level, "lighting fuel", and total.

If I declared the same table with has_total set to False, I would expect to see geo_code, geo_level, and "lighting fuel" created. Does that expectation fit what happens? Or is the table one with no columns?

[longhotsummer]

Good question. There are two issues at play here and they're a bit confusing:

1. Migrations: The data tables are created in the database when they're declared using with FieldTable or SimpleTable, they're entirely independent of the Django migration system (this is largely because we use SQLAlchemy for the data tables and not Django's ORM system). When you declare a new table it's created in the database the moment the declaration is encountered. That could be because you run python manage.py runserver or any other command that loads up the app.
2. The has_total flag doesn't impact the database. It simply indicates whether it makes sense to sum up the values in the table or not, which determines if Wazimap should show the option to express the table's values as percentages. For example, for a table that describes the gender split of a place, it makes sense to sum up all values to determine the number of people involved, and then express each gender as a percentage. This is because each person is counted only once. However, if we have a table that shows how many households in an area have a particular household good (computer, TV, stove, etc.), then we can't determine the number of households in the area by summing up all the values (a household might have many items and so be represented in more than one category).

Whether or not has_total is used, the underlying table structure will be the same (as described here). The total column in the database is not directly related to the has_total flag.

I hope this helps, it's a bit confusing because there's the logical data table (FieldTable, SimpleTable) and the underlying database table.

[cliftonmcintosh]

Yes, that helps. I also tried a SimpleTable for my use case but could not get a migration to run for it either. Do those require that the table in the database already exist? The error I got seemed to imply that SimpleTables expected the table to already by in postgres.

[longhotsummer]

It looks like the behaviour is different for SimpleTable and FieldTable, which is just weird. I've opened #60.

Yes, a SimpleTable must exist in the database first. A FieldTable can also exist first, or it will be created if it doesn't.

[longhotsummer]

Version 0.5.4 released which fixes this.

[longhotsummer]

I jumped the gun, pypi is giving me errors when uploading 0.5.4. I'll keep trying.

[longhotsummer]

0.5.4 is now up.