workforce-data-initiative/skills-airflow

Fix DAG integration test Heisenbug

thcrock opened this issue · 1 comments

This test https://github.com/workforce-data-initiative/skills-airflow/blob/master/tests/api_sync_v1/test_dag.py

occasionally deadlocks on Travis; the Base.metadata.create_all() in ensure_db sometimes tries to create a table which is already there. testing.postgresql should make sure that this doesn't happen between tests (it nukes the old database file!), but it's possible that parallel processes within one test could be clobbering each other.

The fix for this might be schema migrations.

Working on this might be not worth it until upgrading airflow: #9

Reason being that the upgrade changes how DAG integration tests have to be run, so it's entirely possible this will go away after the upgrade. It's also entirely possible that the fix that works under the current version wouldn't work under the new version.