kevlened/fireway

How to run a specific migration file?

kiptoomm opened this issue · 5 comments

kiptoomm commented

Is there a way to run a specific migration if I have multiple scripts living in my migrations/ folder? Lately, I have had a few use cases where I need to re-run a script whose entry is already in the fireway collection on Firestore. However, to actually make a script to be executed, I have found that I have to manually delete the entry corresponding to that script and others that came after it. For example, let's say I have:

migrations/v0.0.1__first_migration.js
migrations/v0.0.2__second_migration.js
migrations/v0.0.3__third_migration.js

If I need to run just v0.0.2__second_migration.js, I have to delete both the corresponding v2 and v3 entries in the fireway collection and then rerun fireway migrate. Wouldn't it be nice if I could do something like fireway migrate --script=v0.0.2?

kiptoomm commented

Is there a way to run a specific migration if I have multiple scripts living in my migrations/ folder? Lately, I have had a few use cases where I need to re-run a script whose entry is already in the fireway collection on Firestore. However, to actually make a script to be executed, I have found that I have to manually delete the entry corresponding to that script and others that came after it. For example, let's say I have:

migrations/v0.0.1__first_migration.js
migrations/v0.0.2__second_migration.js
migrations/v0.0.3__third_migration.js

If I need to run just v0.0.2__second_migration.js, I have to delete both the corresponding v2 and v3 entries in the fireway collection and then rerun fireway migrate. Wouldn't it be nice if I could do something like fireway migrate --script=v0.0.2?

If I delete only the v2, and NOT the v3 entry, doing fireway migrate produces the following, and nothing else happens:

Found 3 migration files
Executing 0 migration files
Finished all firestore migrations
Files scanned:3 executed:0
Docs added:0 created:0 updated:0 set:0 deleted:0

If I delete only the v3 entry, and NOT the v2 entry, only the v0.0.3 script will be run, and it ignores the v0.0.2 script. In other words, I have to delete BOTH v2 and v3 scripts in order for the migrations to be re-run

kevlened commented

Hmm. fireway only does roll-forward migrations, so migrations should only be run once per database (once for staging, once for prod). If you'd like to make additional changes, create another migration file (v0.0.4__something.js), and rerun fireway. The only time you should need to rerun a migration, or remove entries from the fireway collection, is if your last migration failed.

The last code sample you sent had a try...catch block around it. Ideally, you'd remove that and let fireway catch any errors. That way, it'll stop before it runs any more migrations.

If you're iterating on a migration, use --dryrun to ensure you don't modify the production db.

Does that help?

kiptoomm commented

fireway only does roll-forward migrations

I understand, by the way what is the underlying reason for this current limitation?
In any case, it'd be very nice if there was a way to do roll-backs, like alembic does. I know we're not in SQL land here but I'm allowed to dream, right? 😄

The only time you should need to rerun a migration, or remove entries from the fireway collection, is if your last migration failed.

There are times when I have had to rerun a migration because some older code was still being run, and it wrote data in the old schema/format.

If you're iterating on a migration, use --dryrun to ensure you don't modify the production db.

At the moment --dryrun is not working for me - regardless of my use of it, the changes are being applied to the records but the fireway collection entry is not added. I am also impacted by this related issue

kiptoomm commented

@kevlened any insights on the above questions?

kevlened commented

fireway only does roll-forward migrations

I understand, by the way what is the underlying reason for this current limitation?
In any case, it'd be very nice if there was a way to do roll-backs, like alembic does. I know we're not in SQL land here but I'm allowed to dream, right? 😄

I wouldn't stop your dream! However, the NoSQL limitation is exactly the reason. I saw an idea for firestore migration rollbacks by @gregdingle, but I don't plan on overhauling fireway for this flow.

There are times when I have had to rerun a migration because some older code was still being run, and it wrote data in the old schema/format.

Makes sense. To be successful in NoSQL, your client and server code needs to know how to handle multiple schemas (especially if you have mobile clients). You may be able to get around this with a fancy mapping of old to new properties on the server-side. Migrations don't solve the problem unless you have downtime, because there's no way to apply a migration atomically for large collections in firestore. In short, this is always going to be a problem, so your clients and server have to handle all old schemas until you're certain there are no more old clients.

If you're iterating on a migration, use --dryrun to ensure you don't modify the production db.

At the moment --dryrun is not working for me - regardless of my use of it, the changes are being applied to the records but the fireway collection entry is not added. I am also impacted by this related issue

I haven't been able to reproduce this problem. I would look for places that don't use the firestore instance that fireway returns. This includes models you may be trying to share with your clients.

  • 👍1