Running rake db:migrate when version is the newest rollsback the migration.

[brendanstennett]

Basically running rake db:migrate twice in a row reverts the previous migration. I would have expected it to do nothing the second time. Is this a feature or a bug?

[adacosta]

This sounds like a bug. I haven't confirmed the behavior. If you'd like to create the fix, please spin up a pull request with the fix and a test, and I'll merge.

[jarthod]

@brendanstennett I've never noticed this and it seems nobody else reported the same bug so unless you can still reproduce on the latest version I'm closing this issue.

[andrcuns]

@jarthod I'm running in to the same issue using the latest version. I wonder what could cause this. This behaviour completely brakes the application.

worker:~/app$ bundle exec rails db:migrate
[2021-08-22 07:03:37 +0000 tid=19wz] INFO: Migrating to Projects (20210820183037)
==  Projects: migrating =======================================================
==  Projects: migrated (0.0059s) ==============================================

worker:~/app$ bundle exec rails db:migrate
[2021-08-22 07:03:43 +0000 tid=19x5] INFO: Migrating to Projects (20210820183037)
==  Projects: reverting =======================================================
==  Projects: reverted (0.0065s) ==============================================

[jarthod]

@andrcuns I still cannot reproduce with the latest version so if you can find the reason the code is not working as expected in your situation please let us know.

[andrcuns]

@jarthod Yes, it is something weird and most likely data or maybe even OS related?. I can reproduce it in my production environment which runs in a linux based docker container but it works completely fine on a local dev environment.

I will try to dig deep once I have some free time and hopefully find the cause as it renders migrations unusable for my prod setup

[jarthod]

Yes it's hard to say from the outside, you can start by comparing what's inside the data_migrations collection and the output of the rails db:migrate:status tasks between production and development.

[andrcuns]

@jarthod I did compare objects in dev and prod and they seem to be the same:

dev:

$ pry(main)> client[:data_migrations].find.to_a
=> [{"_id"=>BSON::ObjectId('614c3b1e2c5a715261d9926d'), "version"=>"20210820183037"}]

$ bundle exec rails db:migrate:status
database: dependabot_gitlab_development

 Status   Migration ID    Migration Name
--------------------------------------------------
   up     20210820183037  Projects

$ bundle exec rails db:migrate
[2021-09-23 08:55:21 +0000 tid=1a4x] INFO: Migrating to Projects (20210820183037)

prod:

$ irb(main):014:0> migrations.find.to_a
=> [{"_id"=>BSON::ObjectId('614c3bb95fa6c600acd91ea6'), "version"=>"20210820183037"}]

$ bundle exec rails db:migrate:status
database: dependabot

 Status   Migration ID    Migration Name
--------------------------------------------------
   up     20210820183037  Projects

$ bundle exec rails db:migrate
[2021-09-23 09:02:46 +0000 tid=17k9] INFO: Migrating to Projects (20210820183037)
==  Projects: reverting =======================================================
==  Projects: reverted (0.0099s) ==============================================

Yet on the prod running migration will revert it and on development it will not.

Any other pointers where to look for differences? Could it be some rails configuration parameter?

[jarthod]

Interesting, could you share your Mongoid configuration in production? (removing the passwords of course) also can you show the migration file content and path? do you have other migrations in this project? Is the "timestamped_migrations" option set anywhere? (defaults to true). You don't have any MONGOID_CLIENT_NAME env variable set anywhere I suppose?

[andrcuns]

The Mongoid configuration is rather simple and almost identical for development and production aside from timeout values. It's configured dynamically in initializer to support all options via URI which can't be done via yml config.

rails_env = ENV["RAILS_ENV"] || "development"
mongoid_config = {
  hosts: [ENV["MONGODB_URL"] || "localhost:27017"],
  database: ENV["MONGODB_DATABASE"] || "dependabot_gitlab_#{rails_env}",
  options: {
    server_selection_timeout: 1,
    connect_timeout: 1,
    retry_writes: ENV["MONGODB_RETRY_WRITES"] || "true"
  }
}

if Rails.env.production?
  mongoid_config[:options].tap do |options|
    options[:server_selection_timeout] = 5
    options[:connect_timeout] = 5
    options[:user] = ENV["MONGODB_USER"]
    options[:password] = ENV["MONGODB_PASSWORD"]
  end
end

Mongoid.configure do |config|
  config.app_name = "DependabotGitlab"
  config.clients.default = (Rails.env.production? && ENV["MONGODB_URI"]) ? { uri: ENV["MONGODB_URI"] } : mongoid_config # rubocop:disable Style/TernaryParentheses
end

I have only 1 single migration since the app is still in rather early stages and it didn't really have many users so I didn't bother with migrations before. Migration file looks like this:

db/migrate/20210820183037_projects.rb:

class Projects < Mongoid::Migration
  def self.up
    Project.all do |project|
      project.rename(project_id: :id)
    end
  end

  def self.down
    Project.all do |project|
      project.rename(id: :project_id)
    end
  end
end

The MONGOID_CLIENT_NAME is not set and "timestamped_migrations" is not set as well. I was not even aware of such option.

I was also able to replicate this with both local dockerized setup and using MongoDB Atlas, so it doesn't seem to be related to database setup itself.

[andrcuns]

You can see the whole setup here: https://gitlab.com/dependabot-gitlab/dependabot

[jarthod]

Ok thanks, so in production where it's not working you're using MONGODB_URI only? (is it Altas here?) and so no other specific mongodb options are used. Did you find this way of setting up the configuration somewhere in the documentation? it doesn't seem very safe as it does not use the standard writers and thus validations. When configuration Mongoid from ruby it's recommended to use config.load_configuration which is public and tested, because it does validate the config you pass. I'm not saying this is the cause of the problem here but modifying the inside of the default config objects directly may not be a great idea (e.g. mongoid already have a connection initialized to a different DB and is not aware of the change, etc..)

You said you were able to replicate this with a local dockerized setup, that's good to know to make it easier to debug, how did you do this? which MONGODB_URI did you use and which mongodb image / how did you start it? Was this in production env or development env?

[andrcuns]

No, it's not working regardless if MONGODB_URI or plain MONGODB_URL with username and password is used. That is why I mentioned it doesn't seem to be related to database itself.

The way it is configured now is mentioned in the mongoid documentation under configuration page on how to configure it directly with ruby: https://docs.mongodb.com/mongoid/current/tutorials/mongoid-configuration/. Thanks for the pointer about load_configuration method, was not aware of it.

You should be able to reproduce it simply by cloning this repo: https://gitlab.com/dependabot-gitlab/dependabot and starting it via docker compose in production mode.

$ docker compose -f docker-compose.yml -f docker-compose-prod.yml up

[jarthod]

Ok thanks for the confirmation and for the link to the documentation.

I tried running the app locally and after some fiddling (installing docker, docker-compose, then docker-compose again but another more recent version, then changing the ports because it was conflicting with my local instance, etc..) I managed to open a console in production which was reproducing the issue. So I then executed the same commands rails db:migrate does, starting at the entrypoint and that's where I found the issue: the VERSION environment variable is defined (with v0-10-1-f11657d0), and this variable is used by rails migration to force a specific version to migrate to.

So that's what is telling db:migrate to either migrate up or down depending if current_version (0 or 20210820183037) is greater or not than target ("v0-10-1-f11657d0".to_i so 0). So you'll need to remove/rename this variable to avoid the conflict or at least unset it when using db:migrate. This is a standard from Rails (not just mongoid_rails_migrations) so we're not gonna change this implementation.

One thing which could be added to make such debugging easier here is a display of the current value of VERSION (and the parsed number) when the command starts, because it's true that with this kind of docker-compose stack it's easier to miss it than before. I'll let anybody submit a PR for this if they want ;)

[andrcuns]

@jarthod Thanks a lot for clearing this up. Naturally I would not expect for this project to change because of that.

At least if someone stumbles to same issue, there is now some context. I believe VERSION variable is sometimes used when building docker images, but I was not aware of it being used by rails.

[jarthod]

Yes this Rails convention pre-dates docker but it's a very standard variable name made more popular by docker so I suppose you won't be the only one. Here is the issue raised to the Rails team a couple years ago: rails/rails#30707
Apparently they ended up doing something similar to what I recommend in ActiveRecord which is to check the VERSION passed and raise an error (and show the value) if it doesn't look like a migration version: rails/rails#30714