RSpec tests break shoulda-matcher inclusion depending on underlying order

Question

RSpec tests break shoulda-matcher inclusion depending on underlying order

fj opened this issue 12 years ago · 10 comments

We're observing the following issue:

  1) AccessToken
     Failure/Error: it { should validate_presence_of :application }
     NoMethodError:
       undefined method `validate_presence_of' for #<RSpec::Core::ExampleGroup::Nested_82:0x00000010bc8b20>
     # ./spec/models/access_token_spec.rb:5:in `block (2 levels) in <top (required)>'

This error repeats each time there is any use of a Shoulda matcher. For us, the error started appearing when we upgraded to RSpec 2.10.1, and wasn't present in RSpec 2.8. We went back to version 2.9 and it also had the same issue.

It seems to be related to filesystem ordering, because it appears on OSX and not Ubuntu. When Ubuntu users run the files in the OSX order (by pasting the printed list generated from bundle exec rake and running it directly), they experience the errors too. Likewise, if an OSX user pastes the Ubuntu order directly, it will work again for them.

Any hints to narrow down where the problem might be occurring? I opened a similar issue on RSpec here, and there's a similar issue already in Shoulda here, although that only covers one specific matcher whereas this problem happens for any Shoulda matcher.

We do require relevant files from tests, but as far as I know that is the only thing that could be mutating the runtime state in such a way as to affect this, and a require shouldn't affect whether the matchers are present or not.

Answer 1 · 2012-06-06T00:25:27.000Z

Can you paste both the failing (OSX) file order and the passing (Ubuntu) file order?

Answer 2 · 2012-06-06T00:26:10.000Z

Also: do you have any other RSpec matcher gems (like shoulda-matchers), and if so, are they also randomly working/not working?

Answer 3 · 2012-06-06T15:43:09.000Z

Here's the passing order:

bundle exec rspec ./spec/uploaders/image_uploader_spec.rb ./spec/jobs/location_export_spec.rb ./spec/jobs/export_job_spec.rb ./spec/jobs/user_export_spec.rb ./spec/jobs/geocode_visit_spec.rb ./spec/controllers/businesses_controller_spec.rb ./spec/controllers/group_memberships_controller_spec.rb ./spec/controllers/business_transaction_analytics_controller_spec.rb ./spec/controllers/location_transaction_analytics_controller_spec.rb ./spec/controllers/client_applications_controller_spec.rb ./spec/controllers/announcements_controller_spec.rb ./spec/controllers/messages_controller_spec.rb ./spec/controllers/group_analytics_controller_spec.rb ./spec/controllers/notifications_controller_spec.rb ./spec/controllers/locations_analytics_controller_spec.rb ./spec/controllers/group_locations_controller_spec.rb ./spec/controllers/rewards_codes_controller_spec.rb ./spec/controllers/access_tokens_controller_spec.rb ./spec/controllers/offers_controller_spec.rb ./spec/controllers/users_controller_spec.rb ./spec/controllers/twitter_accounts_controller_spec.rb ./spec/controllers/pages_controller_spec.rb ./spec/controllers/group_invitations_controller_spec.rb ./spec/controllers/business_analytics_controller_spec.rb ./spec/controllers/sessions_controller_spec.rb ./spec/controllers/transactions_controller_spec.rb ./spec/controllers/api/v1/businesses_controller_spec.rb ./spec/controllers/api/v1/group_memberships_controller_spec.rb ./spec/controllers/api/v1/notifications_controller_spec.rb ./spec/controllers/api/v1/location_employees_controller_spec.rb ./spec/controllers/api/v1/images_controller_spec.rb ./spec/controllers/api/v1/current_users_controller_spec.rb ./spec/controllers/api/v1/users_controller_spec.rb ./spec/controllers/api/v1/transactions_controller_spec.rb ./spec/controllers/api/v1/locations_controller_spec.rb ./spec/controllers/api/v1/advertisements_controller_spec.rb ./spec/controllers/api/v1/rewards_balance_entries_controller_spec.rb ./spec/controllers/api/v1/identifiers_controller_spec.rb ./spec/controllers/api/v1/categories_controller_spec.rb ./spec/controllers/api/v1/rewards_controller_spec.rb ./spec/controllers/api/v1/redemptions_controller_spec.rb ./spec/controllers/api/v1/status_controller_spec.rb ./spec/controllers/api/v1/groups_controller_spec.rb ./spec/controllers/api/v1/application_controller_spec.rb ./spec/controllers/advertisement_analytics_controller_spec.rb ./spec/controllers/locations_controller_spec.rb ./spec/controllers/contact_requests_controller_spec.rb ./spec/controllers/advertisements_controller_spec.rb ./spec/controllers/rewards_balance_entries_controller_spec.rb ./spec/controllers/identifiers_controller_spec.rb ./spec/controllers/categories_controller_spec.rb ./spec/controllers/rewards_controller_spec.rb ./spec/controllers/redemptions_controller_spec.rb ./spec/controllers/groups_controller_spec.rb ./spec/controllers/managed_businesses_controller_spec.rb ./spec/controllers/application_controller_spec.rb ./spec/acceptance/geocoding_spec.rb ./spec/lib/geo_searchable_spec.rb ./spec/lib/url_expert_spec.rb ./spec/lib/within_geo_scope_spec.rb ./spec/lib/geocoding_service_spec.rb ./spec/lib/blurbable_spec.rb ./spec/lib/url_fields_spec.rb ./spec/lib/o_auth_authenticatable_spec.rb ./spec/lib/identifier_spec.rb ./spec/lib/with_retry_spec.rb ./spec/lib/date_string_validator_spec.rb ./spec/lib/employee_searchable_spec.rb ./spec/lib/record_converter/record_spec.rb ./spec/lib/record_converter/converter_spec.rb ./spec/lib/record_converter/group_location_record_spec.rb ./spec/lib/record_converter/user_record_spec.rb ./spec/lib/ip_validator_spec.rb ./spec/lib/to_comma_separated_spec.rb ./spec/lib/lookup_concerns_spec.rb ./spec/mailers/analytics_mailer_spec.rb ./spec/models/visit_arbiter_spec.rb ./spec/models/messages/group_membership_confirmed_message_spec.rb ./spec/models/messages/group_advertisement_updated_message_spec.rb ./spec/models/messages/group_membership_rejected_message_spec.rb ./spec/models/messages/mass_message_spec.rb ./spec/models/messages/group_location_invitation_message_spec.rb ./spec/models/messages/group_advertisement_created_message_spec.rb ./spec/models/messages/group_user_invitation_message_spec.rb ./spec/models/messages/group_advertisement_deleted_message_spec.rb ./spec/models/messages/business_created_message_spec.rb ./spec/models/reserved_user_spec.rb ./spec/models/group_advertisement_spec.rb ./spec/models/group_search_spec.rb ./spec/models/address_spec.rb ./spec/models/transaction_point_mediator_spec.rb ./spec/models/rewards_balance_entry_spec.rb ./spec/models/address_component_spec.rb ./spec/models/image_spec.rb ./spec/models/offer_spec.rb ./spec/models/referenced_spec.rb ./spec/models/category_search_spec.rb ./spec/models/client_application_spec.rb ./spec/models/business_search_spec.rb ./spec/models/o_auth_mac_token_request_spec.rb ./spec/models/redemption_spec.rb ./spec/models/nonce_spec.rb ./spec/models/random_hash_spec.rb ./spec/models/advertisement_spec.rb ./spec/models/referencing_spec.rb ./spec/models/group_membership_spec.rb ./spec/models/system_message_specification_spec.rb ./spec/models/visit_policies/anonymous_policy_spec.rb ./spec/models/visit_policies/nil_policy_spec.rb ./spec/models/visit_policies/authenticated_policy_spec.rb ./spec/models/category_spec.rb ./spec/models/transaction_redemption_spec.rb ./spec/models/user_role_spec.rb ./spec/models/redemption_point_mediator_spec.rb ./spec/models/points_adjustment_spec.rb ./spec/models/merchant_summary_spec.rb ./spec/models/transaction_spec.rb ./spec/models/url_validator_spec.rb ./spec/models/transaction_group_spec.rb ./spec/models/contact_request_spec.rb ./spec/models/time_string_validator_spec.rb ./spec/models/user_authentication_search_spec.rb ./spec/models/lazy_redemption_spec.rb ./spec/models/group_spec.rb ./spec/models/time_throttled_message_specification_spec.rb ./spec/models/redemption_condition_spec.rb ./spec/models/time_string_spec.rb ./spec/models/analytics/group_timeseries_builder_spec.rb ./spec/models/analytics/visit_request_spec.rb ./spec/models/analytics/timeseries_builder_spec.rb ./spec/models/analytics/visit_timeseries_builder_spec.rb ./spec/models/analytics/transaction_request_spec.rb ./spec/models/analytics/timeseries_request_spec.rb ./spec/models/analytics/group_request_spec.rb ./spec/models/analytics/transaction_timeseries_builder_spec.rb ./spec/models/business_employee_spec.rb ./spec/models/api/v1/location_summary_spec.rb ./spec/models/api/v1/rewards_participant_spec.rb ./spec/models/api/v1/address_spec.rb ./spec/models/api/v1/rewards_balance_entry_spec.rb ./spec/models/api/v1/image_spec.rb ./spec/models/api/v1/offer_spec.rb ./spec/models/api/v1/category_search_spec.rb ./spec/models/api/v1/redemption_spec.rb ./spec/models/api/v1/advertisement_spec.rb ./spec/models/api/v1/category_spec.rb ./spec/models/api/v1/points_adjustment_spec.rb ./spec/models/api/v1/transaction_spec.rb ./spec/models/api/v1/notification_collection_spec.rb ./spec/models/api/v1/group_spec.rb ./spec/models/api/v1/reference_spec.rb ./spec/models/api/v1/location_spec.rb ./spec/models/api/v1/business_spec.rb ./spec/models/api/v1/user_spec.rb ./spec/models/api/v1/reward_spec.rb ./spec/models/api/v1/identifier_search_spec.rb ./spec/models/api/v1/location_search_spec.rb ./spec/models/api/v1/announcement_spec.rb ./spec/models/api/v1/message_spec.rb ./spec/models/api/v1/notification_spec.rb ./spec/models/group_location_spec.rb ./spec/models/role_spec.rb ./spec/models/access_token_request_spec.rb ./spec/models/location_spec.rb ./spec/models/json_resource_spec.rb ./spec/models/business_spec.rb ./spec/models/business_message_recipient_filter_spec.rb ./spec/models/phone_number_spec.rb ./spec/models/user_spec.rb ./spec/models/access_token_spec.rb ./spec/models/reward_spec.rb ./spec/models/visit_spec.rb ./spec/models/country_search_spec.rb ./spec/models/has_country_spec.rb ./spec/models/location_search_spec.rb ./spec/models/date_string_spec.rb ./spec/models/point_calculator_spec.rb ./spec/models/ability_spec.rb ./spec/models/twitter_account_spec.rb ./spec/models/redemption_conditions/user_redemption_count_spec.rb ./spec/models/redemption_conditions/user_redemption_rate_spec.rb ./spec/models/redemption_conditions/sufficient_rewards_points_spec.rb ./spec/models/external_url_spec.rb ./spec/models/message_spec.rb ./spec/models/permissions_spec.rb ./spec/models/hour_range_spec.rb ./spec/models/user_search_spec.rb ./spec/models/notification_spec.rb ./spec/presenters/messages_presenter_spec.rb ./spec/presenters/current_user_presenter_spec.rb ./spec/presenters/merchant_summary_presenter_spec.rb ./spec/presenters/rewards_presenter_spec.rb ./spec/presenters/advertisements_presenter_spec.rb ./spec/helpers/application_helper_spec.rb ./spec/helpers/users_helper_spec.rb ./spec/helpers/advertisements_helper_spec.rb ./spec/contexts/user_updated_context_spec.rb

Here's the failing order:

bundle exec rspec ./spec/acceptance/geocoding_spec.rb ./spec/contexts/user_updated_context_spec.rb ./spec/controllers/access_tokens_controller_spec.rb ./spec/controllers/advertisement_analytics_controller_spec.rb ./spec/controllers/advertisements_controller_spec.rb ./spec/controllers/announcements_controller_spec.rb ./spec/controllers/api/v1/advertisements_controller_spec.rb ./spec/controllers/api/v1/application_controller_spec.rb ./spec/controllers/api/v1/businesses_controller_spec.rb ./spec/controllers/api/v1/categories_controller_spec.rb ./spec/controllers/api/v1/current_users_controller_spec.rb ./spec/controllers/api/v1/group_memberships_controller_spec.rb ./spec/controllers/api/v1/groups_controller_spec.rb ./spec/controllers/api/v1/identifiers_controller_spec.rb ./spec/controllers/api/v1/images_controller_spec.rb ./spec/controllers/api/v1/location_employees_controller_spec.rb ./spec/controllers/api/v1/locations_controller_spec.rb ./spec/controllers/api/v1/notifications_controller_spec.rb ./spec/controllers/api/v1/redemptions_controller_spec.rb ./spec/controllers/api/v1/rewards_balance_entries_controller_spec.rb ./spec/controllers/api/v1/rewards_controller_spec.rb ./spec/controllers/api/v1/status_controller_spec.rb ./spec/controllers/api/v1/transactions_controller_spec.rb ./spec/controllers/api/v1/users_controller_spec.rb ./spec/controllers/application_controller_spec.rb ./spec/controllers/business_analytics_controller_spec.rb ./spec/controllers/business_transaction_analytics_controller_spec.rb ./spec/controllers/businesses_controller_spec.rb ./spec/controllers/categories_controller_spec.rb ./spec/controllers/client_applications_controller_spec.rb ./spec/controllers/contact_requests_controller_spec.rb ./spec/controllers/group_analytics_controller_spec.rb ./spec/controllers/group_invitations_controller_spec.rb ./spec/controllers/group_locations_controller_spec.rb ./spec/controllers/group_memberships_controller_spec.rb ./spec/controllers/groups_controller_spec.rb ./spec/controllers/identifiers_controller_spec.rb ./spec/controllers/location_transaction_analytics_controller_spec.rb ./spec/controllers/locations_analytics_controller_spec.rb ./spec/controllers/locations_controller_spec.rb ./spec/controllers/managed_businesses_controller_spec.rb ./spec/controllers/messages_controller_spec.rb ./spec/controllers/notifications_controller_spec.rb ./spec/controllers/offers_controller_spec.rb ./spec/controllers/pages_controller_spec.rb ./spec/controllers/redemptions_controller_spec.rb ./spec/controllers/rewards_balance_entries_controller_spec.rb ./spec/controllers/rewards_codes_controller_spec.rb ./spec/controllers/rewards_controller_spec.rb ./spec/controllers/sessions_controller_spec.rb ./spec/controllers/transactions_controller_spec.rb ./spec/controllers/twitter_accounts_controller_spec.rb ./spec/controllers/users_controller_spec.rb ./spec/helpers/advertisements_helper_spec.rb ./spec/helpers/application_helper_spec.rb ./spec/helpers/users_helper_spec.rb ./spec/jobs/export_job_spec.rb ./spec/jobs/geocode_visit_spec.rb ./spec/jobs/location_export_spec.rb ./spec/jobs/user_export_spec.rb ./spec/lib/blurbable_spec.rb ./spec/lib/date_string_validator_spec.rb ./spec/lib/employee_searchable_spec.rb ./spec/lib/geo_searchable_spec.rb ./spec/lib/geocoding_service_spec.rb ./spec/lib/identifier_spec.rb ./spec/lib/ip_validator_spec.rb ./spec/lib/lookup_concerns_spec.rb ./spec/lib/o_auth_authenticatable_spec.rb ./spec/lib/record_converter/converter_spec.rb ./spec/lib/record_converter/group_location_record_spec.rb ./spec/lib/record_converter/record_spec.rb ./spec/lib/record_converter/user_record_spec.rb ./spec/lib/to_comma_separated_spec.rb ./spec/lib/url_expert_spec.rb ./spec/lib/url_fields_spec.rb ./spec/lib/with_retry_spec.rb ./spec/lib/within_geo_scope_spec.rb ./spec/mailers/analytics_mailer_spec.rb ./spec/models/ability_spec.rb ./spec/models/access_token_request_spec.rb ./spec/models/access_token_spec.rb ./spec/models/address_component_spec.rb ./spec/models/address_spec.rb ./spec/models/advertisement_spec.rb ./spec/models/analytics/group_request_spec.rb ./spec/models/analytics/group_timeseries_builder_spec.rb ./spec/models/analytics/timeseries_builder_spec.rb ./spec/models/analytics/timeseries_request_spec.rb ./spec/models/analytics/transaction_request_spec.rb ./spec/models/analytics/transaction_timeseries_builder_spec.rb ./spec/models/analytics/visit_request_spec.rb ./spec/models/analytics/visit_timeseries_builder_spec.rb ./spec/models/api/v1/address_spec.rb ./spec/models/api/v1/advertisement_spec.rb ./spec/models/api/v1/announcement_spec.rb ./spec/models/api/v1/business_spec.rb ./spec/models/api/v1/category_search_spec.rb ./spec/models/api/v1/category_spec.rb ./spec/models/api/v1/group_spec.rb ./spec/models/api/v1/identifier_search_spec.rb ./spec/models/api/v1/image_spec.rb ./spec/models/api/v1/location_search_spec.rb ./spec/models/api/v1/location_spec.rb ./spec/models/api/v1/location_summary_spec.rb ./spec/models/api/v1/message_spec.rb ./spec/models/api/v1/notification_collection_spec.rb ./spec/models/api/v1/notification_spec.rb ./spec/models/api/v1/offer_spec.rb ./spec/models/api/v1/points_adjustment_spec.rb ./spec/models/api/v1/redemption_spec.rb ./spec/models/api/v1/reference_spec.rb ./spec/models/api/v1/reward_spec.rb ./spec/models/api/v1/rewards_balance_entry_spec.rb ./spec/models/api/v1/rewards_participant_spec.rb ./spec/models/api/v1/transaction_spec.rb ./spec/models/api/v1/user_spec.rb ./spec/models/business_employee_spec.rb ./spec/models/business_message_recipient_filter_spec.rb ./spec/models/business_search_spec.rb ./spec/models/business_spec.rb ./spec/models/category_search_spec.rb ./spec/models/category_spec.rb ./spec/models/client_application_spec.rb ./spec/models/contact_request_spec.rb ./spec/models/country_search_spec.rb ./spec/models/date_string_spec.rb ./spec/models/external_url_spec.rb ./spec/models/group_advertisement_spec.rb ./spec/models/group_location_spec.rb ./spec/models/group_membership_spec.rb ./spec/models/group_search_spec.rb ./spec/models/group_spec.rb ./spec/models/has_country_spec.rb ./spec/models/hour_range_spec.rb ./spec/models/image_spec.rb ./spec/models/json_resource_spec.rb ./spec/models/lazy_redemption_spec.rb ./spec/models/location_search_spec.rb ./spec/models/location_spec.rb ./spec/models/merchant_summary_spec.rb ./spec/models/message_spec.rb ./spec/models/messages/business_created_message_spec.rb ./spec/models/messages/group_advertisement_created_message_spec.rb ./spec/models/messages/group_advertisement_deleted_message_spec.rb ./spec/models/messages/group_advertisement_updated_message_spec.rb ./spec/models/messages/group_location_invitation_message_spec.rb ./spec/models/messages/group_membership_confirmed_message_spec.rb ./spec/models/messages/group_membership_rejected_message_spec.rb ./spec/models/messages/group_user_invitation_message_spec.rb ./spec/models/messages/mass_message_spec.rb ./spec/models/nonce_spec.rb ./spec/models/notification_spec.rb ./spec/models/o_auth_mac_token_request_spec.rb ./spec/models/offer_spec.rb ./spec/models/permissions_spec.rb ./spec/models/phone_number_spec.rb ./spec/models/point_calculator_spec.rb ./spec/models/points_adjustment_spec.rb ./spec/models/random_hash_spec.rb ./spec/models/redemption_condition_spec.rb ./spec/models/redemption_conditions/sufficient_rewards_points_spec.rb ./spec/models/redemption_conditions/user_redemption_count_spec.rb ./spec/models/redemption_conditions/user_redemption_rate_spec.rb ./spec/models/redemption_point_mediator_spec.rb ./spec/models/redemption_spec.rb ./spec/models/referenced_spec.rb ./spec/models/referencing_spec.rb ./spec/models/reserved_user_spec.rb ./spec/models/reward_spec.rb ./spec/models/rewards_balance_entry_spec.rb ./spec/models/role_spec.rb ./spec/models/system_message_specification_spec.rb ./spec/models/time_string_spec.rb ./spec/models/time_string_validator_spec.rb ./spec/models/time_throttled_message_specification_spec.rb ./spec/models/transaction_group_spec.rb ./spec/models/transaction_point_mediator_spec.rb ./spec/models/transaction_redemption_spec.rb ./spec/models/transaction_spec.rb ./spec/models/twitter_account_spec.rb ./spec/models/url_validator_spec.rb ./spec/models/user_authentication_search_spec.rb ./spec/models/user_role_spec.rb ./spec/models/user_search_spec.rb ./spec/models/user_spec.rb ./spec/models/visit_arbiter_spec.rb ./spec/models/visit_policies/anonymous_policy_spec.rb ./spec/models/visit_policies/authenticated_policy_spec.rb ./spec/models/visit_policies/nil_policy_spec.rb ./spec/models/visit_spec.rb ./spec/presenters/advertisements_presenter_spec.rb ./spec/presenters/current_user_presenter_spec.rb ./spec/presenters/merchant_summary_presenter_spec.rb ./spec/presenters/messages_presenter_spec.rb ./spec/presenters/rewards_presenter_spec.rb ./spec/uploaders/image_uploader_spec.rb

(You can see that the failing order is alphabetical. The passing order seems to be based on the file's mtime.)

Answer 4 · 2012-06-06T15:43:59.000Z

As for other matchers, we do have email_spec, and I believe devise adds some test matchers. Here's the Gemfile.lock:

GIT
  remote: git://github.com/fog/fog.git
  revision: 6d803bed756eeb42b0bf9084bbcc5d301fa04172
  specs:
    fog (1.3.1)
      builder
      excon (~> 0.13.0)
      formatador (~> 0.2.0)
      mime-types
      multi_json (~> 1.0)
      net-scp (~> 1.0.4)
      net-ssh (>= 2.1.3)
      nokogiri (~> 1.5.0)
      ruby-hmac

GEM
  remote: http://rubygems.org/
  specs:
    POpen4 (0.1.4)
      Platform (>= 0.4.0)
      open4
    Platform (0.4.0)
    actionmailer (3.2.5)
      actionpack (= 3.2.5)
      mail (~> 2.4.4)
    actionpack (3.2.5)
      activemodel (= 3.2.5)
      activesupport (= 3.2.5)
      builder (~> 3.0.0)
      erubis (~> 2.7.0)
      journey (~> 1.0.1)
      rack (~> 1.4.0)
      rack-cache (~> 1.2)
      rack-test (~> 0.6.1)
      sprockets (~> 2.1.3)
    activemodel (3.2.5)
      activesupport (= 3.2.5)
      builder (~> 3.0.0)
    activerecord (3.2.5)
      activemodel (= 3.2.5)
      activesupport (= 3.2.5)
      arel (~> 3.0.2)
      tzinfo (~> 0.3.29)
    activeresource (3.2.5)
      activemodel (= 3.2.5)
      activesupport (= 3.2.5)
    activesupport (3.2.5)
      i18n (~> 0.6)
      multi_json (~> 1.0)
    addressable (2.2.8)
    airbrake (3.0.9)
      activesupport
      builder
    archive-tar-minitar (0.5.2)
    arel (3.0.2)
    attr_required (0.0.5)
    aws-sdk (1.5.2)
      httparty (~> 0.7)
      json (~> 1.4)
      nokogiri (>= 1.4.4)
      uuidtools (~> 2.1)
    bcrypt-ruby (3.0.1)
    builder (3.0.0)
    cancan (1.6.4)
    capybara (1.1.2)
      mime-types (>= 1.16)
      nokogiri (>= 1.3.3)
      rack (>= 1.0.0)
      rack-test (>= 0.5.4)
      selenium-webdriver (~> 2.0)
      xpath (~> 0.1.4)
    carrierwave (0.6.2)
      activemodel (>= 3.2.0)
      activesupport (>= 3.2.0)
    childprocess (0.3.2)
      ffi (~> 1.0.6)
    chunky_png (1.2.5)
    columnize (0.3.6)
    compass (0.12.1)
      chunky_png (~> 1.2)
      fssm (>= 0.2.7)
      sass (~> 3.1)
    compass-rails (1.0.1)
      compass (~> 0.12.0)
    countries (0.8.2)
      currencies (= 0.4.0)
      currencies (>= 0.2.0)
    crack (0.3.1)
    cucumber (1.2.0)
      builder (>= 2.1.2)
      diff-lcs (>= 1.1.3)
      gherkin (~> 2.10.0)
      json (>= 1.4.6)
    cucumber-rails (1.3.0)
      capybara (>= 1.1.2)
      cucumber (>= 1.1.8)
      nokogiri (>= 1.5.0)
    currencies (0.4.0)
    daemons (1.1.8)
    database_cleaner (0.7.2)
    devise (2.1.0)
      bcrypt-ruby (~> 3.0)
      orm_adapter (~> 0.0.7)
      railties (~> 3.1)
      warden (~> 1.1.1)
    diff-lcs (1.1.3)
    email_spec (1.2.1)
      mail (~> 2.2)
      rspec (~> 2.0)
    encryptor (1.1.3)
    erubis (2.7.0)
    eventmachine (0.12.10)
    excon (0.13.4)
    factory_girl (1.3.3)
    faraday (0.8.1)
      multipart-post (~> 1.1)
    ffi (1.0.11)
    foreman (0.46.0)
      thor (>= 0.13.6)
    formatador (0.2.3)
    fssm (0.2.9)
    geocoder (1.1.2)
    gherkin (2.10.0)
      json (>= 1.4.6)
    haml (3.1.4)
    haml-rails (0.3.4)
      actionpack (~> 3.0)
      activesupport (~> 3.0)
      haml (~> 3.0)
      railties (~> 3.0)
    hashie (1.2.0)
    heroku (2.26.3)
      heroku-api (~> 0.2.1)
      launchy (>= 0.3.2)
      netrc (~> 0.7.2)
      rest-client (~> 1.6.1)
      rubyzip
    heroku-api (0.2.1)
      excon (~> 0.13.3)
    heroku_san (3.0.1)
      activesupport
      heroku (>= 2)
      heroku-api (>= 0.1.2)
      rake
    hike (1.2.1)
    httparty (0.8.3)
      multi_json (~> 1.0)
      multi_xml
    httpclient (2.2.5)
    i18n (0.6.0)
    journey (1.0.3)
    jquery-rails (2.0.2)
      railties (>= 3.2.0, < 5.0)
      thor (~> 0.14)
    json (1.7.3)
    kaminari (0.13.0)
      actionpack (>= 3.0.0)
      activesupport (>= 3.0.0)
      railties (>= 3.0.0)
    launchy (2.1.0)
      addressable (~> 2.2.6)
    libwebsocket (0.1.3)
      addressable
    linecache19 (0.5.12)
      ruby_core_source (>= 0.1.4)
    mail (2.4.4)
      i18n (>= 0.4.0)
      mime-types (~> 1.16)
      treetop (~> 1.4.8)
    mime-types (1.18)
    multi_json (1.3.6)
    multi_xml (0.5.1)
    multipart-post (1.1.5)
    net-scp (1.0.4)
      net-ssh (>= 1.99.1)
    net-ssh (2.5.2)
    netrc (0.7.2)
    newrelic_rpm (3.3.5)
    nokogiri (1.5.3)
    oauth (0.4.6)
    omniauth (1.1.0)
      hashie (~> 1.2)
      rack
    omniauth-oauth (1.0.1)
      oauth
      omniauth (~> 1.0)
    omniauth-twitter (0.0.11)
      multi_json (~> 1.3)
      omniauth-oauth (~> 1.0)
    open4 (1.3.0)
    orm_adapter (0.0.7)
    pg (0.13.2)
    phony (1.7.1)
    polyglot (0.3.3)
    rack (1.4.1)
    rack-cache (1.2)
      rack (>= 0.4)
    rack-oauth2 (0.14.4)
      activesupport (>= 2.3)
      attr_required (>= 0.0.5)
      httpclient (>= 2.2.0.2)
      i18n
      json (>= 1.4.3)
      rack (>= 1.1)
    rack-protection (1.2.0)
      rack
    rack-ssl (1.3.2)
      rack
    rack-test (0.6.1)
      rack (>= 1.0)
    rails (3.2.5)
      actionmailer (= 3.2.5)
      actionpack (= 3.2.5)
      activerecord (= 3.2.5)
      activeresource (= 3.2.5)
      activesupport (= 3.2.5)
      bundler (~> 1.0)
      railties (= 3.2.5)
    rails3-generators (0.17.4)
      railties (>= 3.0.0)
    railties (3.2.5)
      actionpack (= 3.2.5)
      activesupport (= 3.2.5)
      rack-ssl (~> 1.3.2)
      rake (>= 0.8.7)
      rdoc (~> 3.4)
      thor (>= 0.14.6, < 2.0)
    rake (0.9.2.2)
    rdoc (3.12)
      json (~> 1.4)
    redis (2.2.2)
    redis-namespace (1.0.3)
      redis (< 3.0.0)
    resque (1.20.0)
      multi_json (~> 1.0)
      redis-namespace (~> 1.0.2)
      sinatra (>= 0.9.2)
      vegas (~> 0.1.2)
    rest-client (1.6.7)
      mime-types (>= 1.16)
    rmagick (2.13.1)
    rspec (2.10.0)
      rspec-core (~> 2.10.0)
      rspec-expectations (~> 2.10.0)
      rspec-mocks (~> 2.10.0)
    rspec-core (2.10.1)
    rspec-expectations (2.10.0)
      diff-lcs (~> 1.1.3)
    rspec-mocks (2.10.1)
    rspec-rails (2.10.1)
      actionpack (>= 3.0)
      activesupport (>= 3.0)
      railties (>= 3.0)
      rspec (~> 2.10.0)
    ruby-debug-base19 (0.11.25)
      columnize (>= 0.3.1)
      linecache19 (>= 0.5.11)
      ruby_core_source (>= 0.1.4)
    ruby-debug19 (0.11.6)
      columnize (>= 0.3.1)
      linecache19 (>= 0.5.11)
      ruby-debug-base19 (>= 0.11.19)
    ruby-hmac (0.4.0)
    ruby_core_source (0.1.5)
      archive-tar-minitar (>= 0.5.2)
    ruby_flipper (0.0.3)
    rubyzip (0.9.8)
    sass (3.1.19)
    sass-rails (3.2.5)
      railties (~> 3.2.0)
      sass (>= 3.1.10)
      tilt (~> 1.3)
    selenium-webdriver (2.22.1)
      childprocess (>= 0.2.5)
      ffi (~> 1.0)
      libwebsocket (~> 0.1.3)
      multi_json (~> 1.0)
      rubyzip
    shoulda-matchers (1.1.0)
      activesupport (>= 3.0.0)
    simple_oauth (0.1.8)
    simplecov (0.6.4)
      multi_json (~> 1.0)
      simplecov-html (~> 0.5.3)
    simplecov-html (0.5.3)
    sinatra (1.3.2)
      rack (~> 1.3, >= 1.3.6)
      rack-protection (~> 1.2)
      tilt (~> 1.3, >= 1.3.3)
    sprockets (2.1.3)
      hike (~> 1.2)
      rack (~> 1.0)
      tilt (~> 1.1, != 1.3.0)
    state_machine (1.1.2)
    thin (1.3.1)
      daemons (>= 1.0.9)
      eventmachine (>= 0.12.6)
      rack (>= 1.0.0)
    thor (0.15.2)
    tilt (1.3.3)
    timecop (0.3.5)
    treetop (1.4.10)
      polyglot
      polyglot (>= 0.3.1)
    twitter (2.5.0)
      faraday (~> 0.8)
      multi_json (~> 1.3)
      simple_oauth (~> 0.1.6)
    tzinfo (0.3.33)
    uuidtools (2.1.2)
    vegas (0.1.11)
      rack (>= 1.0.0)
    warden (1.1.1)
      rack (>= 1.0)
    webmock (1.8.7)
      addressable (>= 2.2.7)
      crack (>= 0.1.7)
    xpath (0.1.4)
      nokogiri (~> 1.3)
    yui-compressor (0.9.6)
      POpen4 (>= 0.1.4)

PLATFORMS
  ruby

DEPENDENCIES
  airbrake
  aws-sdk
  bcrypt-ruby
  cancan (= 1.6.4)
  capybara
  carrierwave
  compass-rails
  countries
  cucumber-rails
  database_cleaner
  devise
  email_spec
  encryptor
  factory_girl (= 1.3.3)
  fog!
  foreman
  geocoder
  haml (= 3.1.4)
  haml-rails
  heroku_san
  jquery-rails
  kaminari (= 0.13.0)
  newrelic_rpm
  omniauth
  omniauth-twitter
  pg
  phony
  rack-oauth2
  rack-ssl
  rails (= 3.2.5)
  rails3-generators
  resque
  rmagick (= 2.13.1)
  rspec-rails (~> 2.10)
  ruby-debug19
  ruby_flipper
  sass-rails
  shoulda-matchers
  simplecov
  state_machine (= 1.1.2)
  thin
  timecop
  twitter
  webmock
  yui-compressor

Answer 5 · 2012-06-06T18:36:07.000Z

Well this is a puzzle.

What do you mean by "We do require relevant files from tests"? Do you require spec_helper in every test file?

Answer 6 · 2012-06-07T13:48:12.000Z

What do you mean by "We do require relevant files from tests"?

We require spec_helper if we don't need to load Rails for that test, or spec_helper_with_rails if we do. spec_helper_with_rails requires spec_helper and does everything it does, plus it loads Rails.

We do that so that we can run individual tests that don't need to load Rails without having to load it, which greatly speeds up the workflow and makes our specs fast.

We also require any specific files we're testing when Rails isn't loaded. For example, if we have a spec like:

describe Foo do
  # ...
end

then it probably starts with:

require 'spec_helper'
require 'lib/foo'

Do you require spec_helper in every test file?

Yes, or spec_helper_with_rails if Rails is required.

Answer 7 · 2012-06-07T22:48:52.000Z

It's odd that shoulda would even work without Rails being loaded, since this file requires bits of Rails before it makes the matchers available. Could that be what's happening here? If you do spec_helper_with_rails in every file, do the tests pass even with the failing order?

Answer 8 · 2012-06-07T22:50:00.000Z

Oh, and what version of shoulda, shoulda-matchers, and Rails? Can you paste the contents of spec_helper and spec_helper_with_rails?

I know that's a lot, but I'm really not sure what's going on here so I'd like to check everything.

Answer 9 · 2012-06-08T11:28:19.000Z

Just to be clear, we don't use shoulda-matchers in any files that don't have a require "spec_helper_with_rails". Rails is always required to use the validations, as you noted. We're using Rails 3.2.5, shoulda-matchers 1.1.0, and rspec-rails 2.10.1.

Here's the spec-helpers:

# spec/spec_helper.rb

require 'simplecov'

# Add project root as the default search path.
$:.unshift "#{File.dirname __FILE__}/.."

RSpec.configure do |config|
  config.mock_with :rspec
end

# Include support files.
Dir["#{File.dirname __FILE__}/support/base/**/*.rb"].each { |f| require f }

# spec/spec_helper_with_rails.rb
require 'spec_helper'

ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
require 'rspec/rails'
Dir["#{File.dirname __FILE__}/support/rails/**/*.rb"].each { |f| require f }

RSpec.configure do |config|
  config.use_transactional_fixtures = true
end

Answer 10 · 2012-06-08T17:00:24.000Z

Looking at the rspec-core issue, I tend to agree with justinko's - it's probably not a shoulda (or RSpec) issue.

I'm going to close this - I hope you find the issue! If it does turn out to be shoulda, please feel free to reopen.