Test flakiness is a part of technical debt that ruins your everyday work. It prevents new code from being shipped just because CI is red, and you have to go and restart the build. It creates frustration from the fact that your code may be broken, when in fact it’s not.
Having 50k tests and hundreds of developers makes the chance of introducing a flaky test even higher.
Some cases that I demonstrate are related to test order and some are not. What is the test order and how it’s related? The best practice is to run your tests in random order, to make sure that a test is not coupled with another test, and the order in which they run does not matter.
I will use MiniTest syntax in examples, but RSpec vs MiniTest doesn’t really matter here because all these issues are framework agnostic.
1. Random factories
# assuming the email field has unique constraint 10.times do Customer.create!(email: Faker::Internet.safe_email) end
Do you see anything suspicious here? In most of the times, it will pass. But sometimes Faker may return a random email that has already been used, and your test will crash with uniqueness constraint error.
The right way:
10.times do |n| Customer.create!(email: Faker::Internet.safe_email(n.to_s)) end
The argument tells Faker to return n-th email, instead of a random one.
2. Database records order
assert_equal([1, 2, 3], @products.pluck(:quantity))
While this test may usually pass, the SELECT query without ORDER instruction doesn’t guarantee consistent order of records. To avoid random failures, you should explicitly specify the order:
assert_equal([1, 2, 3], @products.pluck(:quantity).sort) # or assert_equal([1, 2, 3], @products.order(:quantity).pluck(:quantity))
3. Mutating the global environment
BulkEditor.register(User) do attributes(:email, :password) end assert_equal [:email, :password], BulkEditor.attributes_for(@user)
In my case,
BulkEditor used a global variable to store the registered models list.
As a result, after running the test the registry gets dirty.
This may affect other tests that will run after it (make them order dependent).
setup to BulkEditor.register(User) do attributes(:email, :password) end end teardown do BulkEditor.unregister(User) end
I have another real life example of mutating the state:
test "something" do SomeGem::VERSION = '9999.99.11' assert_not @provider.supported? end
Any test that will run after this one will get broken value of
It will also lead to a language-level warning:
warning: already initialized constant SomeGem::VERSION
test "something" do # only the block will get modified value of the constant stub_constant(SomeGem, :VERSION, '9999.99.99') do assert_not @provider.supported? end end
4. Time-sensitive tests
post = publish_delayed_post assert_equal 1.hour.from_now, post.published_at
Normally, the test would pass. But sometimes the post publishing will take a little longer than a millisecond,
published_at will take a little more than
There’s a special helper
assert_in_delta exactly for this case:
post = publish_delayed_post assert_in_delta 1.hour.from_now, post.published_at, 1.second
As an alternative, you can also freeze the time with libraries like Timecop.
5. Require-dependent tests
We had two kinds of test classes: one allowed remote HTTP calls and one not. Here is how it looked like:
# test/unit/remote_api_test.rb require 'remote_test_helper' class RemoteServiceTest < ActiveSupport::TestCase test "something" do # ... end end # test/unit/simple_test.rb require 'test_helper' class SimpleTest < ActiveSupport::TestCase test "something" do # ... end end
A number of tests used
remote_test_helper that allowed the test case to make external HTTP calls.
As you may guess, it perfectly worked when you run a single test. But when running all tests on CI, depending on the test order,
it could happen that every test that was executed after the remote one was allowed to make external calls 😱
You should always keep in mind that
require is global and it’s going to mutate the global state.
A better solution would be to use a macro that modifies only the context of specific test:
# test/unit/remote_api_test.rb require 'test_helper' class RemoteServiceTest < ActiveSupport::TestCase allow_remote_calls! test "something" do # ... end end # test/unit/simple_test.rb require 'test_helper' class SimpleTest < ActiveSupport::TestCase test "something" do # ... end end
Fixing a flaky tests is usually hard and it deserves a separate blog post, so I would suggest you to not even introduce one. If you’re intested, you can use one of links below to read more about flaky tests.