There is some overlap between what could go into the setup phase of a test or into a fixture. There are a few reason why certain pieces of setup code are better suited to a fixture and also why other code might be better off in the test itself. This post explores a few examples to help illustrate how you can decide where to put the setup code to minimise the maintenance cost and maximise the value of the test suit.
For clarity, the following explains what is meant by setup code in a test or in a fixture. The first example puts setup code in a test:
def test_can_connect():
db = DB(...)
db.connect()
The following example defines a fixture for the setup instead:
@pytest.fixture
def db():
return DB(...)
def test_can_connect(db):
db.connect()
To help us understand when it makes sense to move setup code to a fixture, we will look at the following examples:
- Setup code that is shared by multiple tests
- Setup code that needs to be cleaned up after the test finishes
- Setup code that is the required starting point for other tests
- Setup code specific to an individual test
Let’s start with the first one which is setup code that is shared by multiple tests. The benefits of using a fixture in this case is that you don’t have to repeat the same setup code over and over in test after test. Writing a fixture reduces the amount of code that needs to be written and maintained. One of the drawbacks is that the test is no longer self contained. To understand it, developers have to not only read the test code, they also have to read and understand the fixture. This complexity can be managed using test documentation which is discussed in this post: Writing Great Test Documentation. The increased complexity means that it isn’t always worth putting setup code that is repeated on more than one test into a fixture. An example is where setup code is only shared by two tests. In that case, it might be better to leave the setup code in those tests to make them easier to understand. An obvious example where it is worth moving code to a fixture is if the same setup code is repeated on many tests. Any code changes that result in changes to the setup will require editing all of those tests. With a shared fixture, the change to the setup code for all the tests will just need to be made in one place. For example, the following tests all setup their own database connection:
def test_insert_record():
db = DB(...)
connection = db.connect()
record = Record(...)
connection.add(record)
assert record in connection.records
def test_update_record():
db = DB(...)
connection = db.connect()
record = Record(...)
connection.add(record)
record.key = "value"
connection.update(record)
assert connection.records[0].key = "value"
This can instead be done using a fixture:
@pytest.fixture
def connection():
db = DB(...)
return db.connect()
def test_insert_record(connection):
record = Record(...)
connection.add(record)
assert record in connection.records
def test_update_record():
record = Record(...)
connection.add(record)
record.key = "value"
connection.update(record)
assert connection.records[0].key = "value"
Moving on to the next case which is setup code that needs to be cleaned up after a test finishes. The cleanup can be things like removing data from a database, cleaning up a file system or reverting application configuration back to the default. This cleanup is usually done to be able to re-use a test environment that is costly to setup. The following discussion is unique to pytest and any other test frameworks that behave in the following way. pytest will stop running a test after it encounters the first assertion error. This means that any cleanup code in the test after the assertion will not get executed if the assert fails and the test environment will not be cleaned up. This will mean the following tests that rely on a clean test environment may fail even though there isn’t a bug in the code they are testing. In contrast to a test, the cleanup portion of a fixture will get executed even if any of the tests that use it fail. For this reason, I can’t think of any examples where setup code that must be cleaned up should be included in a test rather than moved to a fixture. Even if that requires writing a fixture that is only used by a single test.
For example, the database might need to be cleaned up after a given test. If it is done in the actual test:
def test_insert_record():
db = DB(...)
connection = db.connect()
record = Record(...)
connection.add(record)
assert record in connection.records
connection.cleanup()
Then if the assert fails, the database will still contain the record. This can be avoided by putting the cleanup step into a fixture:
@pytest.fixture
def connection():
db = DB(...)
connection = db.connect()
yield connection
connection.cleanup()
def test_insert_record(connection):
record = Record(...)
connection.add(record)
assert record in connection.records
The next example, setup code that is the required starting point for other tests, warrants more of an explanation. This essentially captures any tests that rely on test execution order. In this case, the tests don’t necessarily share setup code. Instead, the setup done in one test is required by another test. Without the first test getting executed, the second test will not work. Leaving this setup code in a test has several drawbacks:
- From reading the second test, it won’t be clear what is required before the test can be executed. The only indication is that it is listed after the first test. Any future developers will have a hard time figuring out the dependency graph of the tests without experimentation.
- The second test cannot be executed on its own. It is often useful to only run a specific test, especially in the case of integration tests which might take a little time to be executed. This won’t be possible if the first test needs to be executed as well.
- If the first test is deleted or changed, the second test may fail even though there isn’t a bug with the code it is testing. This will be difficult to debug, especially if a given tests depends on more than just one test.
A much clearer and easier to maintain approach is to put the setup into a fixture and have all the tests that require the setup depend on that fixture. This makes it clear what is required for the test to be executed and means that changing one test won’t break another. In the following example the second test depends on the first test getting executed:
@pytest.fixture
def connection():
db = DB(...)
return db.connect()
def test_insert_record(connection):
record = Record(...)
connection.add(record)
assert record in connection.records
def test_update_record():
record = connection.records[0]
record.key = "value"
connection.update(record)
assert connection.records[0].key = "value"
It is better to put the record creation in a fixture:
@pytest.fixture
def connection():
db = DB(...)
return db.connect()
@pytest.fixture
def record(connection):
record = Record(...)
connection.add(record)
return record
def test_insert_record(connection, record):
assert record in connection.records
def test_update_record(connection, record):
record.key = "value"
connection.update(record)
assert connection.records[0].key = "value"
Finally, let’s discuss the case where the setup of a test is unique to a single test. In this case it is generally better not to put the setup code into a fixture to reduce the complexity of the test. Examples where it could still be worth putting the setup into a fixture is if the setup is complex or the setup code distracts from rather than adds to the test. An example is setting up a database connection. This might be complex and it isn’t necessary to understand how the database connection is setup if a test just checks that a function adds a record to a table.
In summary, there are many examples where it is better to put setup code into a fixture. It is usually better to do so if setup code is shared among tests, if the setup code needs to be cleaned up and to avoid introducing dependencies between tests. If setup code is only used by a single test, for clarity is is usually better to leave the setup in the test unless it is complex or distracts from the primary purpose of the test.
David Andersson 