A rule of TDD is writing tests first. In most real world projects, the
software usually contains GUI, web, database and dependencies on
external systems. Usually they are not as simple as examples in junit
cookbook that can be test driven out by just doing unit testing. There
are different kinds of tests in such kind of projects such as
functional tests, unit tests, acceptance tests etc. Different projects
have different styles of implementing TDD. For example, testers may
take the responsibility of writing acceptance tests and developers are
only writing unit tests. I would like to advocate a top-down TDD style
starting from user stories and ending at production code for TDDers.
TDD from starting from user stories
What
Before we start talking, I would like to achieve a shared
understanding of various types of tests with you. We probably have
heard about unit test, functional test, acceptance test, module test,
integration test etc. By unit test we mean tests that only test one
class by mocking out every class it couples with. By functional test
we mean tests that test if groups of clusters of classes meet external
requirements and achieve goals. Module test and integration test are a
kind of functional test that test logically divided module and the
integration between modules, and we just treat acceptance test as one
of kinds of functional tests which tries to cover a more complete
scenario.
Based on our common understanding of these typical kinds of tests in
our application, “top-down approach” says developers should write
acceptance criteria for the user story first, run it and see the
failure, then start to write unit tests, see the unit test fail and
fix them. After finishing an amount of unit tests, go back to run the
acceptance again, see it pass and then we can say this acceptance
criteria has been fulfilled and we can move to next acceptance
criteria and repeat this process.
There are several points in this process:
- There may be several more levels of functional tests(such as module
tests, integration tests) you need to write between acceptance
tests and unit tests depending on how do you organize your tests in
your project.
- Always keep it in mind that doing simplest things to make your
tests pass. Apply this rule when you try to pass the unit tests and
functional tests.
- A prerequisite of this approach is a good user story. A good story
is a INVEST story which is small and testable. If you don’t have
clear acceptance criteria or the story is not testable , you may
find it’s difficult to write acceptance tests first. If your story
is too big, you may find there are too many acceptance tests you
have to make them pass and the story will take you very long time.
Why
This is not the only approach we have. As I said at the beginning,
different projects and different people have their favorites. Here are
some reasons why I like to use this approach.
TDD zealots like me want to test everything in the software and make
everything automatic. We all know the rule “no tests, no code”,
because tests represent the requirements. We think by expressing those
requirements using tests first will make sure that we have understand
the requirements before we start to write production code to meet
those requirements. Tests also provide quality assurance for our
production code, serving the documentation and ground for refactoring
etc.
Things become a little complex and interesting when a story has
different kinds of or levels of tests. When you are writing unit
tests, you may be already clear about the functionality and interfaces
of the class you want to test. But how does the need of this class
come? How do you decide we need this class and we will design like
this? The answer is you decide to create these classes to meet the
functionality described in the story, you are writing unit tests
according to your understanding of the functionality described in the
story which usually comes by your talking with business analysts.
At this time we can say without the requirements of the
story(functionality described in the story and usually represented as
acceptance criteria), we are not clear how will our code look like and
what unit tests to write. Since you write unit tests to test a unit
piece of functionality(usually an public interface of an class) which
is again determined by the functionality represented by the story,
since you have to understand functionality first to write unit tests,
why not first write functional tests to represent the requirement, see
it fail then find out the solution and write the corresponding unit
tests.
So starting from writing acceptance tests will make sure you have
understood the requirements of the story which will help you to choose
a simple solution to fulfill the requirement represented by the
acceptance tests, then write unit tests to based on your clear
understanding of each classes’ functionality in the solution and use
simple code to make them pass. This is a very natural TDD approach.
How?
As we said above, the first step is to turn acceptance criteria into
tests, so one of the most important factors for us is to find tools
that support us to achieve that goal. Functional testing tools are
actually a very important part in our process. Given that there are
already many unit testing tools available for nearly every programming
language, a suitable functional testing tool seems more important for
us.
While unit testing tools can be categorized by programming language,
we can examine those functional testing tools by domains of
problems. Due to differences of technical implementation nature of web
application, windows desktop application and java GUI application
etc., there are different functional testing tools for them.
For web applications, functional testing tools such as Selenium,
Watir(Watin, Watij), Sahi are available. We have Abbot for java GUI
applications, NUnitForm for Windows Form applications and Microsoft
UIAutomation Framework for a wide range of windows application. More
heavy tools like QTP could also be a choice. For other kinds of
software such as library, console application we can either use xUnit
framework to do functional testing or we can use varies of other
approaches to achieve such purposes to some extent.
It seems that functional testing tools for some kinds of applications
are more abundant and mature than those for other kinds, which seems
make implementing the practice in this article easier in some
applications than in others. It is probably true, but just remember
the rationale behind this approach and once you understand and agree
with it and try to apply the rules behind to the greatest possibility,
you can always find other variations and substitutes in practice.
By continuously refactoring your testing code, you can actually make
your tests easier to understand and more business natural(DSL). Some
tools like rbehave and jbehave are already able to allow us to write
our acceptance tests in a way more clearly reflecting business value
and easier to understand by non-technical people. They help developers
to think about the problem and their code from a perspective which
focus more on business value of a story.
Pros and Cons
Pros
- Writing acceptance tests first makes developers think more about
value before starting to code
It’s really hard to write acceptance tests first without clearly
understand what are you going to test. Strictly applying the rule of
“writing tests first” forces people to think more about business
requirement before they start to write any code. This approach
encourages developers to understand the business first by either
talking to business analysts or customers directly, by this way the
misunderstanding of requirement can be reduced to minimum extent and
the discrepancy between business people and development team can be
decreased.
- Test automation is easily achievable from the beginning of the
project
TDD starting from acceptance tests implies automation of all tests in
the project by itself, because it’s impossible for people to get fast
feedback without the ability to automatically build the project and
run tests. With the support of continuous integration tools such as
CruiseControl, regression testing is also easily achievable.
- Clear and simple design, testable code
As mentioned before, we should clearly understand the business value
of the story and the acceptance criteria and then turn them into
tests. The rest of our work is just to use simplest code to make those
tests pass. If writing unit tests first can help us achieve simple
design of each public interface of a class, writing functional tests
first can also help us choose simplest design to implement the
business value.
One byproduct of TDD is a good set of tests which can serve as
documentation of the project. Having a high quality collection of
functional tests can help everyone understand the business requirement
and the functionality of the application by just going through the
functional tests, Meanwhile, unit tests are documents for people to
understand the implementation of each class.
There are tools such as Testdox helping us translate our tests which
can only be understood by technical people into a format more friendly
to non-technical people. Testing frameworks like rspec already have
built-in support of this kind of translation, and other frameworks
like rbehave and jbehave make the tests themselves represented by
business natural language. All of those efforts can make our tests
more representative of business value, more self-explaining and more
easy to understand.
- Delay the implementing and design decision
Given the existence of the tests, developers can make implementation
and design decisions at the last step, after tests being written.
- nurture the good habits of programming
Developers should never only focus on coding and technical
problems. Good developers should have a sense of considering
everything to make what they produced really valuable and with high
quality. Agile software development methodology requires developers to
be more versatile. Besides having capability of generating high
quality code, developers should understand the value of what they are
going to do and be able to question the requirement if they are not so
valuable. Thinking about our code from the perspective of the value
they produced helps us nurture a good habit of simple design. TDD from
stories can help us to achieve this.
- Testing team more effective
With functional tests written by developers and automated, testers can
be more effective. Tester can spend most of their time on other
testing such as performance testing, exploratory testing and manual
testing etc. They can focus their energy on tests that can not
achieved by automated tests. They can spend more time on acceptance
criteria and providing their testing ideas rather than actually write
tests code.
One practical problem is that some testers can not generate very good
tests code. This problem become more serious when the tools need more
programming thinking and the test suite become huge while the project
grows bigger. The problem disappears if developers write most
functional tests.
In agile planning, we usually have to estimate how much work a story
will take. Developers writing functional tests first can make them
more confident and easier to say that a story has been finished if all
tests passed, which helps them estimate the work of each story more
accurately. That will make the release planning easier.
- Not mention fast feedback, iterative release
Cons
While we are talking about many benefits of this TDD style, we should
realize that sometimes it is difficult. For people who have been used
to it, it is very natural and happy to do this, but it might be
difficult to be accepted by other people. For most people, TDD itself
is a mind shift. They might have been used to writing code first and
testing and fixing the problems. The common questions from them are
such that how can I write tests without the code that I am going to
test. Doing TDD and using this approach need people to understand the
problem first, then you can write tests to represent the
requirements. It’s not easy at the beginning, but it’s a good practice
and worth spending time on it. We should make ourselves disciplined
and think about the software from value perspective. Pairing with some
experienced people can also help us quickly be accustomed to this
approach.
TDD from stories need some prerequisites. Firstly we need good
stories, good stories are independent, valuable, estimable, small and
testable stories. Good stories make developers easy to understand the
business value and raise good questions. They are small and testable
so that we can finish them with writing tests first in reasonable
time. Secondly the acceptance criteria of each story should be clear,
so that developers can easily turn those into tests. A good story is
the corner stone of a successful project.
- Difficult in some areas like cpp, game, restricted by the
availability of tool set
We have said that one important factor for this approach is having a
good functional testing tool, which is not always the case. In
practice sometimes it’s very difficult to find a good functional
testing tool for your application because of the programming language
you use, the target platform of your application or the framework your
application depending on. For example, there are not very handy tools
for testing windows desktop application. QTP is a good tool, but it’s
too heavy for TDD. Another example is game. It is also very difficult
to implement TDD in some languages such as C, Cpp.
Tools are evolving. Several years ago it’s also very difficult to do
functional testing for web application, but nowadays there are a bunch
of tools for web application functional testing. The requirement will
drive people to make better tools.
- Build will get slow if not being careful because functional tests
number increases
By definition, functional tests test the real functionality of an
software. Usually we need to hit the databases, transfer data through
the network, writing and reading files etc. in our functional
tests. One common problem is while a project becomes bigger, the time
spent on running all those functional tests will be very long. That
will make us unable to get fast feedback from functional tests and
decrease the productivity of the team. It will make developers
unwilling to run those tests and write functional tests.
Sometimes we have to face this reality, but in most situations we can
find many solutions to solve this problem or at least we can bypass
it. For example, we can write and run functional tests belong to the
story that we are working on during development, and then we run all
tests before checking in our code. In some cases we may divide tests
into smaller groups and run them parallel. We may also be able to
reorganize our tests and use stubs and mocks to test our application
without talking to some external systems.
A sample
Let’s use some sample code to demonstrate the whole idea and
process. Suppose that we are working on a web application, we have a
story about login functionality, which is:
As a user, I want to login to the website, so that I can use
registered user specific functionality.
For this story, we have those acceptance criteria:
* Happy path: successfully login
Given: user go to the login page
When: user input correct username and password and submit
Then: user logged in with a successful message
* Username or password missing
Given: user goto the login page
When: user does not input username or password and submit
Then: user should see an error message with “username or password missing”
* Username or password incorrect
Given: user goto the login page
When: user input wrong username or password and submit
Then: user should see an error message with “username or password is not correct”
Suppose we use Ruby on Rails to make this web application, with the
help of selenium, we can easily write acceptance tests for this story:
<src lang="ruby">
Story "Login", %(
As a user,
I want to sign in the website,
So that I can use registered user specific functionality) do
@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*iexplore", "http://localhost", 10000);
Scenario "user successfully login" do
Given "correct username and password" do
@selenium.start
@selenium.open "http://localhost/users/login"
@selenium.type "username", "someone"
@selenium.type "password", "password"
end
When "login" do
@selenium.click "submit"
end
Then "user logged in successfully" do
@selenium.is_text_present "Welcome, someone!"
@selenium.stop
end
end
other scenarios...
end
</src>
Here we use rbehave, a framework for expressing business acceptance
criteria using ruby code. After we got this tests, we can run them and
see the failure. Now we can think about how to make this test pass. At
this point, we probably need to create a UsersController and an action
called login, and have User model. Once we made that decision, we can
start to write tests for them:
<src lang="ruby">
def test_registered_user_should_able_to_sign_in_and_redirect_to_home_as_default
post :login, :name => "someone", :password => "password"
assert_equal "Welcome, someone!", flash[:notice]
end
</src>
Here we test that the login action of Users control should perform the
login and then set a flash notice, then we will come up with a
controller action:
<src lang="ruby">
def signin
user = User.authenticate(params[:name], params[:password])
flash[:notice] = "Welcome, " + user.nickname + "!"
redirect_to :controller => "home"
end
</src>
Obviously we delegate the authentication to the User model, so we
write the tests for it:
<src lang="ruby">
def should_be_able_to_get_authenticated_user
assert_equal users(:someone), User.authenticate("some user", "user's password")
...
end
</src>
After this we can easily write some simple code to implement this
authenticate method. We don’t have to worry about too much exception
handling because we are currently just focusing on the happy
path. After we write more tests, we will make our code more robust
gradually.
So far I have presented the process of implementing a successful
scenario of a simple story. Notice that we solved a problem by
dividing them into many vertical slices(here are different scenarios),
every time we finish one slice, write the tests first and write the
simplest code to make them pass. In this way we are very confident at
each step, and we can clearly see the value we are creating.
Variations
Sometimes because of lack of testing tools, we can not write full
scenario functional tests for the application. Under this situation we
still want to achieve this goal to the greatest extent, so we write
functional tests for modules instead of whole application(end to
end). For example, if we can not find a very handy functional test
tool for Windows application, we may organize our code to some pattern
like passive view, so that without performing actions through the
view, we still can write functional tests for our application (maybe
through testing the controller). Another example is in a C/S rich
client application, we can test the server module’s functionality to
guarantee that the server works as we expected.
- Dev unit test -> add functional tests
- Extract writing functional tests from dev to tester
It is very attractive for QAs to write functional tests and for
Developers to write unit tests and let them work parallel. It seems
the productivity of the whole team will be greatest. Once the
acceptance criteria has been clarified, dev can write unit tests, and
tester may write functional tests. Ideally it’s true, but it’s
difficult practically. The following are some potential problems
related to this approach:
- Sometimes dev and testers can work parallel, which may increase
productivity(but again it’s dubious, because this may need more
communication and iteration between testers and devs).
- Depending on the tools the QA uses, sometimes it requires QA to
have more advanced programming skill to write and maintain
functional test suite. If QA does not have that level of skill,
it’s difficult for QAs to catch up with devs to write functional
tests for new developed functionalities. If developers deliver the
stories very fast, QAs’ job will be more difficult.
If these problems happen, there are some ways to mitigate it. One way
is to let developer write and maintain some (not all) functional tests
(happy path tests, for example), integrate them into developers’ build
and make sure them pass before developers’ commit.
