A rule of TDD is writing tests first. In most real world projects, the software usually contains GUI, web, database and dependencies on external systems. Usually they are not as simple as examples in junit cookbook that can be test driven out by just doing unit testing. There are different kinds of tests in such kind of projects such as functional tests, unit tests, acceptance tests etc. Different projects have different styles of implementing TDD. For example, testers may take the responsibility of writing acceptance tests and developers are only writing unit tests. I would like to advocate a top-down TDD style starting from user stories and ending at production code for TDDers.
TDD from starting from user stories
What?
Before we start talking, I would like to achieve a shared understanding of various types of tests with you. We probably have heard about unit test, functional test, acceptance test, module test, integration test etc. By unit test we mean tests that only test one class by mocking out every class it couples with. By functional test we mean tests that test if groups of clusters of classes meet external requirements and achieve goals. Module test and integration test are a kind of functional test that test logically divided module and the integration between modules, and we just treat acceptance test as one of kinds of functional tests which tries to cover a more complete scenario.
Based on our common understanding of these typical kinds of tests in our application, “top-down approach” says developers should write acceptance criteria for the user story first, run it and see the failure, then start to write unit tests, see the unit test fail and fix them. After finishing an amount of unit tests, go back to run the acceptance again, see it pass and then we can say this acceptance criteria has been fulfilled and we can move to next acceptance criteria and repeat this process.
There are several points in this process:
* There may be several more levels of functional tests(such as module tests, integration tests) you need to write between acceptance tests and unit tests depending on how do you organize your tests in your project.
* Always keep it in mind that doing simplest things to make your tests pass. Apply this rule when you try to pass the unit tests and functional tests.
* A prerequisite of this approach is a good user story. A good story is a INVEST story which is small and testable. If you don’t have clear acceptance criteria or the story is not testable , you may find it’s difficult to write acceptance tests first. If your story is too big, you may find there are too many acceptance tests you have to make them pass and the story will take you very long time.
Why?
This is not the only approach we have. As I said at the beginning, different projects and different people have their favorites. Here are some reasons why I like to use this approach.
TDD zealots like me want to test everything in the software and make everything automatic. We all know the rule “no tests, no code”, because tests represent the requirements. We think by expressing those requirements using tests first will make sure that we have understand the requirements before we start to write production code to meet those requirements. Tests also provide quality assurance for our production code, serving the documentation and ground for refactoring etc.
Things become a little complex and interesting when a story has different kinds of or levels of tests. When you are writing unit tests, you may be already clear about the functionality and interfaces of the class you want to test. But how does the need of this class come? How do you decide we need this class and we will design like this? The answer is you decide to create these classes to meet the functionality described in the story, you are writing unit tests according to your understanding of the functionality described in the story which usually comes by your talking with business analysts.
At this time we can say without the requirements of the story(functionality described in the story and usually represented as acceptance criteria), we are not clear how will our code look like and what unit tests to write. Since you write unit tests to test a unit piece of functionality(usually an public interface of an class) which is again determined by the functionality represented by the story, since you have to understand functionality first to write unit tests, why not first write functional tests to represent the requirement, see it fail then find out the solution and write the corresponding unit tests.
So starting from writing acceptance tests will make sure you have understood the requirements of the story which will help you to choose a simple solution to fulfill the requirement represented by the acceptance tests, then write unit tests to based on your clear understanding of each classes’ functionality in the solution and use simple code to make them pass. This is a very natural TDD approach.
How?
As we said above, the first step is to turn acceptance criteria into tests, so one of the most important factors for us is to find tools that support us to achieve that goal. Functional testing tools are actually a very important part in our process. Given that there are already many unit testing tools available for nearly every programming language, a suitable functional testing tool seems more important for us.
While unit testing tools can be categorized by programming language, we can examine those functional testing tools by domains of problems. Due to differences of technical implementation nature of web application, windows desktop application and java GUI application etc., there are different functional testing tools for them.
For web applications, functional testing tools such as Selenium, Watir(Watin, Watij), Sahi are available. We have Abbot for java GUI applications, NUnitForm for Windows Form applications and Microsoft UIAutomation Framework for a wide range of windows application. More heavy tools like QTP could also be a choice. For other kinds of software such as library, console application we can either use xUnit framework to do functional testing or we can use varies of other approaches to achieve such purposes to some extent.
It seems that functional testing tools for some kinds of applications are more abundant and mature than those for other kinds, which seems make implementing the practice in this article easier in some applications than in others. It is probably true, but just remember the rationale behind this approach and once you understand and agree with it and try to apply the rules behind to the greatest possibility, you can always find other variations and substitutes in practice.
By continuously refactoring your testing code, you can actually make your tests easier to understand and more business natural(DSL). Some tools like rbehave and jbehave are already able to allow us to write our acceptance tests in a way more clearly reflecting business value and easier to understand by non-technical people. They help developers to think about the problem and their code from a perspective which focus more on business value of a story.
pros and cons
pros:
* writing acceptance tests first makes developers think more about value before starting to code
It’s really hard to write acceptance tests first without clearly understand what are you going to test. Strictly applying the rule of “writing tests first” forces people to think more about business requirement before they start to write any code. This approach encourages developers to understand the business first by either talking to business analysts or customers directly, by this way the misunderstanding of requirement can be reduced to minimum extent and the discrepancy between business people and development team can be decreased.
* test automation is easily achievable from the beginning of the project
TDD starting from acceptance tests implies automation of all tests in the project by itself, because it’s impossible for people to get fast feedback without the ability to automatically build the project and run tests. With the support of continuous integration tools such as CruiseControl, regression testing is also easily achievable.
* clear and simple design, testable code
As mentioned before, we should clearly understand the business value of the story and the acceptance criteria and then turn them into tests. The rest of our work is just to use simplest code to make those tests pass. If writing unit tests first can help us achieve simple design of each public interface of a class, writing functional tests first can also help us choose simplest design to implement the business value.
* tests as documents
One byproduct of TDD is a good set of tests which can serve as documentation of the project. Having a high quality collection of functional tests can help everyone understand the business requirement and the functionality of the application by just going through the functional tests, Meanwhile, unit tests are documents for people to understand the implementation of each class.
There are tools such as Testdox helping us translate our tests which can only be understood by technical people into a format more friendly to non-technical people. Testing frameworks like rspec already have built-in support of this kind of translation, and other frameworks like rbehave and jbehave make the tests themselves represented by business natural language. All of those efforts can make our tests more representative of business value, more self-explaining and more easy to understand.
* delay the implementing and design decision
Given the existence of the tests, developers can make implementation and design decisions at the last step, after tests being written.
* nurture the good habits of programming
Developers should never only focus on coding and technical problems. Good developers should have a sense of considering everything to make what they produced really valuable and with high quality. Agile software development methodology requires developers to be more versatile. Besides having capability of generating high quality code, developers should understand the value of what they are going to do and be able to question the requirement if they are not so valuable. Thinking about our code from the perspective of the value they produced helps us nurture a good habit of simple design. TDD from stories can help us to achieve this.
* testing team more effective
With functional tests written by developers and automated, testers can be more effective. Tester can spend most of their time on other testing such as performance testing, exploratory testing and manual testing etc. They can focus their energy on tests that can not achieved by automated tests. They can spend more time on acceptance criteria and providing their testing ideas rather than actually write tests code.
One practical problem is that some testers can not generate very good tests code. This problem become more serious when the tools need more programming thinking and the test suite become huge while the project grows bigger. The problem disappears if developers write most functional tests.
In agile planning, we usually have to estimate how much work a story will take. Developers writing functional tests first can make them more confident and easier to say that a story has been finished if all tests passed, which helps them estimate the work of each story more accurately. That will make the release planning easier.
* not mention fast feedback, iterative release
cons:
* sometimes it’s not easy
While we are talking about many benefits of this TDD style, we should realize that sometimes it is difficult. For people who have been used to it, it is very natural and happy to do this, but it might be difficult to be accepted by other people. For most people, TDD itself is a mind shift. They might have been used to writing code first and testing and fixing the problems. The common questions from them are such that how can I write tests without the code that I am going to test. Doing TDD and using this approach need people to understand the problem first, then you can write tests to represent the requirements. It’s not easy at the beginning, but it’s a good practice and worth spending time on it. We should make ourselves disciplined and think about the software from value perspective. Pairing with some experienced people can also help us quickly be accustomed to this approach.
TDD from stories need some prerequisites. Firstly we need good stories, good stories are independent, valuable, estimable, small and testable stories. Good stories make developers easy to understand the business value and raise good questions. They are small and testable so that we can finish them with writing tests first in reasonable time. Secondly the acceptance criteria of each story should be clear, so that developers can easily turn those into tests. A good story is the corner stone of a successful project.
* difficult in some areas like cpp, game, restricted by the availability of tool set
We have said that one important factor for this approach is having a good functional testing tool, which is not always the case. In practice sometimes it’s very difficult to find a good functional testing tool for your application because of the programming language you use, the target platform of your application or the framework your application depending on. For example, there are not very handy tools for testing windows desktop application. QTP is a good tool, but it’s too heavy for TDD. Another example is game. It is also very difficult to implement TDD in some languages such as C, Cpp.
Tools are evolving. Several years ago it’s also very difficult to do functional testing for web application, but nowadays there are a bunch of tools for web application functional testing. The requirement will drive people to make better tools.
* build will get slow if not being careful because functional tests number increases
By definition, functional tests test the real functionality of an software. Usually we need to hit the databases, transfer data through the network, writing and reading files etc. in our functional tests. One common problem is while a project becomes bigger, the time spent on running all those functional tests will be very long. That will make us unable to get fast feedback from functional tests and decrease the productivity of the team. It will make developers unwilling to run those tests and write functional tests.
Sometimes we have to face this reality, but in most situations we can find many solutions to solve this problem or at least we can bypass it. For example, we can write and run functional tests belong to the story that we are working on during development, and then we run all tests before checking in our code. In some cases we may divide tests into smaller groups and run them parallel. We may also be able to reorganize our tests and use stubs and mocks to test our application without talking to some external systems.
a sample
Let’s use some sample code to demonstrate the whole idea and process. Suppose that we are working on a web application, we have a story about login functionality, which is:
As a user, I want to login to the website, so that I can use registered user specific functionality.
For this story, we have those acceptance criteria:
* Happy path: successfully login
Given: user go to the login page
When: user input correct username and password and submit
Then: user logged in with a successful message
* username or password missing
Given: user goto the login page
When: user does not input username or password and submit
Then: user should see an error message with “username or password missing”
* username or password incorrect
Given: user goto the login page
When: user input wrong username or password and submit
Then: user should see an error message with “username or password is not correct”
Suppose we use Ruby on Rails to make this web application, with the help of selenium, we can easily write acceptance tests for this story:
Story "Login", %(
As a user,
I want to sign in the website,
So that I can use registered user specific functionality) do
@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*iexplore", "http://localhost", 10000);
Scenario "user successfully login" do
Given "correct username and password" do
@selenium.start
@selenium.open "http://localhost/users/login"
@selenium.type "username", "someone"
@selenium.type "password", "password"
end
When "login" do
@selenium.click "submit"
end
Then "user logged in successfully" do
@selenium.is_text_present "Welcome, someone!"
@selenium.stop
end
end
other scenarios...
endHere we use rbehave, a framework for expressing business acceptance criteria using ruby code. After we got this tests, we can run them and see the failure. Now we can think about how to make this test pass. At this point, we probably need to create a UsersController and an action called login, and have User model. Once we made that decision, we can start to write tests for them:
def test_registered_user_should_able_to_sign_in_and_redirect_to_home_as_default
post :login, :name => "someone", :password => "password"
assert_equal "Welcome, someone!", flash[:notice]
endHere we test that the login action of Users control should perform the login and then set a flash notice, then we will come up with a controller action:
def signin
user = User.authenticate(params[:name], params[:password])
flash[:notice] = "Welcome, " + user.nickname + "!"
redirect_to :controller => "home"
endObviously we delegate the authentication to the User model, so we write the tests for it:
def should_be_able_to_get_authenticated_user
assert_equal users(:someone), User.authenticate("some user", "user's password")
...
endAfter this we can easily write some simple code to implement this authenticate method. We don’t have to worry about too much exception handling because we are currently just focusing on the happy path. After we write more tests, we will make our code more robust gradually.
So far I have presented the process of implementing a successful scenario of a simple story. Notice that we solved a problem by dividing them into many vertical slices(here are different scenarios), every time we finish one slice, write the tests first and write the simplest code to make them pass. In this way we are very confident at each step, and we can clearly see the value we are creating.
variations
* lack of tools support
Sometimes because of lack of testing tools, we can not write full scenario functional tests for the application. Under this situation we still want to achieve this goal to the greatest extent, so we write functional tests for modules instead of whole application(end to end). For example, if we can not find a very handy functional test tool for Windows application, we may organize our code to some pattern like passive view, so that without performing actions through the view, we still can write functional tests for our application (maybe through testing the controller). Another example is in a C/S rich client application, we can test the server module’s functionality to guarantee that the server works as we expected.
* dev unit test -> add functional tests
* extract writing functional tests from dev to tester
It is very attractive for QAs to write functional tests and for Developers to write unit tests and let them work parallel. It seems the productivity of the whole team will be greatest. Once the acceptance criteria has been clarified, dev can write unit tests, and tester may write functional tests. Ideally it’s true, but it’s difficult practically. The following are some potential problems related to this approach:
* Sometimes dev and testers can work parallel, which may increase productivity(but again it’s dubious, because this may need more communication and iteration between testers and devs).
* Depending on the tools the QA uses, sometimes it requires QA to have more advanced programming skill to write and maintain functional test suite. If QA does not have that level of skill, it’s difficult for QAs to catch up with devs to write functional tests for new developed functionalities. If developers deliver the stories very fast, QAs’ job will be more difficult.
If these problems happen, there are some ways to mitigate it. One way is to let developer write and maintain some (not all) functional tests (happy path tests, for example), integrate them into developers’ build and make sure them pass before developers’ commit.