I am currently looking for suitable tools to help with automated testing in one of the projects I have been working on. This post provides an introduction into the topic and gives a brief overview of some of the available frameworks.
During my quest for the killer testing tools I came across two excellent posts elaborating on difficulties hidden in automated testing. The first one points out obstacles hidden in regression tests, a major concern of mine these days. The other post on automated GUI testing dates back to 2007 which makes it especially valuable for me. I subscribe to all the points found against the frameworks of that time.
Here is the list of questions one might ask when considering the test automation proper:
- How do we cope with ever changing requirements when writing our test scripts?
- Do we have to wait for developers to do their job first?
- Do we actually need developers to take ownership of the test scripts?
- How expensive is it, in terms of time and effort, to maintain the automated tests?
I am sure the list could grow quite long but you get the idea. In my opinion, automated testing was rather expensive to establish and maintain several years ago. Today, I believe it is not much easier either. The major change I can see is a wider portfolio of tools supporting different aspects of the testing automation.
I would say the key to success lies in combining several tools and understanding their limitations. I consider unit tests mandatory but what else could be added on top of them?
A minimalistic approach would be to automate acceptance tests. In my experience, stake holders tend to underestimate their participation in testing. Therefore, many bugs occur shortly after the release has been signed off after a round of manual acceptance tests. On the other hand, the customers really are the domain experts. They surely know what is expected and perhaps could provide a few examples to describe the correct behavior. At least, that is the prerequisite the frameworks like Fit or its successor FitNesse have evolved around.
When it comes to capturing the requirements both of them use the tabular approach. The tables serve as examples or hints given by the stake holders to the development team. Fit suggests that business representatives should be free to describe their requirements by using the tools they are familiar with, such as MS Word. Usually, a brief description accompanied with a simple table is provided. The document is then converted into an HTML page and the framework takes it from there by mapping table's content into method calls and variables. FitNesse preserves the core concept, adds a variety of options to it and works as a wikipedia to emphasize the need for constant communication and collaboration.
Since the testing is example-driven, only a limited number of scenarios can be tested. Also, the integration with modern CI servers is questionable. Finally, the collaboration does not go much further beyond a simple wikipedia and does not really enforce the constant need for being up-to-date.
Those who are lucky enough and work in a truly agile environment can make most from automated testing by using the Behavior Driven Development. The ultimate goal is to guarantee that only the required features are developed, tested and delivered. Such features should have a significant business value and are typically defined by stake holders. To set an example, Cucumber is probably the most popular tool at present.
Cucumber overcomes the limitations of the tabular descriptions by introducing a domain-specific language. The initial scenario is written in a structured way using the language close to plain English. Before any coding is done the scenario is reviewed and approved by the business side. Only then the test fixtures are created and bound to individual steps. Testing starts with a first round of failing tests. Implementation is added to fix the failures, followed by another round of testing etc. until all the tests pass (green as a cucumber).
What I find really cool on Cucumber, apart from Gherkin, is the fact it caters for a wide range of technologies and integrates with other tools and frameworks. Since the official site is bloated by some marketing blurbs from 2009 (!) I recommend checking the project's wiki instead.
Finally, there is a variety of tools specialized on automated UI tests, performance testing, integration etc. You might want to check the quick links on this page.
This post merely scratches the surface. I plan to provide a detailed example of using the TestMaker for the automated UI testing. Hope you find the post interesting nevertheless.