TestMaker has been around for a long time, has decent documentation and feels very mature and versatile. I have decided to give it a go and share my findings with you. This post is primarily about conducting Sahi scripts by using a recorder and building up test scenarios.
All right, my intention was pretty straightforward. After playing with the examples the tool comes shipped with I wanted to see how does it perform for real. In one of my previous posts I have built a simple application, basically a booking wizard. Several steps were involved before the booking could be made. As you can imagine there were many details to consider:
- Is the correct step being displayed?
- How about the back-forward navigation? Can it be accessed?
- Are all of the expected fields and options available?
- Have all the details been correctly captured?
These and other questions added up quickly to a test case which was yet to be created. I opted for Sahi as an underlying technology due its great flexibility. I have started with recording of the UI actions.
I have decided to take it easy at first and recorded the whole booking all at once:
Here is the resulting Sahi script:
var $storedVars = new Array()
_navigateTo("http://zezutom.xtreemhost.com/ko_flights.html");
_assertVisible(_listItem("Personal Details"));
_assertVisible(_textbox("fname"));
_assertVisible(_textbox("lname"));
_assertVisible(_submit("Next"));
_setValue(_textbox("fname"),"John");
_setValue(_textbox("lname"),"Doe");
_click(_submit("Next"));
_assertVisible(_listItem("Seat Selection"));
_assertVisible(_select(0));
_setSelected(_select(0),"That dark corner");
_click(_submit("Next"));
_assertVisible(_listItem("Extras"));
_assertVisible(_select(1));
_assertVisible(_select(2));
_assertVisible(_select(3));
_setSelected(_select(1),"The Amazing Spider-Man");
_setSelected(_select(2),"Waterproof");
_setSelected(_select(3),"Get me a bike");
_click(_submit("Next"));
_assertVisible(_listItem("Summary"));
_assertVisible(_tableHeader("Personal Details & Seat Selection"));
_assertVisible(_row("John Doe That dark corner"));
_assertVisible(_tableHeader("Extras (Entertainment & Transport)"));
_assertVisible(_row("Movie: The Amazing Spider-Man"));
_assertVisible(_row("Headset: Waterproof"));
_assertVisible(_row("Car Rental: Get me a bike"));
_assertVisible(_submit("Send"));
_click(_submit("Send"));
_assertVisible(_link("Mailbox (1)"));
Seamless and fast as it went, it was not ideal. I ended up with a monolith inclined to break even when a subtle change was made and difficult to maintain in general. As the next step, I edited the script manually and broke it into chunks. Logically, I wanted to use the Designer afterwards to verify that every single one of them actually ran fine. Here I hit the first limitation:
The recorder does not accept anything but its very own proprietary format.
Even though I was able to export into Sahi the reverse action was not available. In the end I verified the amended scripts using Sahi as such.
Further, be warned that the recorder provides a limited range of controls when compared to crafting the scripts manually. For instance, I wanted to ensure that the correct item in the main menu was highlighted as I was progressing with the booking. That required a check for a CSS class be present in the menu item element. Using Sahi it can be achieved as follows (credit goes to Sumitra):
_set($itemClass, _listItem("Personal Details").className);
_assertEqual("selected", $itemClass);
However, the recorder did not seem to support the aforementioned case. Or at least I could not find an easy way of how to do it.
After all that hard work I could not wait to use my scripts in an actual test scenario. I found the TestMaker Editor intuitive and easy to use and had my first scenario ready in no time:
A little explanation is needed I guess. As I said I split the initial single script into chunks. Each of them represented a single step in the booking wizard. Thus the first script was called Provide Personal Details, followed by Select a Seat and so on until Submit the Booking eventually brought the wizard to the end. That way, there were five scripts representing five wizard steps. There was also an additional script. The only purpose of the extra script was to open up the application in a browser. It was meant to be used as a set-up common to the whole test scenario.
The idea was that the set-up script would start the browser, navigate to the application and leave the browser open until all the tests in the scenario would have been executed. However, to my surprise the test execution failed:
The browser was in fact closed as soon as the set-up script had been executed. Since the subsequent tests relied on the fact the application was running in a browser they were all destined to fail. That brings me to another limitation I find pretty major and disappointing:
TestMaker treats set-ups as self-contained tests. That is true at least for the GUI testing.
To my understanding a set-up is special and should not be considered a regular test. I see this as a bug. The issue was reported more than a year ago and remains unfixed.
That is all I wanted to say with regards to using the TestMaker for automated GUI tests. Despite my last remark I don't want to conclude the topic on a negative note. Considering the complexity of test automation no tool is perfect. I see a great potential in TestMaker and hope the issues I came across will be resolved at some point in the near future.
