For the past few months I’ve been working on a contract to build an automated web testing suite for a large and complex web application. I have used Selenium to build the automated web testing framework so this post explains why I chose Selenium and how great Selenium is as a tool for web testing.
At the September 2009 meeting of the Sydney Business and Technology User Group (SBTUG), I was able to present a high level, business focussed overview of using Selenium for automated web testing. I focussed mainly on IDE but did delve a bit into the code by showing some C# code and running the tests from within Visual Studio. You can see the slides from the presentation on SlideShare. I will present this topic again at the Sydney Alt.Net user group with a bit more of a .Net developer focus.
So to start with, I love Selnium. It’s free and it comes in 3 different flavours that can be used by different parts of your team. The end users and the testers can use IDE, the dev team can use RC and the people doing the serious performance testing can be using Grid – all with the same test code (or versions of the same code).
But this project was not your ordinary Selenium project. Most selenium projects are done in Java or Ruby and if they are done in C# they use NUnit. My project was done in C# using the Visual Studio Team Suite Test Edition (VSTT for short). This is because all the tests for the main non web-based app are built in VSTT and I needed to work in the same environment.
Also, the website I was testing is built with a Javascript Library called ExtJS. One of the “features” of ExtJS is that it builds the HTML Element ID’s on the fly, each time the application is run. Other constraints my project has is that the app only works in Firefox 2 (yes Firefox 2 – yes, it is not a supported browser any more, and whilst not as bad as IE6, it is still an issue!), so therefore I could not use the in-built recorder in VSTT, which only records in IE.
So, I wanted a testing tool that I could record the actions that the users took through the system, that would handle the issues with the ExtJS framework, would work well with VSTT and would scale up to the eventual result of running 1000 web tests in a 2 hour period with 27 concurrent browsers operating. Selenium provided me with all that goodness, and it can run tests on all the major browsers also.
I’m not going to go into the basics of creating web tests in Selenium as there are some great tutorials out there, and the step by step guides on the Selenium website are very good also, and I’ve covered a lot of it in my presentation slides. I will delve a bit into what made this project so challenging and how Selenium helped me overcome it.
ExtJS is an interesting JavaScript framework, and I understand why developers want to use it. From my limited understanding of it, it seems to build the HTML on the fly as it’s needed. So if there is a grid that is 50 columns wide and 500 rows long and many of the fields have a combo box behind them, it will not create the combo box code for the specific cell until you click on the cell. (You could say, why does a Web App have a table that big, and you would be making a very valid point).
However, from a black-box testing scenario (ie I was given access to the Web App but no access to the development team), ExtJS is a very difficult framework to deal with. The element names are uniquely generated long ID’s that change every time the app is run. It means that rather than referencing a simple input box by the name assigned to it, we have to use an Xpath reference to refer to the input box. Selenium is great that it can handle Xpath refrerences, but Xpath references are a pain in many other ways. Firstly, they are long and cumbersome – especially with ExtJS. Eg a pop up box is not on a separate HTML page, it is just built with a DIV at the bottom of the main code, so to interact with an input box inside a pop up box, I first have to find some way of uniquely identifying the pop up box (ie maybe by the title), then find out which DIV holds the whole pop up box, then search within that div for an input box that is next to a label that I know the name of.
This is one way of doing it but it is very fragile and is not a great long term strategy as it needs to be re-built any time there are small changes to the Web App. It is amazing then, when after I had worked out this strategy that I found an excellent post (UPDATE: The original post has been taken down, so the Author kindly let me put the PDF of it on my post) explaining this very problem that I had encountered, and explaining a fantastic way how to overcome the fragility of testing with the ExtJS framework. Now it’s not something I could do in my short 3 month stint, but if you are ever thinking about testing an ExtJS Web App with Selenium I would highly recommend it. It was great to connect with Lindsay Kay, the author of the post just to say Hi and know that someone else in the world understood what I was going through to build these tests.
Again, I won’t delve into the wonders of Xpath in this post, but there are some great links in my slides about how to use Xpath, and the No.1 tool to use for Xpath if the Firebug Console – it Rocks!
I am also not going to bore you with the intricacies of the VSTT test suite, except to say that once it is set up and running it is a very good test environment as it has excellent reporting and debugging and great integration with TFS (but we only had VSS to work with and it worked fine). However, I have seen a few short demo’s of the new test tools in Visual Studio 2010 and I can’t wait to actually use it in a real test environment. The only drawback is that it does require TFS, which for some small dev houses can be a bit of an expensive option.
What I do want to finish this post of with is about scaling selenium using the Selenium Grid. I was really apprehensive about using Selenium RC and Grid as they seem difficult and use Java and the command line and Apache Ant, none of which I was familiar with. The Selenium Grid step by step guide actually makes it very straightforward to get it all up and running. In no time you can fire up a grid and multiple browsers and then run multiple tests in parallel, each within its own browser.
Purists say that you can’t use Selenium for Load Testing, mainly because of the hardware required to run multiple browsers on the same machine (I used a server box dedicated to just running browsers and was able to run about 15 at once and I could do around 5-8 on a single desktop PC). However, there are two great companies out there both using the goodness that is Amazon EC2 to solve this hardware issue for you.
Sauce Labs allows you to use your own code and run tests in the cloud. It has many environment / browser options to choose from and it is amazingly cheap. It is currently in early beta and probably has some things it could improve with reporting but it is a great service, with great support. It is really cool firing off tests on your own desktop and having them run in the cloud.
BrowserMob is a bit more expensive than Sauce Labs but also has fantastic support and lots and lots of options, including bandwidth limiting and the ability to use real browsers or just send http header requests. Browser Mob requires that the tests are built in their app and this can be a bit time consuming. They also have a great range of reports and the ability to run SQL queries against the result set.
Overall, I have found this 3 month contract very interesting and I have learnt a whole heap of new skills. I could not have done this project with out the team around me setting up the VSTT Environment and teaching me some much needed C# skills, but with all the Selenium, Xpath and ExtJS stuff I just had to learn all that by myself. It has shown me that I really can do anything, that I have the right mind set to just learn new stuff, work out how it all works, then document it and hand it over to the team that will be taking it on from now.
Now onto the next challenge, what ever it may be…