Video: Ron Adams — Automated Integration Testing with YUI Test, Selenium and Hudson

Video published 2009-10-29.

• Download iPhone/iPod-compatible video file (320x240, .m4v)
• Slides

Transcript:

Ron Adams: My name is Ron Adams, and obviously I'm presenting on Automated Integration Testing with YUI Test, Selenium, and Hudson. Before I begin, I just want to let everybody know that there are CLAs here for engineers that might want to contribute to the YUI Gallery at any time in the future. You can come down here and pick one up, fill that out, leave it for us, and that takes care of that step if you ever feel like you might be contributing with us.

I have been working on Yahoo! media products for about three years now. I started off with the now defunct Yahoo! Broadway — not because of anything I did, just because it is. [laughs] Then I moved on to OMG, and for a few months I was working on Frontend for them, API layer stuff, and backend. Then after a few months I moved onto Yahoo! Fantasy Sports, and that's where I've been for the last two years, making fun stuff happen. Now I'm integration testing and unit testing.

I've got a lot of stuff to cover. There are three pretty big technologies here, so I want to let you know that I can't go too in depth into any one of them, but I'll probably spend a little bit more time on YUI Test than Selenium or Hudson since this is a Frontend engineering conference. I want to get a brief survey: how many engineers here actually use a unit testing framework on their JavaScript to ensure quality? OK, so a relatively good number. Of you guys, how many of you have looked into ways to automate it in your build process? So we've got about ten, twelve, thirteen. OK, so I don't have to convince you guys about the potential value of having a system integrated into your build process for testing your JavaScript.

But some of you may be thinking: why should I test my code? It's a legitimate question, but it's certainly got a lot of answers. If you're into numbers, there was a study done in 2006 which took into consideration a set of companies in the software development industry, some of which didn't use any test development, some of which created their tests before they went into development, and some of which made the products, then made the tests, and then used them to ensure quality after they were done.

What the findings showed were that 18 to 50 per cent more test cases passed for the groups that were actually using any sort of test driven development in their process. For those that were testing first — so they were planning out their objects, they were planning out how their implementations were going to work first, they built their tests, then went into development to have the tests pass — there was 88 per cent less effort per line of code for those groups. And even for the groups that tested last — so after they did their development — there was 57 per cent less effort per line of code, which is an interesting metric, effort per line of code [laughs]. You might think there were probably some differences between the skill level of those groups, and there really was no statistical difference between the skill levels. There's not too much research out there on this, but I have a feeling there's going to be more and more as the years progress.

What are other reasons you would test? Obviously the big one is higher quality. For instance, you might be on a project with a small number of engineers and decide you need to hire a bunch of talent because you need to get some new stuff done. They're working on global implementation, or maybe some utilities or something like that which are not as well documented, and when they get in there and start messing around stuff starts breaking. Well, if you had created tests for that initially, you could catch those things early and maybe not have them get to production.

It helps you evaluate your dependencies. While in true unit testing you don't want to have any dependencies that aren't mocked or faked, system integration testing helps you evaluate your dependencies. For instance, if you've got YUI 2 and then you're upgrading to YUI 3, if you have tests for all your stuff, you can swap out 2 for 3, run all your tests, and then see if your code is going to break. You can test the backward compatibility of your own code. So if you're creating a new widget for your site, or you're enhancing an older widget or some sort of global utility or something, you can run that test. Especially if you can run the tests in your development environment, that can help immensely as you're actually doing the development. Then obviously, all these things result in peace of mind and reduced stress, which everybody looks forward to when they come to work. [laughs] At least, that's what they want.

I want to get into some definitions, just so everyone's on the same page. First of all, how many people here are familiar with unit testing, test driven development? OK, good. I still want to go over the definitions a little bit so that we can evaluate what each of these definitions means for JavaScript. Test driven development is sort of the whole enchilada — it is the definition for planning out your code, writing your tests, and then going into development with the idea in mind that you want to develop in order to make the tests pass. Then that will create a much higher quality code base. Unit testing itself is testing the smallest piece of a module in total isolation. You have total control over all the dependencies, you fake everything out. Object data, or any sort of data that you're feeding into a publicly accessible method, is controlled by you, and you know what to expect out of it so that you can assert whether or not the stuff that you actually are getting out of it equates to what you expected to get out of it. You need to have a lot of control in that sort of scenario.

Integration testing you don't need as much control, so it's essentially testing all of your components together in a system where there could be some mocks, there could be some fakes, but it's not in total isolation, and it exposes issues between dependencies that may be working with one another. Then there's system integration testing, and that's just taking your tests and throwing them in a live data environment. You're processing your feeds, and your feeds are actually populating your backend APIs that you're interacting with to make your code work. You bring in your tests, and you're testing in that live environment.

JavaScript is pretty special — like we didn't know that — because of many factors, and that impacts whether or not you can do true unit testing or integration testing. There are no fakes for the libraries we're using, or for the DOM itself. If you're using maybe a connection manager or something like that, and you depend on that, and you call connection manager, and in the method that you're testing you go out to try and find a URL and it doesn't exist, or you're offline, or something like that, then that causes a problem. What would be great is if instead of using that dependency you use a fake version of the dependency, but those sorts of things aren't really robust at this time, or exist. [laughs]

And also with DOM, there's no way to emulate each of the different DOMs so that you can test the quirkiness of each of the browsers. You actually have to test within the browser in order to see where the code is going to fail cross-browser. We're just in relative infancy here, so while other server side languages have more robust unit testing systems like PhpUnit and JUnit, and we still have some, they're still in relative infancy. Also code coverage, or being able to record the percentage of lines that are actually executed in your unit tests, there's very few tools out there for JavaScript at this time.

Can we do unit testing, or should we just do integration testing? I think a combination of the two is probably most comprehensive. If you try to do command line unit testing on a small piece of a component, you're not going to get all those browser quirks and catch all those issues with cross-browser execution. I feel as though integration testing is arguably more valuable for JavaScript. I think that you can do a lot of the same testing that you would do command line in browser, it's just there are those extra dependencies. I think a lot of engineers cringe when they hear that because they want full control, but it's a little hard at this time and with the state of the technology to make true unit testing happen with JavaScript, especially DHTML or DOM manipulation, or JavaScript.

Before we move on any further, we need to become familiar with continuous integration. How many people use the continuous integration tool in their development? OK, not as many as do unit testing, which is interesting. But it kind of explains why not everybody looked into automating their unit tests. One great resource is Martin Fowler. If you take a look at his website, he's got articles on continuous integration, mocking, faking objects, and doing unit testing. He's an international speaker and he's really well versed in this stuff. He's also into agile development. I encourage you to go and look at this article, in particular, and look at his definition for continuous integration. What he says is that you need to constantly integrate your developer work, at least daily, to help expose the development areas. Those of you that use CI in their build process, you guys know the benefit of that, obviously.

The system should also auto-magically build and install packages or files, and you want to install them on a production clone so that you can see whether or not your stuff is going to be unstable in the production environment. The system should also schedule tests to run as part of the build process, in parallel with it, nightly, whatever you need to do. And then it should turn any lights red that need to be turned red, to tell people that your build is unstable and that it's really broken.

With all those things in mind, what does it take to automate JavaScript? It's pretty clear that there are three things that you have to do: you have to have a unit testing framework to actually create the unit tests and to execute them in browser. You need a way to launch that browser, and Selenium is a pretty good pick for that; it handles a lot of stuff. I'll go a little bit deeper into Selenium in just a little bit. And then you obviously need a system to track and monitor the tests results and the health of your build, and display the results in some user-friendly manner, ideally. If we're looking at this scenario, we need to keep in mind that Hudson is JUnit-friendly, so it can easily consume JUnit result data. If you can't get your test results in JUnit just automatically, you'll need some sort of conversion system to make that happen. But you can find stuff out there, for sure, for pretty much anything.

Let's take a look at YUI Test first. I encourage you all to go to the YUI Test developer page and take a look at the documentation and the video and everything. It's developed by Nicholas Zakas, who's an incredibly talented developer. He made a really flexible unit testing system which plays well with other libraries, so if you're not using YUI — and I assume you are, but please feel free to continue to use whatever library you want, though YUI Test can help you regardless — provides a mechanism to send test results to a given host's entry point using the YUI Test reporter. That's really helpful, and it's open source, so obviously you can contribute and if you come up with some sort of system that you might want to contribute via GitHub, go for it.

YUI Test 2 and 3 features some things that are consistent between the two of them: their setUp and tearDown methods. Most of you who do unit testing will know about this sort of pattern where you set up your test, and then you create object data for the test that you're going to run, and then in tear down you destroy those objects. You want to do that especially in JavaScript so that you release memory, because the setUp and tearDown gets run every time your test methods are run with YUI Test. There are all kinds of assertions for sameness, data type, special values. You can simulate user interaction, or user actions, click events. There are all kinds of stuff for you to explore there. And support for asynchronous testing as well, as well as intentional errors. If you know something's going to fail, or you want to test the error handling of your objects or your methods, there is support there so that if your methods actually fail you can still pass your assert. Then you can just ignore test methods if you need to, which I've actually found to be pretty useful when testing in multiple contexts, and which I'll talk about later.

New to YUI Test 3, which is really cool, is the support of mock objects. In that example I talked about earlier — having the connection manager or something go and request some data when you're offline, which doesn't work out so well — you can create a mock version of that object and substitute it for what you were depending on earlier. This helps control issues that might pop up with using that dependency in your tests.

Now I want to take a look at a live demo — let me know if you can't hear me because I'm bending over here. First of all I just want to go over YUI Test and take a look at it on the developer page, because I didn't want to try and cram it all into a slide. This is how to actually implement a test with YUI Test. In this example, you're namespacing. You'd create your test case, you name it, get the set Up and tearDown which I spoke on earlier, about how they get run every time. A test method is run. In set up, I find that if I want to test an instantiable object then I will create this.testobj, then instantiate it with mock fake data, and then use it down in the testing methods, down here. Here's a test name, and then you get your assert so you can make assertions on the data that has been returned from the method that you're testing. A good convention is, if you're testing a publicly accessible method of an object you have, the name of the test method is just 'test' and then the method name. In CamelCase that's always a good pattern. No matter what, your test methods have to start with test, because YUI Test just runs them automatically if they begin with test.

This is a very simple example, but it's pretty much as in depth as I can go at this point. Here's the creation of another test case just for good measure. What you'd do is create your testing suite, so you just instantiate a new suite, name it, then add those test cases that you created earlier it to it, and then you run them. You use test runner, add your suite to it, then run test runner. It will open up the YUI Logger and log all the results of all your test methods. It'll say pass, fail, it'll be green, red. Ignores are gray. I can show you an example of that. This is a demo on my development machine of… I work on the Fantasy Sports game, so I create tests for all the new development we're doing, and this is the edit schedule tool, commissioner tool, in the Fantasy football games. This is the test for it — hopefully it loads. The page actually loaded, and I'm executing the tests. This is actually one of the continuous integration tests that we can run each night. I added some fails for you, so you could see how the passes are green and the fails are red. Then it gives you a completed at time, the total account on the number of fails in the total. So that's just an example of YUI Test in an actual environment.

Now we can talk about Selenium a little bit. Like I said, we're not going to deep dive into it because of time, but you can also always go to SeleniumHQ.org and read more about it. It's a fantastic tool for providing the ability to launch almost any A-grade browser. It does it via Selenium remote control. With Selenium remote control you can have a task, have it execute on a Windows machine on whatever A-grade browser you want, it'll launch that browser, launch the test in the browser, and then you can actually see it if you're looking at the machine with a monitor, which I'll show you later. Not all of the Yahoo! defined A-grade browser spectrum is covered, but you'll get a lot of good support there and it'll open up your eyes to a lot of possible issues. It supports several expressive languages for extending and creating new tasks. You can pretty much develop in whatever expressive language you want, and create your tasks in them.

The Selenium grid allows you to run all these tests on multiple servers in parallel. You can take care of all your testing a lot faster if you're using this sort of system. It's open source, just like everything else we've been talking about, and there's this great Selenium IDE. For people that are new to Selenium, that's an extension to Firefox. You plug it in, press record, and then interact with the DOM, do whatever you need to do, and it'll record all your actions and then save them. It will then export them in a task structure in whatever language you want. When you actually run that task, it will execute it just like you did in your own browser, and your task will work with any browser, not just Firefox.

Now let's take a look at Hudson, obviously. hudson-ci.org, and you guys have got to go there because like I said, I don't want to go too in depth, but that's a great resource for this. It's a Java based continuous integration system. It will run your Selenium tasks, and it's perfect for exactly what we're looking at here. It consumes JUnit format, as I talked about earlier, which may require a converter. One interesting thing about YUI Test that I was talking with Nicholas Zakas about were ways to possibly improve YUI Test. He just added to the upcoming release the ability to convert the data to JUnit, so you call the test runner and you say get results, and then you pass it the fact that you want them formatted in JUnit. You can just grab that results set in JUnit as of the next release, or you can get it off GitHub if you want. It can alert you if the build breaks, obviously, or if your test fails. That's pretty common for any CI tool. But it's really widely used, so it has a lot of support and active development. You can extend it, which is great, and a lot of people are doing that open source, again, which is fantastic.

This is just a brief look at the dashboard homepage. You can create new jobs, configure your environment. It shows what's going to be built next, and then actually if stuff has been built or is building, it shows you the progression of them. Then over here, obviously, it shows not the actual full results but whether or not the builds of your packages actually passed or failed, and when their last success was, their last failure, and how long it takes for them to run. There are obviously a lot of other pages to Hudson that you can click through, and I will show you a little bit more of that when I do the end to end demo at the end. But like I said, if you go to http://hudsonci.org, there is a button that's like 'try now' or something like that. If you hit it, it's pretty much an instant install and you just go to local host 8080, and you can try it on your own machine immediately. It's great. Just installing it is really easy to do.

I want to talk about the actual architecture of a system like this. Now that we have which tools we could use in place, we have to architect this in a manner that is the most flexible, and can open up many possibilities. You don't want to limit yourself, obviously. I've found there to be three major pieces to this, which correspond with the different tools that we're using. I'll go into a little bit more depth on each one of these. You've got the test provider, which is the system with which you create your tests, and then provide to the external world the list of tests that are possible to run in your environment. The test launcher is obviously Selenium. Then you've got to launch these in the browser, and then you need the CI system. Then there's this cycle of task execution and result consumption that comes out of this. You want to make sure it's in a loosely coupled environment so you can swap out the CI tool, you could swap out the Selenium tool, or your test provider implementation at any time, and not influence one another.

This is an ideal architecture at this time, for the system that I'm talking about. You've got the CI system which tells the test launcher to launch any tests that are available. It goes to the web service to find the list of tests that are currently available, and consumes that. The web service exists on the test provider infrastructure, and then goes back to test launcher. Test launcher runs all the tests by executing the tests, the tests presentation aspect of the test provider consumes that, and then converts it to whatever result format is needed for the CI system to then consume and report. This is pretty much what I did for our property.

If we were talking about the test provider there are some important elements to consider. Obviously, test provider is just a label I came up with so we can discuss it more in depth; it's an abstract term for whatever framework you need to implement in order to have it work for your property. It just needs to present tests in a browser in some way, shape, or form. You need to be easily able to create new tests, because I know people that will not do any testing if it's not easy to maintain, create new tests, and that sort of thing. In the actual system itself, the test provider should try to eliminate as many variables as possible. Like I said, we want to create as many mocks, or fakes, of data that we can when we run our tests. Ideally you'd want to test contextually, so in multiple contexts.

For instance, I developed a player node for Fantasy Sports, and in some contexts, in some games, it doesn't have any tabs and it doesn't have any views. In another context it could have tabs and it could have multiple views. I want to be able to re-use the same YUI unit test file in both cases, but then extend it with some other file which is like the contextual test, and which would just extend the YUI Test, add ignores, and add failure checks or error checks when you test in the other environment. Also, you want to have different possible HTML stub files for each context. I'll discuss more about contextual testing in just a little bit.

Like I said earlier, it's best if this system is loosely coupled from your test launcher and CI stuff. I mean, not only do you want to have that web service façade — or some sort of façade — to inform the test launcher of the tests that need to be run, but also keep it out of the actual file structure of your actual product, and keep it separate so that it's easy to package it up. Only install it in environments it actually needs to be installed in, like DEV, or QA, and not have it go out to production. The web service is one example of how to do this, but it's supplied the tests in a loosely coupled way so that those tests exist in the test provider area. You're not maintaining a static list tests in the Selenium task or something like that, because then you've got developers that need to create a new test over here in my test provider framework, then I've got to go over and edit this Selenium static list too. Have your test provider auto-magically create that list of tests.

How do you eliminate variability in the test provider area? Like I said earlier, you can use HTML stubs so that you're not testing dynamic modules, you just render the module on a browser, grab that HTML, and then stick it in a file so that you can use it in your tests. Like I said, dynamic data, if you absolutely need to use a dynamic module, make sure all the data that it's using is fully mocked, and fully fake. If it's not, then you could have some issues just rendering the test. You expect the module to be a certain way in the browser. Obviously it's is a good practice no matter what, but avoid dependencies that are in a beta stage because otherwise you're going to have some variability there. Use your production template, if you can, in your tests. You obviously need some sort of template to render these tests. If you're including Seed or something in your production page, include that same file if you can in your test template, and try and make them line up as much as possible so that you've decreased the variability between what you're doing in production and what you're doing when you render your tests.

You also want to be able to test in multiple contexts. How I like to illustrate that is it's like the examples at the YUI documentation — you're all familiar with that, I'm sure — where they introduce a certain widget, or something, on a few different pages to show you the functionality or the features of them in action. The same sort of concept should be used for this sort of multiple context testing. At least, that's what I've found useful. To help facilitate this, file names should include context IDs so that you know exactly which files are associated with a given contextual test. We'll talk about a good file naming convention in just a second. Like I said, each contextual test could use a different HTML stub file, just like the examples on YUI. Then if you need to test contextually, so that you don't reproduce code and don't repeat your principles, extend your base test. Have a default YUI test for the given file, and then if you need additional stuff contextually, create an extension to that base test class in a contextual file.

The naming convention that I feel covers all this stuff pretty well is like this: you've got file name.JS, which is your base JavaScript file. All the test assets that are associated with a given test would have this sort of CamelCase file name test. This is just something I came up with; obviously it's not the be all end all. But you want to have some sort of naming convention to show new users that are coming into the infrastructure that they can definitely recognize what's associated with what. You've got the context as the second dot, and then you've got component type, and extension. I've found that having a good, explicit context type, like ID here, is a good situation. You can then write the component type. I have mock HTML, mock data, here as an example, and then your extension. So if you're creating some sort of framework for this, and you're getting all the files associated with a given test, you can create an array of all those different pieces and see whether or not the context is default. If it's default, then it know it's that default test. If it's a different context then you can treat it another way.

I'm covering a lot of stuff, so if anybody has any questions about this at any time, feel free. I haven't seen any hands yet, but please do.

When you're establishing the loose coupling, one good thing that I think when I was implementing it is that I've just got a quick and simple web service. You've got your YUI Test list, and the ID of this would be the ID at the top of a unique test that you have. The URL to run it in browser here is a contextual test. I just say the sort of system that I felt like I wanted to come up with was a unique test ID, a colon, and then the context, so that you can pop off that context and then have it if you're including files which mock data files to include in your test when you're running in the browser.

Moving on to the test launcher a little bit, here's an example. You can't just copy this and make this happen somewhere — this is just pseudo, I've just put it together really quick — but it still illustrates what you can do. You create your new Selenium, you start it, you open up that URL that you need to execute, and wait for it to load. After it's done loading you just have to wait for test runner, or the YUI logger, to execute. Just check to see if testing completed at is in there — if it is, then stop, parse that log, and convert it into JUnit. Then write the report to some file for the CI tool to be able to consume. This is another situation, like I said, Nicholas Zakas added this new way to fetch the data in JUnit XML. Selenium can evaluate the JavaScript in the browser that it's launched, so you can use Selenium eval to check whether or not test runner is running. Then if it's not, you just output the results, you get it from test runner, and format it in JUnit XML. Like I said, it's not available right now in YUI, but that functionality should be in the next build.

Right now I want to demo an end to end situation which takes all of this stuff into context. Here is a demo Hudson that I created with our QA, Pranav, on Fantasy Sports — fantastic guy, and he works so quickly. Here is the dashboard to run just this demo YUI Test. Like I said, Hudson has all kinds of features — you can configure your test and execute it right now, which I'm going to do in just a little bit. And then there's all kinds of meta information about the current test that you're running. There's also this test result graph, and that's obviously really user friendly. It converts that JUnit output into a nicely formatted graph. And then your build history, so every time that this thing is run, you can see how many times it passed something, and what's its recent situation is.

This CI tool exists on a machine in Sunnyvale and I'm going to enable my Selenium server on this machine, so this is my test launcher that I've just enabled. Then the test launcher is going to communicate with a machine in Santa Monica that I've set up, which actually has a list to consume of all the tests that need to be run and executed. It's going to find two tests on my machine in Santa Monica and execute them. For this demo, I just needed to be sure I entered the right IP of this laptop. Then let's watch it go. You'll see, in a second, the build history over on the left hand side, and it's just showing you that currently the build is executing. Now my other Firefox instance is launching. So it's going to launch the first test in my browser, and it's going to log all that output. It's executing closed, and now it's executing the second test in that web service that I have.

There are some failures there, and then it kills the Firefox. If you go back to the Hudson you'll see that it completed, the light's yellow, and that means that it's unstable — because like I said, I had those three failures in there so you could take a look at it. You can get to the results of the test via the graph, if you want to. Click on there. OK, on the last build for this there were three failures, and here you can look at some detailed results. You can see that on my test drop match up and test disabled saver sets, in both of the contexts that I was testing in, it failed. The JUnit output for something like this would look like a test suite, test case, the name of the test, and how long it took. You can input that in there. Then if there was some sort of error, you can log that. And you can see what the actual error was by clicking down here, on 'drop test match up', where it says unexpected error. That's what you see here, unexpected error. That's sort of like an end to end demo of what's possible if you set up this sort of architecture.

I want to also talk about system integration testing. If you've gotten to the point where you have this integration system to run nightly, it's probably also nice to create a system where your tests automatically get included in your development environment, so that you can do true test driven development with ease. Obviously, you want to use the same YUI Test files as integration test so that you're not duping test code here and there. Include the tests on the page if they're available, but if not, don't do anything. Let the developer or the QA person that might be involved control the test display, and whether or not the test is displayed, with maybe a cookie or a query string parameter.

I can show you an example that I've created in my development environment. Say I'm working on the edit schedule page, and everything's looking pretty cool. I think all my stuff is working really well. If I install my YUI Test environment then I can enable test driven development. This just sets a cookie. The cookie is set, and then if I refresh it'll include all the tests, logger will show up, and I can see whether or not the work I'm doing is actually passing. Since the cookie is set, I can navigate through the entire site, and if a test is available for a given page then it will include that test automatically. That's good for QA. If they wanted smoke tests, a bunch of different things, and they just set the cookie, then they could see any page that could potentially have that test actually run it. Then if I just disable that cookie, the next time this loads the logger won't be there, but the developer also has the flexibility of adding YUI Test equals 1, and then also having the logger output displace. This is full, live data in a development environment, but this obviously has a lot of value and benefit to a developer that might be wanting to do test driven development.

That's pretty much it. Like I say, I don't want to do a huge deep dive into any one particular thing, so I encourage you to go to YUI and go to these links. I think the slides will be available somewhere after the conference, but here are a few things. You'll find them with ease if you do a search. I encourage you to take a look at those videos.