Continuous Integration: Selenium RC v.s. XUnit tests

Since April 2008 I've been a Consultant/QE Architect at Sermo in Cambridge, MA USA on Ruby on Rails agile teams. The first team started as an experiment to prove rapid development of rails applications, composited with the JBoss-based java core community, could work seamlessly. We have continued to successfully add several more rails applications with this approach. We are now undergoing a major rewrite of the core community and it's applications entirely in ruby on rails. This new design includes formal SOA interfaces. We are continuously refining our scrum lifecycle as well as our test automation approaches.

We have been focusing on continuous integration with Cruise Control and comprehensive regression testing, including Test:Unit and Selenium tests, an automated test plan generator, and ci_reporter test report, that run with every SVN commit. I also added nightly and weekly batch runs for runtime and more timely tests. The big issue we have been wresting with is...at what level {unit, functional, integration, runtime, browser DOM, load} should acceptance/regression tests be created? This question led to some interesting and healthy debate between development and test staff.

To summarize the testing levels...

• Rails Test::Unit test levels:
• unit test coverage is important to verify the methods and their paths
• functional tests validate controllers operate as intended, including environment and database configuration
• integration tests validate systemic operations that cross controllers, in render pages properly, irrespective of the browser, including AJAX responses for page load
• Runtime tests are run on a deployed fleet, either headless or in a simulated browser DOM
• Selenium tests exercise interactive AJAX and JavaScript in the page, requiring a separate client machine running the selenium-server.jar (Selenium RC for rails) or webrat gem (includes selenium-server.jar and the webrat IDL)
• Jmeter for load tests (including performance counters)

We found that the most important part of a story is the list of acceptance tests. This list shapes and clearly defines the expectations of the story and what makes it complete. Many times we stub out tests for stories in the sprint in test suites at the beginning a sprint and can be implemented by either a test or development engineer. The biggest problem we found was that adding too many selenium tests made the automated build validation time increase dramatically (4 to 10 times that of integration tests), difficult for developers to run regression tests prior to source code commit, and more fragile as GUI implementation changed. We had to re-factor many tests from selenium to Test::Unit functional or integration tests to improve test performance and reliability.

The key to successful continuous integration is to test continuously, either with TDD practices or TIA (test immediately afterward), as part of accepting stories. This includes all code implemented for the story and any additional tests to cover the acceptance criteria. Testing at the lowest level feasible for code coverage is important for test efficiency. This may require the creation or test fixtures, mocking response expectations, and data factories.

Testing should include both happy path and negative tests (exception handling). Development Engineers need to have a sense of ownership for regression tests. Quality Engineers need to have a sense of test coverage completeness. Together the scrum team needs to hold themselves and each other accountable for not leaving test coverage technical debt beyond the story acceptance. Plan this test engineering time into the story. It may mean that your velocity is a little less than not doing this, but the overall sustainable stride is greater and you really do catch problems prior to (or at the time of) committing changes.

There is a place for automating at the GUI with Selenium or a comparable HTML element or application control automation tool. By limiting the use of these tools to testing AJAX or behavior that requires interactive javascript to render the page, workflows between systems, use cases (data driven), and cross-browser testing.

What we found is that ideally... 1. Test Engineers embedded in a development scrum team should have the ability to:

• read and exercise application code
• author unit test cases
• create and work with test fixtures, test mocks and test data factories
• assess adequate test coverage for development stories

2. Test Engineers chartered with testing external to the development teams should be able to:

• deploy to fleets (fully automated is preferred)
• read and understand mocked interfaces (to exercise actual interfaces)
• author and exercise run-time tests (cover GUI and API workflows across the system)
• author and exercise performance/load tests

Continuous Integration assures a solid application code base with full test coverage. It engages all engineers in responsibility for application testing. It allows dedicated Test Engineers to focus on system-level functionality, deployment, load, and user experience.

Nokia Test: Are you really agile?

I have recently had discussions with several management colleagues about agile and find that most are not really getting far enough to realize the full value of agile. Jeff Sutherland and many other champions for the Scrum model presecribe the Nokia Agile Test as a litmus test to deterimine if a team really is agile. This test was developed by Nokia internally to assess development teams at Nokia and their partners. Nokia has the largest number of certified ScrumMasters in a company in the world today. Nokia first determines if the team is able to abopt Scrum by determining if they are doing iterative developement.

• Iterations must be timeboxed to less than six weeks
• Software must be tested and working at the end of an iteration
• Iteration must start before specification is complete

Next, the Nokia Scrum Test...

• You know who the product owner is
• There is a product backlog prioritized by business value
• The product backlog has estimates created by the team
• The team generates burndown charts and knows their velocity
• There are no project managers (or anyone else) disrupting the work of the team

I would also add some items to the list...

• The story includes clearly defined acceptance test(s) [validates requirement complete]
• The accpetance test(s) are automated, part of the code base, and run as a regression suite on a regular basis
• The story is not fully complete (implemented) until the acceptance test(s) are automated

The bottom line is that there are many companies that think they are doing Scrum, but aren't really and are plagued by legacy processes and measures. It is fine to take small steps to migrate your organization to agile, but keep going until you really get there.

Agile Lesson Learned: Iterate in the Marketplace

Convoq (a.k.a. Applied Messaging, Zingdom Communications) closed it's doors Nov 30, 2007. I feel very fortunate to work with a very talented and skilled group of professionals in these 5 years. I had a tremendous experience in leadership and management roles that fostered career growth for me. For a timeline of this business and a summary of what happened at Convoq read the blog entry "Convoq and Zingdom - Five Years" by Chris Herot, CTO and co-founder http://herot.typepad.com/cherot/2007/12/convoq-and-zing.html . I have to comment on Chris Herot's list of lessons learned...

• I can't say enough about the first bullet in Chris' list of lessons learned: "Iterate in the marketplace and not in the conferenceroom. Agile is the only way to go." Especially in startup companies or divisions, where the need for a product or service is identified, but not yet clear how to meet that need, you have to get working product into the marketplace to fail fast. You need a culture that can work with the learnings and let that drive the iterations priorities, along with the larger strategic goals. This doesn't mean you have to be reactionary to what people are asking for, but rather what people will use and how they use it.

• The second bullet in Chris' list of lessons learned is also important to highlight: "Just because you are using agile methods doesn't mean you don't have to plan. Write your stories before you begin an iteration, but don't waste a lot of time on the details that aren't needed until later." This plays well into my blog entry "Agile peak performance in early startups" http://djellison.blogspot.com/2007/10/agile-peak-performance-in-early.html . Not having stories ready and prioritized by the product owner (voice of business and customer) before the beginning of an iteration breaks the cadence that is so critical to consistent agile velocity. Keep the stories written at the requirements level, include the user statement for each use case (As an 'actor', I need to 'task to perform', such that 'goal to accomplish'), include known acceptance tests (key tests that the requirement is built right), and the adjusted priority (1 to 10).

Iterating in the marketplace, and rapidly acting on the findings to adjust the stories and their relative order of priority for the subsequent iteration, allows you to "build the right product."

Hiring Quality Engineers for Scrum Teams

I have frequently been asked "how do you go about hiring good Quality Engineers for an Agile team?" I really feel strongly that there are several proficiencies that needed to round out a strong test group for agile teams. These proficiencies can be enbodied in more than one person, but rarely is any one Engineer proficient in all of these disciplines.

• Domain Knowledge:

As with any Quality Assurance (Engineering) team, there needs to be enough representation of the customer for each actor (user) of the system. This requires insights into the goals and intended use of of the system, and most likely may be someone hired from the user community.
• Development Prociency:
• Unit Tests and representative Acceptance Tests in an XUnit Framework -- This is a tennant capability for working with the developer on a story to write story acceptance tests first (or during the story development).
• Separation of GUI/Presentation tier from Business Logic tier -- This is key to being able to exercise the business logic (where most of the work is accomplished) in tests that reside with the source of that production code. There is no need to drive these test with the overhead of the client presentation tiers involved. The client and presentation tiers need their own representative acceptance test for regression, and exhaustive tests (e.g. different environments).
• Exhaustive Tests through data driven tables -- There are several approaches and tools for exercising acceptance tests with many combinations of data sets (e.g. FitNesse or Solenium tables, xml, csv, etc.)
• Test Fixtures -- This is key to exposing all of the system for comprehensive story acceptance tests. As a company matures with agile, I prefer to create a Test Automation Architect role who is responsible for oversight and creation of enough test fixture infrastructure to enable the team to develop comprehensive regression test coverage that can run following build and unit tests completion daily.
• Traditional Tests
• Performance Test Automation -- This includes both local and load tests as found in traditional waterfall and other iterative SDL's (software Development Lifecycles).
• Manual Tests -- There are some test that are purely asthetic or user experience oriented to assure that the application is pleasing to use and accomplishes all the user tasks and goals of each actor identified for the system.
• Process Engineering:

This is discipline of best practices and well defined procedures that need to be in place for consistency across the team, and to maintain a project cadence. As with software, process can be over-engineered and requires a sensitivity to what is working efficiently, when to turn up the process to improve consistency and predictability, and when to turn down the process to eliminate unnecessary overhead.

The question that usually follows this list is "how do you find a Quality Engineer with a Development Proficiency?" There are talented Quality Engineers with CS and MSCS degrees with years of experience in both development and test engineering. If you can't find them, create them. Find a developer who has an affinity for methodical test coverage in their own work, or who is good at and enjoys testing their own work to step in to a Quality Engineering role. I have been able to convert a few developers to this role and the results are fantastic.

Deep Lean Conference

I attended the Deep Lean conference held at MIT on November 3 & 4 2007. It was an information-packed 2 days with Jeff Sutherland, Mary Poppendieck (with Tim Poppendieck contributing by answering some questions), and Nancy Van Schooenderwoert. Jay Conne (Current President of the Agile Bazaar) hosted and added to discussions.

The weekend covered the origins or agile software development (including many references to The Toyota Way), the cultural and mindset aspects, the techniques for optimal agile, business proof of why agile is best for software development, and great war stories from the field. There were many references given for further reading and study. There was some discussion about how CMMI fits with agile and how if an organization is praticing Scrum well it is already performing as a CMMI level 3 organization. I suppose this may be true in theory, but deserves looking at this carefully. The last session on Sunday was an open panel with many very good questions asked. I also had the opportunity to sit across from Mary Poppendiek at dinner on Saturday evening where I heard more interesting stories.

There were a few things that I thought were particularly interesting during the weekend.

• 80% of the features delivered in a waterfall lifecycle are never used, therefore, in an agile lifecycle the 20% that are used are delivered first. Then you realize you are done and can sell it in 1/5 of the time.
• Scrum needs to be applied to other parts of the organization, not just software development. For example, Sales contracts need to be crafted with the idea of time-boxed commitments and delivery with feeback expectations on a regular basis.
• Executive planning and commitment measures (KPI's) need to be consistent with time-boxed development cycles. You can't have an agile lifecycle with waterfall measures.
• Scrum (or any agile iteration) needs a cadence (see my blog entry 'Agile Peak Performance in Early Startups') to keep the stride of the engineering organization. Re-prioritization of stories and changes in the work to be performed must be planned for a following iteration so as to not disrupt the current iteration.
• Retrospectives are great for reflecting on the iteration just completed, but process improvement in agile teams must be much more than that. It must include a "Kaisen Mind": a stop the line attitude to fixing problems (defects and process) as they occur and regular reflection on things that need to improve for optimal performance and quality. Don't accumulate technical or procedural debt.
• Product Owner is a key role that requires the right person to both represent the customer, and be able to maintain prioritized stories that meet the objectives of the business and the timely needs of the development organization. Product Management still requires traditional Product Marketing Principles and should not be confused with the Product Owner role of breaking down the detail features and prioritizing them, even if both of these roles are the same person.

The speakers intentionally didn't dwell on the mechanics of the Scrum Master or Product Owner, but did talk about the importance of these roles and did answer specific questions about these roles. I asked the panel "...with your experiences, do you recommend managing stories and tasks on paper cards, streadsheets, or an agile lifecycle management tool?" Jeff Sutherland responded that he always uses the paper cards and the traditional Scrum Board in the war room as the regular way of managing the work. He does use other tools, depending on the size of the project and how the teams are dispersed, but will generate cards from the electronic system for use in the war room.

In conclusion, I felt I got my $600 worth of the conference. I highly recommend this conference to leaders, managers, and executives who are either practicing or thinking about practicing agile lifecycles in thier organization. You will leave a believer, have a means for justifying making the move to agile, and have an arsenal of references to share and for further reading.

http://www.agilebazaar.org/DeepLeanAgenda.htm

My Latest Toy - AT&T Tilt

I finally found a sufficient convergence device to entice me to make the leap and integrate my PDA, GPS, camera, and phone. The AT&T Tilt 9825 (htc Kaiser) is an incredible device that embodies all of these in a similar (smaller) form factor of my aging Palm M130 PDA. It's far smaller than my aging Kodak DC camera (1.2mp) with a 3mp auto-focus and 10x digital zoom sensor. It has a bult-in GPS receiver and with Window Mobile 6 it runs MapTech mobile to give me all my marine and land navigational maps and features of a GPS device. The 2.8" display is clear and sufficient size to interact with applications. It has a touch-screen keypad for use of a phone or you can slide and tilt out the keypad with large buttons. When you do, the screen rotates and you can use it in your hands with thumbs or set it on a surface and use multiple fingers. The phone sounds great as a handheld, works with my bluetooth headset, and has a speakerphone (although the speakerphone in my old Nextel Motorola still sounded better in the middle of a conference room table).

I can use this as an mp3 player and can get streaming video that works quite well. Although I'm paying $40 per month for broadband instead of the $15 media access, the convenience is great. It support 802.11b/g WiFi and world-ready broadband. It synchronizes with Outlook contacts as my prior phone, but has a richer integration. I get email on this device from my earthlink account (leaving the email on the server), and then suck down the email on my notebook. I am able to IM in my MSN, AIM, or Yahoo accounts with ease, although I wish I had Trillian mobile to aggregate them.

It has a push-to-talk feature that I miss from my Nextel phone, but until I have other members of my family or friends with that feature it is shelf-ware for me for now.

My biggest fear I had was windows crashing in a call. Friends of mine have had this happen to them on the Treo and other devices. I bought this on October 5th (first day released in the US) and I am happy to say that this has not happened to me (yet).

New England Agile Bazaar October Meeting

I attended the Agile Bazaar meeting this month at Tufts University. This was a panel discussion format moderated by Ron Morsicato with an open format including a series of questions crafted by Ron that were geared towards adoption of agile practices. The panelists:

• Don Roby, Cyrus Innovation. A developer of highly usable solutions with a proven blend of Agile programming techniques, user-centered design, and deep insight derived from the firm's financial services experience.
• Brad Kain, Quoin, Inc. A builder of sophisticated applications for media, publishing, retail, finance, life sciences, and other industries.
• Giora Morein, BigVisible Solutions. A premiere provider of Agile enablement, training and software solutions.

There seemed to be consensus that agile practices are most popular in both small startup companies (especially in SaaS) and large financial services companies (like Fidelity Investments). The reason small startups are adopting embracing these practices is to address the rapid pace and reduction of technical debt. It is easier for these organizations to adopt agile because they can start fresh with this approach and don't have the cultural overhead to limit the pace of change. The larger financial services organizations are adopting agile to address the management of many small projects as opposed to most service and product teams. There was a discussion about hybrid methologies and the experience the panelists have had in the field. Most organizations to not practice a specific methology but a collection of practices, and then give the collection a name specific to their organization.

• Scrum or similar methods that describe the way in which projects are managed at the detail level, with a product owner.
• Some subset of XP development practices, especially TDD (test driven development) - a lot of resistence to peer programming - very few are using the notion of an on-site customer.
• Managing the business with CMMI or similar infrastucture framework is common with larger organizations, incuding phases and gates. There is evidence that CMMI is compatible with agile project management practices and certainly with XP coding practices.

Organizations looking to make the change to an agile lifecycle have common things they don't want to let go of. For example, building narrow funtionality (through all tiers) and delivering often, logging and tracking all defects found after a coding phase, constant communication within a team, and the dreaded exposure of peer programming. Architects of web services need to have a vision for the tiers needed for the service, but unlike a waterfall approach where you can complete a majority of a tier before moving on the the next, only to portion of each tier relevent to a specific feature in scope for the iteration is developed. The challenge of the change agent is to let them use familiar tools in a different way that moves them to changing the practices. For example, working with QE at the time of development and treating most defects loosely as part of the development exercise, and only escaping (after closing a story) defects can be logged and tracked as before.

Offshore, nearshore, and collocated teams were discussed. There is contention with many companies as they seek to reduce overhead by utilizing outsourced development services. Most of the companies in India, Eastern Europe, and Canada move people from project to project often, keeping their signature engineers on strategically high profile projects, especially to win new business. They tend to resist agile project managment approaches and have cultural challenges with sharing perceived bad news. Giora talked about a survey he conducted for a company with teams locally, India, and Canada, which had a series of projects together. This survey included both quatitative and qualitative questions about their experiences. 83% said the most successful projects included personnel spending time in the other locations, suggesting that face time really makes a difference in the success of a project. These respondents also said that they preferred to be in the same time zone. However, Giora mentioned that one of the most successful projects included members of the team in India spending time in the local project teams and having development in both time zones had the effect of 18 hour days. There was no evidence of increased productivity though. There was also talk about setting up your own team in India if you want to use agile practices. We did not seem to come to any conclusions on making what makes an outsourced project work with agile.

http://agilebazaar.org
http://groups.yahoo.com/group/AgileBazaar