Agile Testing with Lisa Crispin

I wrote a blog post on StickyMinds about my views on the importance of being nice. I particularly like some of the stories that commenters have shared about how they were able to get a good working relationship with difficult coworkers. Please share your stories!

It’s that time of year where we may have a chance for extra socializing and pleasant distractions from the short, dark (at least in this hemisphere) days. Here’s a bit of holiday cheer from me, my husband Bob Downing, our dogs Tango and Bruno, and our donkeys Ernest and Chester.

Happy Holidays!

Eric Willeke, Liz Keogh and Jean Tabaka have inspired me (and many others, I am sure) with the following, which they produced when they got together in Boulder last Friday.

I’m currently reading some draft chapters of Jurgen Appelo’s upcoming book. His work is affirming my belief that successful software projects are the result of getting good people and letting them do their best work. There are so many aspects to this – reading Jurgen’s work, I am starting to understand that people are the most complex component of any project! Everything hinges on our ability to innovate, to be creative. So, we must be a community of thinkers.

One item in this list urges us to embrace newcomers. I was welcomed into the agile community 9+ years ago. I work hard to pay that nurturing forward, so that others may experience gaining knowledge that in turn drives their creativity. And besides all that – why not always be nice to each other? I feel another blog post coming on.

A Community of Thinkers

I am a member of a community of thinkers.

I believe that communities exist as homes for professionals to learn, teach, and reflect on their work.

I challenge each community in the software industry to:

• reflect and honor the practitioners who make its existence possible;
• provide an excellent experience for its members;
• support the excellent experience its members provide for their clients and colleagues in all aspects of their professional interactions;
• exemplify, as a body, the professional and humane behavior of its members;
• engage and collaborate within and across communities through respectful exploration of diverse and divergent insights;
• embrace newcomers to the community openly and to celebrate ongoing journeys; and
• thrive on the sustained health of the community and its members through continual reflection and improvement.

I believe that leaders in each community have a responsibility to exhibit these behaviors, and that people who exhibit these behaviors will become leaders.

I am a member of a community of thinkers. If I should happen to be a catalyst more than others, I consider that a tribute to those who have inspired me.

”A Community of Thinkers” by Liz Keogh, Jean Tabaka and Eric Willeke is licensed under a Creative Commons Attribution-Share Alike 3.0 License. Please attribute to the distributor of your copy or derivative.

Beautiful Testing, with a wide variety of fascinating chapters contributed by leading software professionals, is available. Not only will it inspire you and your team to improve your testing and your product, but all royalties go to a good cause. I’m honored to have written a chapter. Please see the page on my site for more details and a free copy of my chapter – to whet your appetite for the rest of the book!

I love conferences – I get to meet so many smart practitioners and come home bursting with new ideas. I’m back from Agile Development Practices and I learned a ton. One thing that’s on my mind both from the conference and other reading I’ve done this week is the importance of a culture that enables and even promotes learning.

It’s easy to forget that failure is good – if you can fail fast enough. That’s one reason agile works so well – short iterations and work transparency let us fail fast, and everyone knows when we fail. But failure shouldn’t only be seen as negative. We learn a lot more when we fail.

I recently read a great post titled “Are you building a learning suppression system by Daryl Kulak from his upcoming book. Here’s an excerpt.

What happens in organizations is that people get punished for committing sins of commission but they do not get punished for sins of omission. This shapes the mind of a person into saying “No, we better not try that,” attempting to avoid the nasty results of commission errors while ignoring the problems of omission errors, which don’t seem to cause as much havoc.

We don’t want to suppress innovation. Mistakes should be tolerated, not punished. We want a learning organization so our teams can always adapt to change, and thrive.

More on Learning from Linda Rising

Linda gave an enlightening talk Wednesday at ADP about retrospectives, and she talked a lot about having time for thinking and learning. These are my notes on her talk, so I hope I haven’t misinterpreted anything she said. You can find more material direct from her on her site.

When confronted with practices such as pair programming or retrospectives, some managers have this reaction: “We don’t have time for thinking”.

But we aren’t trying to stop, analyze the root cause of all of our problems, and fix them all. We need to do little experiments to see if they solve some problem we’re having. If the solution works, we don’t even have to know why, we can just use it. Linda compared this to tuning a musical instrument – we can make tiny changes, and these give us lots of value.

Linda quoted Tom DeMarco’s book, Slack. You can’t learn or change if you have no slack. Linda cited research showing that not being able to express feelings and thoughts in the normal course of business weighs us down, slows us down. It sounded a lot like technical debt to me! We fret about things that aren’t going well, and if we don’t get a chance to experiment with ways to improve them, we become less and less productive, more and more unhappy.

Going back to that idea that we must tolerate failure, it’s important that we don’t criticize or blame our teammates. Every team member should feel safe to raise issues and propose ideas. Linda cited Norm Kerth’s prime directive for retrospectives: We must believe that everyone did the best job he or she could, given what was known at the time. Everyone used their skills and abilities, the available resources, to their best advantage, given the situation at hand. When we start from there, we can avoid pointing fingers, and simply try to learn: what’s going well? What’s not going so well? How can we improve? What still puzzles us?

Linda advises us to watch for stereotypes: “These people are just no good at…” She was talking in the context of retrospectives, but this is good advice for everyday work. We even stereotype ourselves. We have to believe, instead, that we can get better. Then we will.

Focus on experiments. Identify the next experiment, and ask the three questions about it at the next retrospective. Share knowledge. Look for patterns.

Time to Learn

As Linda noted, everyone says they want to learn, but few take the time to do so. Take some time to learn something new today. See what you can do to promote a learning culture within your organization. Show your manager the Google 20% rule, and the value that Google has derived from that! Lots of companies, big and small, have something similar: Wiki Wednesdays, Engineering Sprints, one hour per day devoted to professional growth. People on agile teams love learning, and it makes us happy. When we’re happy, and when we are always learning and experimenting, we can do our best work. To me, that’s what agile is all about.

I meet more and more teams, both within my company and externally, who are getting really good at agile testing. They’ve adopted a Whole Team approach, they write tests first, they collaborate and communicate well, they use feedback to improve. Agile teams are functioning much better overall than what I was seeing a couple of years ago.

One perennial issue – and this isn’t new with agile – is that testers and teams still struggle with how to design and write automated tests that have a good return on investment. That is, the tests are reliable, stable, maintainable, and provide good value for the amount of work that goes into them. We also all struggle with how much to automate, what to not automate, what to keep in our regression suite.

Janet and I devoted a section of our book to automation, and we presented principles, techniques and strategies that will help. We also include this in our Agile Testing course, and we both do conference tutorials on how to succeed with automation in agile projects. I think in addition to that, a lot of people need some practical hands-on experience. They need to be guided through an example of automating tests for a story, then they need to try it for themselves.

I’m developing an internal class for people at my company, and we also have ideas for workshops and other ways for teams who have solved some automation problems and developed good practices to share with other teams. I just mind mapped the ideas we have so far. At Adam Goucher’s suggestion, I’m posting my first draft here.

It’s my hope that I could extend this to a more generic class that would be appropriate for a conference tutorial. I’ve talked with some folks about collaborating on something like this and kind of making it open source, with several of us who could teach it, like the Watir tutorial that Bret Pettichord, Charley Baker and others have taught. But, others better than I have tried to teach hands-on courses like this and run into issues. What tools would we use that can easily be installed on all participants’ laptops? What if the participants have different skill levels? So, I don’t know if this is really possible. I’m interested in ideas, or if anyone has done this successfully.

Actually I’m sure this has been done successfully by people like Elfriede Dustin who have been specializing in test automation for years – I need to get her book out and re-read it. I’m not trying to steal anyone’s idea or say that this is a new original idea I thought up. I just see a gap in many teams’ ability to create the ‘right’ automated regression tests.

So, here is the mind map, you’ll have to open it in a new window (I’m sure there’s a better way to do this but I have to catch a plane), feedback welcomed.

Last month I had the great pleasure to participate in Agile Testing Days. I did a tutorial on using the Agile Testing Quadrants for test planning, and a keynote on 7 key success factors for agile testing success. More importantly, I got to attend a lot of great sessions by leading agile testing practitioners.

There were so many good sessions and I’ll only mention a few here. Mark your calendars for next year’s Agile testing Days, Oct. 4 – 6.

My slogan after this conference comes from Dave Evans (and from Mike Scott – not sure which one of them coined the phrase but they both talked about it): Respect the Tests. More about that in a minute.

Elisabeth Hendrickson always inspires me, and her keynote didn’t let me down. I’ve heard her definition of agile before, but it bears repeating: Delivering value in the form of releasable software at frequent, regular intervals at a sustainable pace, while adapting ot the changing needs of business. She quoted Mary Poppendieck as saying, if you want to go very fast, you have to be very disciplined. Sustainable pace means that we keep technical debt down so that the code doesn’t impede us.

Elisabeth presented her four sources of technical risk: Ambiguity, assumptions, dependencies and capacity, and her 7 key testing practices: TDD, ATDD, Exploratory Testing, Automated System Tests, Collective Test Ownership, CI, and one more that I think is automated unit tests (I can’t read my notes!)

Elisabeth also quoted Tobias Mayer, who refers to the Product Owner as the Voice of the What, and the dev team as the Tribe of the How. I like this, but I take issue with it a bit. The Poppendiecks have influenced me to experiment with having the dev team learn the business inside out, so the dev team itself can help the business prioritize and even think of features. This worked well for my team at ePlan, and I have to agree that funneling all the business requirements through the Product Owner has a lot of drawbacks.

What was new to me in this talk was something that several of us had talked about a night or two before, the idea of Collective Test Ownership. This ties in with the Whole Team Approach. Everyone on the team takes responsibility for quality, testing, and test automation.

Another great session, again from someone whose work I’m familiar with, was David Evans‘ talk. One big takeaway for me was this: If a test is worth writing, it’s worth automating, and it must always pass in the future. I often see teams that ignore failures in their CI tests because “We have to get these new stories out, this is the PO’s priority.” So the PO doesn’t care whether what she prioritized in previous sprints still works or not? When the CI breaks, the team should stop the line and get it green.

Dave pointed out that if all tests pass, you don’t need extra metrics. Defects are evidence of missing tests. So don’t call it a defect – call it a failing test, write the failing test and fix it right away. If the customer really doesn’t want it fixed now, write a story for later.

Dave recommend fixing time and quality, but don’t fix functional scope. If time is running out, thin the requirements. It’s not like leaving a wheel off or leaving out the engine. Respect the tests – any test worth writing is worth running all the time.

Dave presented his Agile Quality Framework. I won’t go into all the details, but one of my favorite parts is applying balloon patterns. An empty balloon is still a balloon, but if you don’t have the rubber outside, it’s hard to have the air and then add the rubber later. Do the rubber first – the CI, the test framework.

Mike Scott elaborated on the balloon pattern in his demo of Testify Wizard. In one hour, Mike went from zero to a full CI process, including the project two continuous builds, one running unit tests, and one running FitNesse tests. It even generated code coverage reports! I think he did this with both Java and .Net during the same session. He did this with a couple of pushes of the button,some minor configuration and a wave of his crutch.

This shows how easy it is to respect the tests. It’s not that hard to build a framework to integrate our code and run our automated tests so that we get quick feedback all the time.

I work for a mid-sized software company with around 27 development teams, mostly Scrum but also a few doing Lean and Kanban. We have a large and diverse product of HR-related software, I think there are around 6 million LOC, on three different platforms, including an ASP product and a .NET product. I’m fairly new, I started at the beginning of June, so I’m still learning about how everything gets done. We release four times a year and the software has won many prestigious awards. I’ve been impressed with this large-scale implementation of agile development. Just the speed of the daily Scrum of Scrums is amazing.

Many of the teams do a great job of automating their tests, and a few teams have their own continuous integration (CI) that runs multiple times a day. Other teams have automated tests, but must wait for the company-wide build to run, which is usually only once per day and sometimes not that often. They don’t get feedback fast enough, and that slows them down – a lot.

I’ve been asking why some teams struggle to automate tests, and why some teams don’t have their own CI. I’ve also asked my own team why most of our functional tests are automated through the GUI, rather than behind the GUI. We use FitNesse, but mainly to drive GUI tests with SWAT.

The answer I get a lot is “It’s the framework”. We have three architecture teams, and there is a framework (maybe there is more than one, I don’t fully understand it yet) that apparently makes it hard to test at the page or object layer. In addition, some teams’ code has tentacles into legacy code written in Delphi whose tests are written in WinRunner. Also, the current framework appears to not be fully supported with automated unit tests, it was developed a few years back. Programmers new to the framework seem to have a hard time understanding how to work with it. The framework clearly causes a lot of frustration. A new framework is under development, but it won’t be available for some time.

In order to understand the existing framework and find out whether the new one will facilitate testing behind the GUI, I was fortunate to get to meet with the leader of the architecture side of the development group. He gave me an overview of the new framework’s current design (which may change), aimed at making our product easily configurable by customers, using web services. It looks like this design allows testing under the GUI level.

Additionally, I learned that it is possible to test under the GUI in the current framework, and I met someone on another team who has automated many FitNesse tests at the object level. I plan to get her to show me how these tests work, and see if what she did would work for our code.

I haven’t worked at a company this large since the early 90s, and I’ve never worked in a company, agile or otherwise, that had “architecture teams” and “frameworks”. I get it that our product needs to look and work consistently across the board, so it does make some sense to centralize the design of the UI and perhaps other parts of the system. However, if the architecture is done by one team, what does that mean for programmers on other teams? Is the fun part of their job taken away? Do they get a chance to think for themselves and improve the product?

It also seems to me that there needs to be a lot of communication amongst the entire development community of our company, that all programmers should have some input to the architecture process. I’ve found that communication between teams in general is a bit lacking. Each team is so focused on delivering its part of the product, they don’t always feel they have time to go talk to other teams. So, for example, perhaps nobody on my team has talked to the person who did a lot of FitNesse functional tests.

Does your company have an architecture team, and/or a framework that everyone works with? Please comment and share your experiences. As a tester, I’d like to know what I can do to help my team deliver the highest possible quality when we have to work with a framework designed by someone else.

I’m doing a tutorial and a keynote at Agile Testing Days in Berlin. I think this is the first big agile testing conference to be held in Germany. Take a look at the program – it’s an awesome lineup of agilists. And since the most helpful part of conferences is always the informal connections you make, I’m happy to say that I know of a lot of interesting people planning to participate also.

Here are a couple of discount codes:

10% for the Tutorials: code SPEAKERSTUT010

20% for the conference: Code SPEAKERCON020

I hope to see you there!

If you’re in the Oslo area, don’t forget I am doing an Agile Testing course at Program Utvikling the following week.

There are a lot of creative, smart people on our team! Unfortunately I missed the discussion where these ideas come out, but we are trying some ways to organize such a big team. We will have “cells” or sub-teams, each one will take on a set of stories.

Today, each cell wrote high level acceptance tests together. Our cell started out doing BDD style tests on a FitNesse wiki. We used my desktop and the group in Weston VNC’d in and projected it on screen. But they couldn’t scroll up and down. And tests were taking too long to write. In our pomodoro retro, we decided to do high level test cases together as bullet points only, then have a pair flesh the tests out into BDD style and automate them.

We also decided to try Google docs. We kept using VNC and projecting the screen, but since the page is saved often, each person could also look at the doc on his own machine and scroll up and down. Both changes worked better. I had to split off for a long phone call with someone from an internal development team (that’s a whole ‘nother blog post), and when I came back they had finished the tests.

We are doing what I just learned is set-based engineering to decide whether we’d rather write our SWAT tests in C# and Visual Studio, or in FitNesse. We will do both this sprint and see which is better. My teammate Maykel started writing FitNesse fixtures that would allow using BDD tests to run SWAT from FitNesse, and I think he also had some way to make the tests run faster in FitNesse. One pair will automate the acceptance tests, while the other pair does a spike.

Thanks to the people who commented on yesterday’s post, I feel much better about having the testers get together occasionally. It’s true that we should have a testing community within our agile team, just as we have started one company-wide. I think we are on the right path with this team, despite it being larger than I would like.

I’ve worked on agile teams where programmers, testers, DBAs, sys admins and other roles all saw themselves as part of one developer team for so long, the idea of a separate test team gives me a chill. Yet even on experienced agile teams, there are still testers who see themselves as a “test team”.

I need to investigate more what they mean by this, I might be misinterpreting it. My new team is the combination of three teams, totalling 17 people (not counting the business folks). I think that’s unwieldy. Today, an experienced tester on my new team, who has worked on an agile team for some period of time (I’m not sure how long), proposed that the “test team” get together and discuss practices and strategy. I about blew a gasket, but that didn’t keep the meeting from happening (I’m always on phone and video with my team, so I can’t hep being in the meetings.)

The content of the meeting was reasonable enough. One topic was patterns of tests that we do, for example, for a numeric field you would always try certain test cases such as alpha and special characters, null, etc. Standard stuff.

The next topic was about automating tests on a page and end-to-end level, in addition to at the story level. This bugged me. If you tie tests to a page in the UI, and later stuff gets moved around so that what used to be on that one page is now on different pages, your tests will break. As far as automating end-to-end tests. we definitely have to be careful since the ROI can be bad on automating those.

My teammate Chris McMahon explained how our team up to now has been trying to group tests at the domain level. One FitNesse test contains all tests related to a rating scale, for example, until there are about 200 test steps and then we split it up into some reasonable sub-grouping. We’re testing at a feature level, no matter where we are in the UI.

The next topic was localization. Our app accommodates data in three languages. Someone proposed that we design our tests so that they could be run for any language. This sounds good to me, but how hard is it to do, and is it worth it? Is the localization already tested by the team that does the framework for it? I suggested we definitely needed programmer agreement for this topic, and everyone agreed.

I then brought up topics I’ve been wanting to raise, the non-functional testing. Our company has teams to do performance, scalability, reliability and security testing, but isn’t our team still responsible to make sure those happen in a timely manner? This discussion proved helpful for me, as I learned about the teams that do this testing, and how we can become involved and do a lot of it ourselves. I’m glad to get these activities out in the open and on our minds so they won’t be overlooked.

In the end I needn’t have been so spooked by the idea of testers meeting together. As it turned out, the programmers got into a design discussion with the database guy, and we were not involved in that (welll, I didn’t know about it until it was underway), so it was an opportunity for the testers to talk. But I don’t want that to happen too often.

I’m looking forward to starting work on stories. I think our test design will evolve as we write new code. We will pair on everything, so I hope that means the programmers and testers will remain engaged with each other.