Overcoming test automation challenges

Janet Gregory and I enjoyed participating in the Quality in Agile conference in Vancouver April 20-21. We paired on a keynote: “Do testers need to code… to be useful?” Our opinion in a nutshell: testers need technical awareness to collaborate effectively with all their team members, but our software delivery teams already should have expert coders!

Even if they don’t write code, testers need to participate in automating regression tests and other useful automation, in collaboration with programmers, business stakeholders and others. Janet and I facilitated an all-day workshop on advanced topics in agile testing, with a focus on automation.

Challenges around automation

After some introductory slides, the 15 workshop participants self-organized into three smaller groups, choosing to sit with people that had similar goals for the day, or who had experience related to their goals. Each person listed their team’s impediments to automation, one per sticky note, and we grouped these on a big wall chart.

Next, everyone dot voted on the topics they wanted to tackle during the workshop. The top three vote-getters were:

• Culture and responsibility – whose job is it to automate?
• Lack of time for automation activities
• Things that make tests hard to automate, such as complexity

(Note, you can find higher resolution photos of all the session wall charts, including those not on this post.)

Formulating the problem and brainstorming ideas to overcome it

Each group was tasked with writing their own problem statement for the culture and responsibility topic. It is challenging to write a good problem statement! You can see an example at the bottom of the mind map at left. Once the problem was defined, everyone picked up a Sharpie and each team mind mapped on their big piece of easel pad paper.

One group focused on a lack of shared vision and investment in automation at the company level. Another saw a lack of education on both sides.

I thought it was interesting that the third group exploring culture and responsibility mentioned doing social activities together, and honing soft and tech skills including being respectful of each other.

For topic #2, after each group wrote their problem statement around the lack of time for automation, we tried a different brainstorming technique: Brainwriting. Each person wrote their ideas for dealing with complexity and other things that make automation difficult on a plain piece of paper. Every three minutes, they passed their paper to the group member to their right. They read what was written already, then wrote more ideas. This continued until each person within the group had written on each paper. Most people agreed that reading other peoples’ ideas jogged new ones for themselves. This technique lets people who might not be comfortable coming forward to draw on a mind map or say their ideas aloud contribute equally.

For topic #3 (sample problem statement to the left), we did “brainwriting with a twist”, an idea of Janet’s. Each team started by drawing, mindmapping or brainwriting ideas on a big flip chart page. After 10 minutes, each group moved to the next group’s flip chart, read the problem statement and ideas, and added their own. Some specific ways to design better automation code came out of this, as well as ideas for better tester-coder collaboration and ways to make these problems more visible.

Designing experiments

Of course, there is more to problem solving than brainstorming ideas. Janet presented a model of Esther Derby’s (left): define the problem and desired outcome, understand the context and requirements around potential solutions, design experiments, try them and evaluate the results. Each team spent time coming up with experiments they will try when back with their own teams. We hope that participants will report back to us on how their experiments went!

Got automation challenges – or any challenges related to quality and testing, for that matter? Get your team together, try out some brainstorming techniques, make it comfortable and safe for each person to contribute their ideas. Identify the biggest problem, brainstorm a couple of experiments to try to make that problem smaller, and use your retrospectives to evaluate the results. Keep experimenting, inspecting and adapting. Over time, your problems will be smaller and your successes bigger. But remember to celebrate even the small successes!