“The Fear Factor in Software Testing: Recognize it, address it, transform it” – this is how Suzanne Kraaij entitled her MoTaCon talk. Another way to look at it, for me, is “fear-based testing vs. risk-based testing”. This talk was an eye-opener for me! I think many of us experience fear-based testing without realizing it. Certainly I never thought of this common situation from Suzanne’s perspective, though after watching her talk, it seems obvious!
Insights from Suzanne’s talk
Most of us in the software business have experience something like this: A new software product change is released, an important customer complains, or customer support receives an avalanche of tickets. The news gets out via social media. The business and software development leaders want action. “Who’s fault is this? Test all the things!”
Suzanne pointed out that this reactive anxiety, distrust, and constant need for reassurance brings undesired outcomes. Even if a problem affects only a subset of customers, the backlash can be big. Attempts to test everything leads to more missed problems. The result is exhaustion and burnout. Critical testing activities such as exploratory testing get skipped with a focus on excessive automated regression testing.
Suzanne had excellent suggestions for ways to avoid this trap. We have to start by identifying “what we are afraid might happen”, and make choices based on risk and priorities. And we need to make those choices and reasons for them visible to everyone.
We have to decide whether we want coverage, or confidence. There are plenty of ways to handle risk and gain confidence besides automated testing. Just a few examples: monitoring, alerting, observability, release strategies such as feature toggles or graduated rollouts.
When you resist the fear factor: an example from my past
I was reminded of an experience a team of mine had which could have descended into the fear trap. As we tried to deploy to production more frequently, we saw a huge rise in the number of serious production problems. Long story short, through retrospectives, the team realized that the frequent prod deploys sucked up all of the time of the 30-person organization’s two testers. We had no time to do exploratory testing, which was our only chance to catch the type of bugs going out undetected.
Luckily, our managers understood the value of exploratory testing. We testers were asked to start pairing with developers (who already practiced pair programming) as they coded test-first. We organized exploratory testing workshops for the teams, and provided resources like testing heuristics checklists. Exploratory testing skills were added to developers’ skills matrix for their career advancement.
As they learned, the devs were able to take on exploratory testing at the user story level. Charters for feature-level testing were added to our backlog and anyone on the team could pick those up as features were completed. Production problems quickly declined, and we were able to reliably deploy twice a week. It was a giant step towards embracing continuous delivery, which was our goal.
Looking back, I think a lot of other automation-focused software organizations would have panicked and started adding more automated regression tests. And indeed, we did revisit those later, and found they needed an overhaul. But that wouldn’t have addressed the issue fast enough. As Suzanne advised, making problems visible, reflecting on them, and trying experiments to address them leads to way better outcomes than fear and blame.
Watch Suzanne’s talk if you can!
If you are a member of Ministry of Testing’s MoTaVerse, you can watch this talk on the platform. (And if you aren’t, consider joining if you can, or see if your employer will take advantages of the many resources and opportunities that come with it).
