Most of us have more we’d like to do than the time to do it, even if we’re in an experienced agile team working at a sustainable pace. A couple of us that work on our iOS team wanted to try automating some UI regression tests. The programmers write their code using TDD with good test coverage, but UI regression testing was all done manually and tended to be a bit hit-or-miss. However, we were stretched thin on the testing side, and that idea didn’t get to the top of the priority list.
Luckily, a highly experienced tester, JoEllen Carter (@testingmojo), joined our team. Soon after that, when we were regression testing an iOS release, JoEllen found a showstopper regression failure. Automating regression tests through the UI to catch failures such as that more quickly became a more compelling idea.
Getting started
The developer who is also the iOS product owner championed the automation effort, and knew some testers in our company’s Toronto office who were successfully automating mobile UI tests. He set up a meeting so we could pick their brains. They recommended using Apple’s built-in test automation tools in Instruments, along with the tuneup.js library, and kindly sent us a book, examples of their own tests, and other helpful documentation.
The PO helped us install and build the iOS code, and spike some rudimentary automation with Instruments. We agreed on some simple scripts to try, and he put stories for those in the backlog. Since the main focus of our team is our web-based SaaS product, and we testers wear other hats including helping with customer support, it was hard to find time to devote to the iOS test automation. JoEllen suggested blocking out an hour every day to pair on it. This proved key to getting traction.
Pairing FTW
You know how it’s good to be the dumbest person in the room? This worked for me as I watched JoEllen fearlessly use the record function in Instruments as well as the logging feature to see how the iOS app pages were laid out and how to refer to the different elements. Every time we paired, we made good progress, starting with the simplest of scripts and incrementally adding to them. Demands on our time for more business-critical work sometimes meant we let the iOS automation effort slide, but we’ve experimented with trying a different time of day when distractions are fewer.
Obstacles to overcome
We had a few scripts written when the element names changed unexpectedly due to updates to support iOS 8. We were discouraged because we’d had no warning about it from the programmers. They work in a different office two timezones away, and though we have a daily standup and are on chat all day, we testers are usually focused on other products. We discussed this with the PO, and agreed it was time to get our test scripts into CI, so that if we weren’t warned about big changes, they’d at least be immediately obvious to everyone. We wrote more stories for the steps needed.
Some obstacles to making the UI automation scripts part of CI seemed tough. For example, we needed to figure out how to run the tests from the command line rather than from within Instruments. After one pairing session of Internet searching, reading doc, and trial-and-error, we nailed it! By “we” I mean JoEllen, with me contributing information from searches and a bit of shell command expertise. We also got help at one point with a Javascript question from a pair of programmers that work on our web-based app.
Overcoming my own weenieness
I have personally experienced a lot of fear and trepidation about the UI test automation effort. I don’t know Javascript, I’m a newbie at iOS (even as a user) and mobile app testing in general, and I always find test automation hard even though I also really enjoy doing it. But I finally forced myself to ‘drive’ and felt much better actually trying things. We have a plan for refactoring to make the scripts easier to run and maintain, and of course more tasks to do to get them into CI.
Lessons learned
I can’t advise you on the best techniques for iOS UI test automation, but I can share some general takeaways from this experience so far:
- Has anyone else in the company already succeeded with a similar effort? Ask them for help – they’re probably happy to share their experiences.
- Pairing brings much better chances for success than working alone, for so many reasons.
- Put time on the calendar every day to work on an effort like this, even if it’s only one hour (that is also good advice if you want to write a book!) Experiment with different times of day. Right after lunch has been a good time for us, before getting pulled back into the routine, and before afternoon meetings.
- Bring up obstacles during stand-ups, set up meetings to discuss them with people who can help.
- Write stories for automation tasks to make them visible.
I still have some misgivings about our automation effort. I’d like for the programmers to be more involved. After all, test automation is coding, and they are the people who code all day every day, rather than one hour or so a day at best. However, we have lots of support from the PO who is also a programmer, and we’re moving towards making this automation a part of the existing test automation already in the product’s CI.
If your team has a tough testing problem to solve, try pairing to spike a solution. Reflect and adapt as you go! Oh, and hiring excellent experienced testers like JoEllen is also a good idea, but don’t be trying to poach her away from our team.
