In the past couple months I blogged about the spike that JoEllen Carter (@TestingMojo) and I have been doing on automating UI smoke tests for our team’s iOS app: Pairing on a Mission and Continuing the Mission… and continually improving. Today we did a tech talk for our team reporting on what we’ve done so far and asking for help in choosing our next steps.
We used this mind map to guide our talk. We explained the goals for our spike: find an automation solution for consistent, repeatable UI smoke tests that would keep bad regressions out of the app store. We demoed one of our test scripts, showed how we had designed our tests, explained how discovering ‘search with predicate’ made our scripts much easier to write and maintain. We went over the pluses and minuses of our experience so far, pointing out the frustrating roadblocks we encountered, but also the pleasure of learning and working on something fun and cool.
We shared our thought that now is a good time to assess the value of the automated UI tests and how to move forward. Our iOS app is being completely redesigned, so the scripts we created for our spike will have to be re-done from scratch. We still have to solve the problems that keep us from putting our tests into our CI.
There are several options. We could try an external service provider. We could try other tool sets and frameworks. Should we abandon the UI automation idea and spend our time doing manual exploratory testing? Our iOS app already has about 6,000 unit tests and a number of integration tests that operate through the API. However, we have had bad regressions before that could only be found via the UI, so we know we need something.
We got some good ideas from the rest of the team. One was to ask the developer community within our company if they have any iOS UI automation experiences to share, since we know there are many other iOS projects. We posted a question on the development forum and have already had some good input.
This effort is inconclusive, so why am I blogging about this? Right before I started writing this, Jason Barile posted an interesting question on Twitter:
“…is your team really clear on what problems you’re trying to solve with automation and what success looks like?”
Our team has a long history of incredible success using test-driven development and automated regression tests at all levels from unit to UI. We have our share of “flaky tests”, but we get a good return on our automation investment. That doesn’t mean that automation is always the solution. We’ll have to figure things out.
Personally, I don’t like doing manual regression testing, so I hope we can find a way to run high-level UI automation smoke tests in our CI. Then we’d have more time for manual exploratory testing, and I’m learning that could be even more critical with mobile apps than other types. We shall keep experimenting and collaborating, and finding ways to shorten our feedback loop and ensure our users have a good experience with each new release of our app.
