Last May, we were asked to enlarge our small team by hiring another tester and another programmer. We’ve made a few hiring decisions in the past which turned out to be mistakes, so we decided to enhance our interview process. After a lot of brainstorming and research, we came up with some ideas to try. Today we got to use our new process for the first time when we interviewed a tester candidate.
We’ve always done group interviews, where all team members are in the room, and we all ask questions, focusing on open-ended, behavioral style questions (see my article about hiring agile testers for examples). We do this for about 45 minutes, also allowing time for the candidate to ask us questions. Then, we hook up a laptop to the projector, navigate to an interesting page in our UI, and ask the candidate to do manual exploratory testing on it, explaining to us what she’s doing and why as she tests. We also asked the candidate to come up with a simple SQL query that involves joining two tables, and for simple Unix commands (“How do you navigate to a directory, how do you see what files are in the directory).
When we discussed what’s important to us in a tester candidate, some criteria came out that we couldn’t evaluate well with our existing process. We need a tester who has some programming experience (though she doesn’t have to be a programmer), who can maintain her own test environments, who knows how to work with customers in an agile setting to flesh a user story out with examples and requirements. Here are the activities we added with those goals in mind.
We had someone play the customer role, describe a user story in the “As a… I want… so that…” format, and asked the candidate to talk with the customer and get requirements for the story. The candidate today asked some great questions, covering not only functional requirements, but other considerations such as response time and load. In fact, I was rather awestruck at how quickly the candidate fired off questions and thought of many different aspects of quality. We stopped him after about 5 minutes because we could tell he knew how to ask good questions.
Then we asked the candidate what tests he thought he might automate for that story. He started with the happy path, then came up with good boundary and edge cases for regression. Again, we stopped him after 5 minutes.
The next part of the interview was the one we were most nervous about. We had prepared a straightforward automated GUI test script in the tool we use, Canoo WebTest. We walked the candidate through the actual GUI to show what we were testing, and walked through the test. We had included in the test one module and one variable, to show that the tool supports these concepts. The rest of the script was all hard coded and included a lot of duplication, which we hoped was obvious. We asked the candidate how he might refactor the test to make it easier to maintain. He immediately grasped that the tool allowed extracting duplication into modules, and had good insight into where variables would work better than hard-coded values. He even came up with an idea to drive the test with different input values. We felt comfortable that he has a good understanding of basic design patterns and principles, such as Don’t Repeat Yourself.
Finally, I paired with the candidate to deploy a test script. Our plan was to have him read the instructions on our wiki for this, and try to do it himself, but we were short of time, so I talked him through it. First we showed him our Jenkins build UI, and the terminal window for the server where we would deploy. We asked how he might get the file to deploy, and he suggested FTP, which was a good answer. We actually copy the link for the file in the browser and use wget, so we talked him through that. It was obvious that he was well-versed in Unix commands, as he easily navigated to the right directory, renamed the file as we requested, even set his editor to vi so he could edit previous commands. We had no doubt that he is capable of learning fast, and can maintain his own test environments.
At the end of the two-hour interview, our candidate commented (un-asked!) on how much he liked our interview process. He liked the group setting, and he enjoyed the various activities we had him do. It was awesome to get this unexpected feedback.
We did our own little retrospective later, and felt we should have prepared better – we wasted some interview time in setting up the laptop and projector. We also need to watch the time better, and stick more to our planned agenda in terms of how much time to spend on each part of the interview. We all rated the candidate on some various aspects, such as his technical skills, his requirements-gathering skills, whether he seems like a good fit. He got a big thumbs-up from everyone, so I hope we’ll be able to hire him!
I got long-winded on this, sorry. If you have questions on any of this, feel free to email me. I have a lot of resources that we gathered in our effort to develop a better interview process.
Appendix: Here is the job ad we ended up with, thanks to Elisabeth Hendrickson‘s help.
Established Software Company with Startup Energy Seeks Software Tester
We provide internet-based 401(k) services to small companies. We’re part of Paychex, an established, solid, stable company, but we still have the close-knit feel of a small startup.
We’re looking for a few good people to round out our team.
What’s in it for you:
· Work with a dials-to-11 agile team. Our developers do test-driven development, refactoring, continuous integration, acceptance test-driven development. We value interactions over processes and tools, and working software over comprehensive documentation. And we work at a sustainable pace. Lisa Crispin is a tester on the team. If you’ve ever heard her talk about the great team she works with in any of her conference presentations, this is that team.
· Commitment to ongoing professional development. We are committed to the ongoing professional development of our team members. This is a great opportunity for anyone who values learning. We send our people to conferences and training, and support learning at work.
· Dynamic and fun work environment. We recognize that great software comes from happy people. We have a great team environment that fosters a sense of fun and joy in our work.
We are more interested in your passion for delivering value to customers than in your formal qualifications. However, to become a software tester at ePlan Services, you will need:
· Experience with various software testing techniques.
· Programming or scripting skill in at least one languagesuch as Java, C, shell script, or Ruby.
· Some experience with databases, data modeling, and SQL.
Interested? Please send your resume with a short cover letter telling us why you think this position is a good fit for you to:
9 comments on “Improving our interview process”
That’s an *awesome* interview process. Not only was it a good way to get a feel for the candidate’s thought process and technical skills, but it also sounds like you got a solid feeling for his cultural fit – and, cultural fit seems to get lost during the interview process with so many companies. I hope you can bring him on, too!!
thanks, we will be extending an offer, fingers crossed!
very interesting to read! your process sounds very thorough but also exhausting (but it couldn’t be more exhausting than some of the interviews I went through – 3 different interviews in 3 days).
I hope you found your man, good luck!
We made similar changes to interview tactics after some hiring surprises.
I found that I was often letting candidates in interviews gloss over their answers with generics. I think now in retrospect I simply wanted to find people to hire and thus wanted to believe the best from their answers.
But it became clear that some coding skills were not as sharp as I had thought, or their analytical skills or test skills were lacking.
I changed our 2nd interview to be much more technical (fitting to our method of testing), giving the candidate a very simple program to “test”.
Prior to the exposure to the interview challenge I talked to the candidate about what we think are important skills of a tester in our org, (i.e asking questions, investigative, challenging poor design, engage in development process etc). I talked through those things as to give an indication on what things we would be rating them on.
After being exposed to the app under test, candidates were encouraged to identify and code up their automated tests. This meant they could no longer get away with answers like, “Oh I would use some test data that does xxx”, or “I’ll check that the app gives me Y when I do X”. They actually had to show me how they would do it. It also meant they had to meet that challenge of “how can a computer test like I did as a person”
I took on the role of a developer and would answer questions about the app under test and also support the tester with test tools. I knew in advance they would probably need to be able to read in a log file, they would probably ask for some white box inspection mechanisms and be able put the app into some particular states from their test, so all those things were prepared in advance and I would supply code snippets if asked.
Sometimes there was gentle pushing in the certain direction to allow them to take off if stalled, but the process did a much better job at finding suitable people for our test team than our previous interview exercises.
All our current test team also went through the exercise and we had excellent feed back too. One of our testers even said he didn’t realize how much he was allowed to ask from the developers and this test taught him he could ask for a lot more support and push back on things he thought were truly broken.
I love that you showed the tester candidates that they could collaborate with a developer and work together. It sounds like you found a way to really see how each person works. Did anyone feel intimidated by the challenge? Or were you able to keep a relaxed atmosphere so they could think about what they were doing?
Lisa, do you have any post where you explain why do you use Canoo? What´s the main advantage to selenium?
Hi Amanda,
If you go to the ‘articles’ page on my site, there’s a link to a pdf of a review of Canoo WebTest (a “Tool Look” published in Methods and Tools) that I did a few years ago.
Selenium didn’t even exist when we started using Canoo WebTest. We chose it because it uses xml for specifying tests, and is run via ant, which made it super-easy to integrate into our continuous build process. The programmers on my team would rather code tests in Java, but XML is not too bad. We’ve had a great ROI on our WebTest scripts, we write straightforward tests with very little logic (it’s possible to put a lot of logic in the tests using Groovy or another scripting language, but we’ve preferred simplicity), we are able to modularize them for maintainability, and the scripts have kept many a regression bug at the GUI level from slipping into production. WebTest doesn’t do a terrific job with fancy GUI stuff done with dojos, though it handles Javascript ok. It is based on HtmlUnit, and doesn’t actually drive a browser, just uses HTTP and simulates whatever browser/version you want. That is ok for us because we keep our GUI thin, and don’t have a lot of browser compatibility issues.
We also use Watir for GUI testing, mainly to help with exploratory testing, those scripts have a lot of logic and “smartness”. We also use the Watir tests to verify releases to staging and production environments.
We did try Selenium early on in its history, but it didn’t work well then. We may start using it in the future, if it starts handling tests of code with dojos well, if the programmers can build us testers a framework so that we can specify tests without having to code Java.
Our GUI tests are only a small part of our regression automation – we automate as much as possible at unit and API levels.
Lisa,
Have you read _Agile Hiring_? It’s written by someone I know and respect in the biz, and I’m curious as to your thoughts on the book, or how it gels with your experiences.
Excellent blog posts!
Hi Charles,
I *have* read Agile Hiring, by Sean Landis, and somehow I had completely spaced it out of my head. I used to recommend it all the time but somehow it fell off my radar. Thanks for the reminder! I need to review it too, I could have sworn that I had.
— Lisa