Twitter Linkedin Youtube Mastodon Rss
logo

Using the Agile Testing Quadrants

Someone on the agile-testing Yahoogroup mailing list posted a link to a blog post in which he proceeded to misuse, maul and maim the Agile Testing Quadrants. There is no way to put comments on that blog post to try to refute his claim that the quadrants are somehow a waterfall process. Since other people might misunderstand the purpose of the quadrants, I’d like to put a quick explanation here.

Agile Testing Quadrants

You might want to start with Brian Marick‘s original posts on his agile testing matrix, which we called the Quadrants and (with his permission) made the heart of our Agile Testing book.

The quadrant numbering system does NOT imply any order. You don’t work through the quadrants from 1 to 4, in a waterfall style. It’s just an arbitrary numbering so that, in our book and when we are talking about the quadrants, we can say “Q1” instead of “technology-facing tests that support the team”. 

Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. However, I have worked on projects where we started out with performance testing (which is in Q4) on a spike of the architecture, because that was the most important criterion for the feature. If your customers are uncertain about their requirements, you might even do a spike and start with exploratory testing (Q3).

Q3 and Q4 testing pretty much require that some code be written and deployable, but most teams iterate through the quadrants rapidly, working in small increments. Write a test for some small chunk of a feature, write the code, once the test is passing, perhaps automate more tests for it, do exploratory testing on it, do security or load testing on it, whatever, then add the next small chunk and go through the whole process again.

The quadrants are merely a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it. There are no hard and fast rules about what goes in what quadrant. Think through them as you do your release, theme, and iteration planning, so your whole team starts out by thinking about testing first.

Michael Huetterman adds “Outside-in, barrier-free, collaborative” to the middle of the quadrants, see his Agile Record article or his excellent book Agile ALM

 

Visit my Presentations page for some slide decks that contain more information on the quadrants, or check out our book. I’m always happy to talk about the quadrants, just send me an email!

85 comments on “Using the Agile Testing Quadrants”

  1. Interesting that someone actually took to apply the testing quadrants in a sequestial order, by design. We use the quadrants to organize and be aware of the types of testing that needs to be considered during our iterations. But I must say, in my current org, not always by choice, but due to resource constraints, some Q4 activities (primarily, load and performance) do get carried out by shared resources in a sequential fashion. On some projects, this gets done as packout activity. We are hoping to change this over time.

    Another use we have found for the agile testing quadrants is for assessing new hire skillsets. We have few managers responsible for hiring for testers, so the quadrants allow them to figure out the skill gaps to fill on their own teams. Candidates with experience in more quadrants, provides more testing versatility to teams.

  2. Thanks for telling us how you’re using the quadrants on your team! I love the idea to use them to help id gaps in skill sets, great idea.

  3. Great post, thanks. I agree that the matrix is a big help to coordinate activities; it can be used to structure test categories as well as a base for communication of aspects that are addressed in projects continuously, in iterative and incremental steps.

    Thanks for pointing to my resources that include an updated testing matrix. Comparing it to your excellent version it includes some updates. Beside editorial changes it contains different prioritizations, more emphasize acceptance tests (because of the book focus), and assigns both, TDD and BDD to the testing ecosystem. I’ve also stressed inherent properties by adding “Outside-in, barrier-free, collaborative” to the center of the quadrants. One reason for that is that I wanted to underline the dependencies all those quadrants to each other have, e.g. to show that to critique the product always also supports the team, and that all different activities should be glued together by processes and tools to result in a comprehensive outside-in approach (=> Agile ALM). Thus another dimension was necessary for presenting that.

    Lisa, I find it very essential that you’ve pointed to the fact that there are no orders or a specific sequence with these quadrants. Indeed, projects differ as well as processes and prios, and many different interpretations of the matrix may exist. That said, I think it is important to mention that it often does not lead to the best solution if you solely focus on one quadrant. I agree that in many cases projects start with Q2. But the backbone should include a strong net of unit tests (because they deliver feedback much faster than other test categories do and are essential for a good internal software quality).

    This may result in a concurrent creation of unit and acceptance tests, or with a start with Q1, depending on the context; but not in a waterfallish way: iterating over different test categories continuously is a good approach, but software development should better address both, testing and coding, thus instead of having a pseudo Agile approach (that would press activities into many mini-waterfalls), you should better focus on the importance of all test categories and having them in focus continuously, to gain from best results of your collaborative, barrier-free and silo-free development.

  4. Having seen someone else just misunderstand what this picture is trying to show, I’d like to offer another bit of advice.

    On the left-hand side where it says “supporting the team,” you may find it helpful to think “supporting development” instead. The left-hand side is testing that occurs prior to writing the code; the right-hand side is after writing the code.

  5. I like this presentation of agile testing. There are distinct areas of testing that, while related, are clearly independent. For example, usability is very different from security yet they need to work together in the finished product.

    Given these inter-relationships among test types, maybe there could be a central testing box where the parts come together. This may more accurately reflect the real world but would, admittedly, add complexity to the diagram.

    Anyway, food for thought. Thanks for sharing!

  6. This is a really interesting case. It had never occurred to me that anyone might put the interpretation on the quadrants that Frank did, but reading his piece I could understand how he got there, after a career being battered and bruised in traditional developments where everything is linear and sequential.

    I always thought that one of the benefits of the matrix as a presentation technique was that it made it clear the quadrants aren’t sequential. I was wrong! It obviously isn’t clear to everyone.

    Something that seems utterly clear to me, given my experience, attitudes and prejudices, may seem completely opaque to someone else whose perspective has been shaped differently.

    People can misunderstand just about anything. All communication is subject to confusion and distortion at the receiving end. Even if the requirement is written right it will probably be read wrong! It’s a lesson we need to keep remembering.

  7. Hello George, I’m not very sure what you refer to and I don’t want to judge any interpretation as wrong or better than a different one. But is testing really done either prior or after writing code? Just one example, how is unit testing done without having any code at all? To oppose “development” to “product” may lead to a silo-full, non-agile development process. From an editorial viewpoint it can be suboptimal to oppose a noun to a verb. But perhaps you just suggest to change the text from “supporting the team” to “supporting the team during development”, but this is probably a bit too long for a catchy illustration.

  8. I really like Michael’s additions to the middle of the quadrants, it emphasizes that they are not hard and fast delineations, but just different dimensions.

    I’m not sure why Brian Marick went with “support the team” instead of “support development”. Maybe people might misinterpret “development” as only meaning coding and programmers, as opposed to including testing as an integral part?

    It’s good to continue to evolve these. Michael Bolton had some interesting ideas for changing the terminology as well. What I hope the quadrants promote is the whole team approach to testing and quality.

    Thanks everyone for these great comments. I’m going to incorporate these ideas in to my version of the quadrants.

  9. Inspired by the book. This is how I use quadrants. Every user story is rated in an quadrants (removed the lines from the quadrants because the lines are not strict.)
    On the x axe (likelihood) and the y axe the impact to the user. (what is the impact for the user when it Fails). Each quadrants represent an set of advised test techniques with their corresponding documentation.
    These techniques must be tailored to the organization and project.
    See http://pascaldufour.nl for my blog on test specification and the blog on specification which also uses quadrants

  10. Let’s be clear on one thing that quadrants are not a silver bullet rather a guide to build your test strategy. They should not be looked at sequentially, testing happens in parallel depending on the readiness of the story/feature. They equally apply to traditional waterfall testing model as it does to Agile except it fits nicely with Mike Cohn’s Test Automation pyramid (http://bit.ly/7MW2Wf) with Q1 sitting at the bottom, Q3 at the top and Q2 taking up the middle slot. Q4 can be a standalone or merged with other quadrants to a degree. Depending on how you expand or squeeze the quadrants within the pyramid to gain efficiency within the team would affect delivery schedule.

    Quadrants are by no means complete in my opinion in the current agile context and require tweaking at the edges. There are other types of testing that can be encapsulated within the quadrants and some testing types moved around between quadrants. The legends within clouds in each corner are there for a purpose but again not exact representation of how the testing is carried out in today’s agile context for each type. For example exploratory testing is not completely manual anymore; there are tools to partly automate it. Security testing is not totally automated and tool dependent, lots of it is carried out manually to map functionality to its vulnerability for attack. Some security testing can be performed at API level and some through GUI interface, therefore, it can occupy Q2 and Q3 quadrants. Similarly Performance testing can be partly in Q2 and Q3 as well if API level tests can be constructed and crowd testing approach is used. There are calls to introduce UAT during a sprint which will change the landscape of quadrants to some extent.

    I agree Q2/Q3 are business facing testing but I am not comfortable with Q1/Q4 as totally technology facing testing because a large chuck of it is carried out manually.

    In the traditional sense, Q1 and Q2 (supporting development) have been developer’s remit whereas Q3/Q4 (product verification) tester but Agile context has changed all that. The roles of developers and testers are converging and the trend is unlikely to change in the near future. With BA, Scrum masters, QAs and users moving into testing space that would change the landscape of the quadrants. I think we should revisit the quadrants and update it appropriately to incorporate the current thinking. Don’t be surprised if it looks completely different.

    I am not going to expand more on what role quadrants play in testing which has been nicely put by Janet Gregory. She did an excellent job explaining it in her talk during her visit to London last August. The presentation is available through SkillsMatter here (http://bit.ly/qTsGq0).

  11. Thanks for your excellent insights, Mohinder. I like the idea that we can keep evolving the quadrants to help guide our testing as an integral part of software development, using the whole-team approach. I hope everyone who commented here will continue to write, blog and present their additions/changes to the quadrants. I plan to incorporate many of the ideas here.

  12. Hi, Michael,

    You asked “But is testing really done either prior or after writing code? Just one example, how is unit testing done without having any code at all?”

    Unit testing without having any code at all is exactly how Test Driven Development is done. Doing the same thing with business expectations is called Acceptance Test Driven Development or Executable Specifications. The actual work of implementing the tests and implementing the code being tested might proceed somewhat in parallel, but the tests should lead. In this way they can inform the programmer when the particular requirement has been met.

    Yes, this is a different way of thinking about testing. That’s the purpose of the Agile Testing Quadrants diagram–to illustrate this difference.

  13. Hi George, thanks for your wonderful thoughts. I agree with most of your points in your last comment, although those catchwords are heavily overloaded and can mean a lot different things, from different perspectives. I don’t agree with your point that “unit testing” is the same as “test driven development”, and it looks like that you take those two thinks as synonyms respectively you take the latter as an explanation for something that I’ve not asked for.

  14. I felt that I should update my comments with the new information I received during CITCON London conference yesterday and also something I missed from last comments. People who do continuous performance testing have started to implement it at unit level as part of their build. I listened to a lightning talk from a guy who introduced low level security testing in the CI build using selenium driver. If you revisit quadrants in the light of new information how people are engraining these test strategy it makes you think that the model introduced in 2003 is not applicable in many contexts today. The quadrants may be ok in some situations but I believe it will be challenged in its entirety and the argument will rumble on. My constructive advice would be that you throw away the quadrants and build your own that works for you and tell the whole world about it so that others can learn from it and help improve the model. My advice to those reading my comments would be that unlike me read Lisa and Janet’s Agile Testing book between the lines and pick the best bits from it that applies to your own environment. Quadrants concept is a good starting point but needs more work, therefore, you have to take the model in its current form and enhance it so it makes sense for you.

  15. No matter how you look at it, projects, requirements and contexts differ, and many other perspectives exist. Another view that was shortly raised in this thread already (“Support Programming”/”Critique Product”), is in “Implementing Lean Software Development” (Poppendieck&Poppendieck), pg. 199 et seq.)

  17. […] screen shows the status of the automated functional and/or Acceptance tests – see zone two of the testing quadrants! These tests are more integrated and (as a result) slow and work at the story/feature level and are […]

  26. @Lisa, I like the Agile Testing Quadrants in your book “Agile Testing”. On the page 100 of the book, it says “Some people use the term ‘acceptance tests’ to describe Quadrant 2 tests, …” I think the “acceptance test” here means tests of ATDD. And according to Markus Gartner’s book “ATDD by Example”, ATDD has many aliases, such as BDD, Specification by Example, Agile Acceptance Testing, Story Testing. But personally, I like the other name “Executable Specification” described in Gojko Adzic’s book “Specification by Example”. So I’d like to suggest that you could add ATDD or Executable Specification in Q2 because I think both of them are more popular than “story tests”.

  27. Hi Ben, it’s true that people don’t call them “story tests” so much anymore. Personally I like “business-facing tests that support the team”, but that’s a mouthful. I also like Gojko’s term SBE. BDD can also be used for unit tests, so that can be confusing. It’ll be interesting to see if any one term wins out. ATDD seems most popular now.

  30. Just chatted to my team. I do love the picture of the testing quadrants – it gives a high level feel to “all the kinds of testing that might need to happen”. Now as the lead tester, my testers may not do ALL of those pieces, but I ought to have an idea of who is doing what within our larger group.

  37. […] Testing products & services requires a diverse skill set. I believe software developers should strive to do as much of this as possible themselves. Books like Agile Testing and How Google Tests Software point the way. However these books also make it clear that several categories of high value testing and clarification are most effectively pursued by dedicated testing experts. Agile Testing is structured around Brian Marick’s testing quadrants: […]

  45. Always enjoy reading your posts Lisa. I’ve never thought the quadrants had a running order and great to have that backed up in your post. I’m just slowly introducing the quadrants at work. I’ll keep you updated how it’s going

  46. Thank you, Leigh! I look forward to hearing how your team uses the quadrants. They’re such a useful visual model for me.

  53. Hello Lisa, thanks for the post!
    I know there are no hard rules, but I’d like to know where would you put the “code review” activity (if you think that could be covered by this matrix)
    Thanks in advance!

  54. Hi Federico,
    Good question! Personally, I feel code reviews have the same goal and purpose as TDD. They get more eyes on the code, they’re a chance to discuss code design, make sure agreed-upon standards and practices are followed, produce the desired quality level. So I’d put them in Q1, technology-facing tests that support the team. What is your opinion?
    — Lisa

  55. Thanks for the reply Lisa!

    I’ve been discussing it with some colleges and I cannot get a side. I agree with you about what you stated, but Q1 has relation with “automated” cloud, right? so, how is this automated? Ok, we can think in the automation SonarQube adds to that, but as you guessed, I am talking about the code review done for the rest of the team, or at least for some other human than the one who coded. Then, I find this as an inconsistency, and that’s why I’m confused about that.
    On the other hand, if we base our analysis on the other version of the matrix, which says left side for the validation of what we know, and the right side for the tasks focused on looking for what we don’t know, it’s also difficult to get to an agreement on that. I mean, you can think that when you code you are checking things that you are expecting, that you know, and some others argue that you are looking something knew looking for new things that you don’t know.
    Sorry for the boundary testing made on the matrix, but I really like the thoughts around that 🙂
    Best regards!

    BTW, hope to see you next year and discuss this (or other stuff) in person 🙂

  56. Hi Federico,
    The great thing about the quadrants model is that it gets us thinking about all of this. The important thing is to get your team discussing all the types of testing activities you will need, make sure you have what you need to do it at the appropriate time. I’d just stick up a poster with quadrants drawn on it and use sticky notes to put up the different activities, and move them if they seem to belong elsewhere when you actually do them. It’s a tool, not a rule!

    And yes, it would be fun to discuss in person! I’ll be at European Testing Conference, TestBash Dublin, Agile Testing Days, and who knows what else…
    — Lisa

  61. Hi Lisa,

    Just about to read through AT & MAT as I do have difficulty in marrying Agile with sound testing processes. I’ve always considered the testing levels of;
    Unit Testing – done by Devs and ‘informal’ (Q1 in the quadrants)
    Functional/Regression testing – done by testers (and SDETs) and ‘formal’ (test fails, raise a bug etc) (Q2 in the quadrants)
    Integration testing – done by systems integrators/testers (Q2 as well)
    UAT testing – done by customers or agents (Q3)
    Non-Functional – done by SMEs (Q4)

    Current team state (using the Test Pyramid) that with loads of Unit tests, there can be much less Functional/regression tests. But that doesn’t sit well with me as it means Functional/regression testing has lower much coverage and there is less visibility of unit tests coverage for risk analysis etc. I have the mantra of “if you don’t test it you say ‘it works'” and if you have not got the evidence of testing then you cant say you’ve tested it. Is that a Waterfall attitude I have? If there are more Unit tests, is it ok to lower coverage of Functional/Regression testing (functional/regression testing)?

    Anyway, keeping fingers crossed that AT and MAT will help me make the transition.

    Mat

    PS. I’ve a couple of articles on this on LinkedIn, but primarily this one; https://www.linkedin.com/pulse/test-pyramid-focus-testing-mathieu-walker/?trackingId=BKXmnwyYKWjHbpLgXqnoCg%3D%3D Am I totally wrong in my thoughts here? Or am i getting the wrong end of stick?

  62. Hi Matt,
    I think you have a totally valid and important point.

    The Quadrants and the Pyramid are just models to help us think. The Quadrants help us think of all the testing activities we need and whether we have the people, skills and resources to do them at the right time. The test automation pyramid encourages us to think about ROI. If the unit tests don’t give us the confidence we need, then the automation we do may not fit that model. In More Agile Testing, we include several adaptations of both models by other people.

    I just listened to this podcast the other day, and was startled by their practice to only do automation from the user interface – though in their case that was the API, so not quite like going through a web UI. Still, they had good justification for testing exactly what the end users will use, and not worrying about potential bugs in the code that nobody ends up using.

    I’ll check out your LinkedIn article, thanks!

  68. […] 敏捷测试则提倡尽早测试、频繁测试，QA要从需求分析阶段就开始介入，QA参与从需求到发布的整个生命周期中各个阶段。在整个QA的过程中，会依据敏捷测试象限，在不同阶段采取不同的测试；还会根据测试金字塔的分层指导，加强自动化测试，制定有利于项目质量保证的测试策略，同时也会做探索性测试，尽量去发现更多不易发现的缺陷。敏捷QA的流程如下： […]

Leave a Reply

Your email address will not be published. Required fields are marked *

3 agile testing books
Agile Testing Books
Search
Categories
Archives

Recent Posts:

All Posts
logo
WORK WITH LISA
Twitter Linkedin Youtube Mastodon Rss

©2024 Agile Testing with Lisa Crispin. All rights reserved.

Copyright Notice