Property-based or generative testing – insights from European Testing Conference

I had really good intentions to blog more about things I learned at European Testing Conference, because I took pretty good sketch notes for myself. But I exceeded my WIP pretty badly between ETC and Booster Conference, and then I broke my leg. I’m making time now to share what I learned from at least one more ETC session!

Property-based Testing

Romeu Moura explained property-based testing to us. The whole time, I kept thinking, “This sounds just like generative testing, which I learned from my former Pivotal Tracker teammate Ashton Kemmerling. The generative tests found such otherwise-impossible-to-pinpoint bugs with Pivotal Tracker.” I just did some research and found this post from Jason Steinhauser which confirmed my suspicions. That post also has this really good definition from Jessica Kerr:

Property-based tests make statements about the output of your code based on the input, and these statements are verified for many different possible inputs. – Jessica Kerr (@jessitron)

Let’s look at what I learned specifically from Romeo, because it added a lot to my knowledge. He used the bowling kata to illustrate a lot of his points, and I have to admit, I just don’t get the bowling kata completely. But I’m going to try to repeat what he said anyway.

Not just for “modern” code

Romeo emphasized that property-based testing works for legacy code and non-functional programming languages (I’ll say, because on Tracker we used JS). He began by talking about the need to know business rules and examples, and that tests should make the business rules clear.

With parameterized tests, you pass in the examples to a method with an assertion. There is a danger of positives that violate business rules, for example, an invalid bowling score like (3,9). You can’t have more than 7 as the second number in that score.

Property-based tests have valid ranges and generate random values within those ranges. They explain the behavior you are sure about and increase happiness. Our code knows more than almost anyone in the company. Property-based testing asks the code questions, based on our own assumptions.

There are several frameworks for property-based testing (I found some mentioned in this post). But property-based testing isn’t just the frameworks.

A property is something that is always true about an object. it’s ideas about your business. It is different than fuzzing, which is ignorant of your code. (However, you do fuzzing in property-based testing, my understanding is the difference is that the fuzzing is based on these properties that you know about your code, whereas pure fuzzing is totally ignorant of the code.)

Why use it?

The goal of property-based testing is resilience. Thinking back to Pivotal Tracker, we had a single-page JS app with a ton of business logic in the UI. It is designed for some infinite number of users to be able to update the same project tracking information concurrently. The page constantly re-renders to show each user the latest information. That code was written by experienced developers using leading practices such as TDD and pair programming. When you have sophisticated code, you get sophisticated bugs. We had a recurrent problem reported to customer support for months and none of us could reproduce it. A generative test found it within minutes. The developers included some generative tests in the unit test suite for these complicated areas.

Property-based testing doesn’t replace unit testing, it is a complement to it. We like to think we know our code, but if we really knew it perfectly, we’d never have bugs. I am surprised that I don’t hear more people talk about property-based or generative testing. Your team might want to check it out!