ScalaCheck property-based testing

Note/Introduction: This article is a reprint of my original article, An introduction to ScalaCheck and property-based testing. The article is a short chapter from my book, Functional Programming, Simplified, and it begins after I have just talked about pure functions.

ScalaCheck and property-based testing

Once all of your functions work like algebraic equations, when you then look at an individual function it’s a simple step to wonder:

• Are there some general properties I can state about my function?
• If there are general properties, is there a way I can test those properties without having to write a bunch of little unit tests?

The answer to the first question is typically “Yes,” and ScalaCheck is a tool that lets you answer “yes” to the second question. In this lesson I’ll demonstrate how ScalaCheck works (and explain everything I just wrote).

Overview

Here’s the elevator pitch on ScalaCheck, from the book, ScalaCheck: The Definitive Guide:

ScalaCheck is a tool for testing Scala and Java programs, based on property specifications and automatic test data generation. The basic idea is that you define a property that specifies the behaviour of a method or some unit of code, and ScalaCheck checks that the property holds. All test data are generated automatically in a random fashion, so you don’t have to worry about any missed cases.

The rest of this lesson will show how ScalaCheck works, and how it compares to other testing tools, like JUnit and ScalaTest.

Introduction

Let’s say that you’ve just written this function:

def increaseRandomly(i: Int) = {
    val randomNum = getRandomIntFrom1To100()
    i + randomNum
}

Because it increments the value it's given by a random amount, it isn’t a pure function, but it is a relatively simple function. It adds a random number that varies from 1 to 100 to the number it’s given.

Two notes about this code:

• I show the source code for getRandomIntFrom1To100 later
• I use an impure function with a random number generator because it makes the tests that follow more interesting

Having written this function, you now want to test it. If you were writing unit tests you’d start to write a number of test cases to prove (or disprove) that the function works, but with ScalaCheck you start differently. The first thing you do is ask the question, “What are some general properties or observations I can make about how increaseRandomly works?”

When I think about the general behavior of this function, I know that the number given to it will always be increased by at least 1. Therefore, I can make this observation:

• Regardless of how many times it's called, the result of increaseRandomly should always be greater than the number given to it as an input parameter

It may be greater by 1, or it may be greater by any number up to 100, but in general, the result should always be greater; it should never be equal to or lower than the input value. ScalaCheck refers to this general observation as a “property” of the function.

Writing a test

The way you write this “general property test” with ScalaCheck looks like this:

property("increaseRandomly") = forAll { input: Int =>
    val result = MathUtils.increaseRandomly(input)
    result > input
}

I’ll explain that code in a moment, but once you get used to the syntax, you’ll find that you can read it as, “The property named increaseRandomly says that for all Int values, call the function increaseRandomly. Then for every Int value increaseRandomly is given, its result should be greater than its input.”

As you get more familiar with ScalaCheck you’ll read that more concisely as, “For any Int value, the result of increaseRandomly should always be greater than its input value.”

Explaining the property test

Here’s a breakdown of how that code works. First, the test begins with this line, where you give the property test a name:

property("increaseRandomly") = ...

This is similar to JUnit, but ScalaCheck refers to this as a “property” rather than a “test.”

Next, that line continues:

property("increaseRandomly") = forAll { input: Int =>

As you know from reading this book, this piece of code indicates the beginning of an anonymous function:

{ input: Int =>

This tells you that forAll is a case class or function that takes a block of code as an input value.

What you have no way of knowing at this time is that forAll is an important ingredient in the ScalaCheck recipe: most property tests begin with forAll. When you see forAll, you can read it as, “For all possible elements of the type shown, run the block of code that follows.”

forAll has some options that I’ll show in a little while, but with this specific example, just know that forAll sees the Int parameter input, and responds by generating random Int values that it will feed into your anonymous function, one at a time. By default, forAll generates 100 random Int values to test your function, with a special emphasis on edge conditions like 0, Int.MinValue, and Int.MaxValue.

The rest of the code in the anonymous function is standard Scala. The increaseRandomly function is called, and then forAll requires a Boolean value to be returned at the end of the block:

property("increaseRandomly") = forAll { input: Int =>
    val result = MathUtils.increaseRandomly(input)
    result > input
}

As I showed earlier in the book, you can infer from this code that forAll’s type signature looks something like this:

def forAll(codeBlock: => Boolean)

Running the test

The easiest way to run the test is to check my project out of Github, and then run it from your IDE or from the SBT command line. The source code is available at this Github URL:

• github.com/alvinj/FPScalaCheck

The code for this lesson is in these two files in that project:

• utils.MathUtils under src/main/scala
• utils.IncreaseRandomlySpec under src/test/scala

One note about the project code: As usual, SBT project dependencies are declared in the build.sbt file. I import both ScalaCheck and ScalaTest dependencies with these lines of code so I can show the differences between the two approaches in these lessons:

// all of these imports will only work under 'src/test/scala'
libraryDependencies ++= Seq(
    "org.scalacheck" %% "scalacheck" % "1.13.4" % "test",  //scalacheck
    "org.scalactic" %% "scalactic" % "3.0.1" % "test",     //scalatest
    "org.scalatest" %% "scalatest" % "3.0.1" % "test"      //scalatest
)

Running the test

Given that SBT configuration, when you run the test (such as with sbt test) you’ll see output that looks like this:

! AddOneSpec.increaseRandomly: Falsified after 13 passed tests.
> ARG_0: 2147483647
Found 1 failing properties.

Wait, what? Falsified? There’s an error in my function? That can’t be right ...

If you know your Int values well, you know that indeed there is a problem with my function. When you take the number 2147483647 — also known as Int.MaxValue — and add 1 to it, you see this result in the REPL:

scala> 2147483647 + 1
res0: Int = -2147483648

As shown, adding 1 to Int.MaxValue causes the result to roll over to Int.MinValue. This causes my property test to fail.

Note that there’s nothing wrong with this property specification:

property("increaseRandomly") = forAll { input: Int =>
    val result = MathUtils.increaseRandomly(input)
    result > input
}

This is the correct way to use ScalaCheck to state, “When increaseRandomly is given any Int, the value it returns should be greater than the value it’s given.” The problem is with the increaseRandomly function, specifically what happens when it adds any positive number to Int.MaxValue.

Looking at how a ScalaCheck property test works

I’ll fix increaseRandomly shortly, but before I do that, it will help to see what ScalaCheck just did to “falsify” my property test. You can learn more about ScalaCheck by adding a print statement to the property:

property("increaseRandomly") = forAll { input: Int =>
    println(s"input = $input")
    val result = MathUtils.increaseRandomly(input)
    result > input
}

When you add that print statement and run the test again, you’ll see lines of output that look like this:

input = -2147483648
input = 2147483647
input = 1073741823
input = -1073741823
input = 536870911
input = -536870911
input = 268435455
input = -268435455
...
...
...
! AddOneSpec.increaseRandomly: Falsified after 1 passed tests.
> ARG_0: 2147483647
Found 1 failing properties.

The input values are generated randomly, so you’ll see different values each time you run a test, but the general process is that ScalaCheck will run up to 100 tests against your function, trying to “falsify” the property you stated. It will do this whether you’re testing against Ints, Strings, or any other data type you specify, including your own custom data types.

What “falsify” means (and does)

“Falsifying” a property is similar to a unit test: If a ScalaCheck property test returns false, the test is considered to be failed. More accurately, it means that the property you stated about your function has been proven false, or wrong.

ScalaCheck: The Definitive Guide, describes how property tests work like this:

“When ScalaCheck tests a property created with the forAll method, it tries to falsify it by assigning different values to the parameters of the provided function, and evaluating the boolean result. If it can’t locate a set of arguments that makes the property false, then ScalaCheck will regard the property as passed.”

Test case simplification

ScalaCheck takes this even further with a feature known as test case simplification. ScalaCheck: The Definitive Guide describes this feature as follows:

“Test case simplification is a powerful feature of ScalaCheck. It is enabled by the fact that properties are abstract, and ScalaCheck therefore has control over the test data that is used. As soon as ScalaCheck finds a set of arguments that makes a property false, it tries to simplify those arguments. For example, if a property takes a list of integers as its parameter, then ScalaCheck will first generate many different integer lists and feed them to the property. If it stumbles across a list that makes the property false, ScalaCheck will test the property with smaller and smaller variants of that list, as long as the property still fails. Then ScalaCheck prints both the smallest list that still causes property failure, and the original list it stumbled across. By default, the first generated parameter is called ARG_0.”

The book continues:

“In the end, the smallest possible test case that makes the property false will be presented along with the the original arguments that caused the initial failure.”

This process doesn’t work in all cases, but when it works it’s a nice way to cut to the root of the problem.

Fixing increaseRandomly

The problem with increaseRandomly is that when it’s given Int.MaxValue as an input parameter, the number it returns flips over to being a negative value:

scala> Int.MaxValue + 1
res0: Int = -2147483648

One way to fix the problem is to force the calculation to take place as a Long rather than an Int. I show how to do this in the REPL:

scala> Int.MaxValue + 1.toLong
res1: Long = 2147483648

To implement the solution, add that change to the body of increaseRandomly, and then declare that it returns a Long value:

def increaseRandomly(i: Int): Long = {
    val randomNum = getRandomIntFrom1To100()
    i + randomNum.toLong
}

When I make that change, remove the println statement from my property test, and then run the test, I see this output:

+ AddOneSpec.increaseRandomly: OK, passed 100 tests.

This tells me that increaseRandomly passed 100 tests that were thrown at it by the ScalaCheck framework. I can now feel comfortable that the general property I stated — increaseRandomly’s result must always be greater than its input value — is correct.

ScalaCheck also works with OOP

As you saw with this example, ScalaCheck not only works with pure functions, but with impure methods as well. It also works with OOP code. In fact, there isn’t anything specific to ScalaCheck and FP, with one exception:

• It’s a lot easier to test FP functions than it is to test OOP methods, because FP functions don’t deal with hidden state

As I wrote in the lesson about the benefits of pure functions, this attribute alone makes pure functions easier to test in general, and easier to test in ScalaCheck specifically. With OOP, you often need to set up state before testing a method, but with FP functions there is significantly less setup work: output depends only on input.

ScalaCheck concepts

Although this was a simple example, it demonstrated many of the most important things to know about ScalaCheck:

• You use it to test general properties of your functions
• Therefore, you don’t write low-level unit tests
• By default, each function is tested 100 times (100 times if all tests succeed; less than that if a test is falsified)
• The tests are run with randomly-generated data
• What I haven’t discussed yet is that the test data is created with generators
• You can use built-in generators, as I did in this example, letting ScalaCheck generate random integers; you can also write your own generators, which you’ll need to do for your custom data types

I’ll discuss generators more in the next lesson.

Other concepts are similar to JUnit. For instance, you can create many property tests in one file, and you can also put tests in many different files.

Benefits of property-based testing (from the ScalaCheck book)

ScalaCheck: The Definitive Guide, lists the following benefits of using ScalaCheck as compared to other testing frameworks:

• Test coverage can increase: Because test cases are generated randomly, many more tests are thrown at your functions than you’ll get with static unit tests
• Specification completeness: Because you define exactly how your function should work under all conditions, it’s similar to writing a test specification
• Maintenance: Because a single property often compares to many individual unit tests, code size and repetition decreases, and refactoring is easier
• Test readability: Again, it can take many unit tests to compare to a single property test, which at the very least reduces the amount of code you need to read
• Test case simplification: I discussed this in the “What ‘falsify’ means (and does)” lesson, but in short, ScalaCheck will try to find the smallest possible test case that makes a property test false

Disadvantages of property-based testing

The major disadvantages of ScalaCheck compared to traditional unit tests are:

• Because it runs 100 tests for each property, and generates random data for each test, it’s arguably 100 times slower than a unit test
• While sometimes it can be easy to state a property of a function in words, it can occasionally be difficult to implement that property in Scala code
• When you know that one or more specific fringe conditions can cause a problem with a function, you may want the comfort of ensuring those tests are run with static ScalaTest (or JUnit) unit tests

In regards to the first point, it’s important to add this caveat: a single property test is often the equivalent of many individual unit tests.

Where ScalaCheck works well

When I first learned about ScalaCheck, I wondered if it could replace unit tests completely. The reality I’ve found is that ScalaCheck works well for some cases, and unit tests works well for others.

The example I showed demonstrates a case where ScalaCheck works well:

• You can clearly state one or more general properties about how a function works in English (or whatever your preferred spoken language is).
• You can easily express those same properties in Scala code. That is, it doesn’t feel like a great deal of work (or a hack) to create the necessary generators and property tests.

Where ScalaCheck doesn’t work as well

I haven’t yet found a simple, “OMG, don’t even try to use ScalaCheck for these tests!” rule of thumb, but I can state where ScalaCheck doesn’t work as well by writing the opposite of the previous rules. Specifically, don’t use ScalaCheck where:

• You can’t clearly state in English the general properties about how a function works. (Hopefully this will be very rare now that you write pure functions.)
• You can’t easily write property tests in Scala to express those properties, i.e., it feels like a tremendous workaround or hack to create the necessary generators and tests.

Bonus: A ScalaCheck thought exercise

Before I go, here’s a “thought exercise” about ScalaCheck and property-based testing that’s related to a Haskell function I read about recently.

The short story is that Haskell has a function named nub, and its purpose is to reduce duplicate elements in a list so that the list will contain only unique elements. In Scala it’s roughly equivalent to calling toSet on a list and then converting that result back to a list, like this:

scala> val x = List(1,1,2,2,3)
x: List[Int] = List(1, 1, 2, 2, 3)

scala> val y = x.toSet.toList
y: List[Int] = List(1, 2, 3)

So the questions I was thinking about related to ScalaCheck and property-based testing are:

• What are the “properties” of nub?
• How would I write property-based tests for nub?

I encourage you to look away from this blog post now and ask yourself those same questions, and write down your answers to those questions before reading on.

My thoughts

To be honest, I haven’t put much work into this, but these are the first properties of nub that came to mind:

• A list returned by nub must have no duplicate elements.
• The returned list must contain one and only one copy of each element that was in the original list. This is different than the first property in that you verify that every element that was in the original list is still in the new list.
• The length of the returned list must be less-than or equal to the length of the original list (newList.length <= originalList.length); the list can’t grow.

I think that accurately describes the properties of nub. The question now is, “If I was writing a Scala nub function, how would I write property-based tests for nub?” I’m not going to answer that question here, but I encourage you to write down (or at least think about) how you’d actually write property tests for nub. I will say that I think that creating ScalaCheck generators for nub will be easy ... the rest I leave as an exercise for the reader.

Yours in QA Analysis and Testing,
Alvin Alexander
Currently in Boulder, Colorado