In a previous post I talked about unit testing for speed, and in particular testing for big-O scalability. The next step is catching cases where you’ve changed not the scalability, but the direct efficiency of your code.

If your first thought is “how this is different from running benchmarks?”, well, good point! An excellent starting point for performance is implementing a benchmark that runs automatically in CI, on every single pull request. If you haven’t got that, you probably want to go do that first.

Once you have implemented CI benchmarks, they will typically run when you submit a pull request or the equivalent. And if you’re doing performance work, that’s hopefully just a formality, as you likely have been benchmarking your code locally as you work.

But what happens when you or a colleague are working on features or bugfixes, and accidentally modify a performance-critical code path? You make changes, run the tests locally, run a linter, open a pull request… and now the benchmark runs, and tells you that your code has made things slower. This is annoying, because now you have to go back and figure out which specific change was the cause.

So what you really want is to get some sense of whether performance changed much earlier in the process, giving you immediate feedback when you’re running tests locally. Since a reliable benchmark environment is hard, switching to a test might allow for an early warning.

But again, how is a test different than a benchmark?

• A benchmark tells you how speed has changed: is your code slower or faster?
• A test gives you less information; it can either pass or fail, and that’s it.

On the one hand, less information is less information. On the other hand, this gives us more freedom in how we implement the test.

Instead of trying to figure out if our code got slower, we’re going to implement a test that tells us if performance changed. Or at least, probably changed. It may be faster, it may be slower, but if performance changes, our test will fail.

That means we will get some amount of false positives, when something changed but it didn’t affect performance meaningfully, or even made things faster. But that’s OK, this isn’t a benchmark, it’s just there as a warning: you did something to the code that might have affected performance.

If you change your code, the number of CPU instructions it takes to run your code will (almost always) change. Does that change mean your code is slower? Not necessarily. But it does mean your code’s speed might have changed, and that is the information we want.

There’s two ways you can get a CPU instruction count. One approach is to use Valgrind’s Cachegrind/Callgrind tools, that use a simulated CPU so you get very consistent instruction count numbers across runs and across computers. It’s possible to get this to run on only part of a running program, i.e. a test, but the Python library I know of hasn’t been touched for a long time.

The other approach is to use your actual CPU instruction counts, information that most CPUs will provide. I’ll use this approach in the rest of the article, but the Cachegrind-based approach is likely also viable.

Can’t we just use CPU instruction counts for a benchmark instead of a test? Doesn’t a higher instruction count means your code is slower, and a lower count means your code is faster? In some cases, yes.

But sometimes your code’s speed is dependent on more than just instruction counts. CPU memory caches, instruction-level parallelism, branch (mis)prediction: a variety of CPU hardware features mean that code with a lower CPU instruction count might actually end up being slower.

As a result, changing CPU instruction counts indicate the possibility of a speed change. You’ll often still need a separate benchmark setup to figure out if speed has really changed, measuring elapsed time instead of CPU instruction count, especially if you’re writing code in a compiled language.

A useful mental model here is that there are different sources of performance: algorithmic, use of compiled language instead of writing Python, and CPU hardware behavior. CPU instruction counts are a decent measure of speed changes from the first two, but not the third. You can learn more in my upcoming book, Practices of Performance.

Let’s see how we can use CPU instruction counts to test if performance has changed. Again, I’ll assume there’s already a benchmark running in CI. If I didn’t have a benchmark in CI, that would be the first thing to do.

Here’s a function whose speed I care about:

def wordcount(txt):
    result = {}
    for word in txt.split():
        word = word.lower()
        if word in result:
            result[word] += 1
        else:
            result[word] = 1
    return result

Let’s write a test that will fail if the number of CPU instructions used changes. I’m going to use the py-perf-event library, which lets you get access to the information provided by Linux’s perf_event_open() system call. For our purposes that gives us a way to access CPU instruction counts for the current process.

Initially the test will look like this:

from py_perf_event import measure, Hardware
from wordcount import wordcount

with open("emma.txt") as f:
    DATA = f.read()

def test_speed():
    [instruction_count] = measure(
        [Hardware.INSTRUCTIONS],
        wordcount,
        DATA
    )
    assert instruction_count == "FIXME"

Running this a few times I get:

$ pytest test_speed.py
...
FAILED AssertionError: assert 312776810 == 'FIXME'
$ pytest test_speed.py
...
FAILED AssertionError: assert 312129614 == 'FIXME'
$ pytest test_speed.py
...
FAILED AssertionError: assert 313115900 == 'FIXME'

The results are somewhat noisy, which we’ll deal with in the next section. For now, we’ll just change the test and hardcode an expected value with low precision:

def test_speed():
    [instruction_count] = measure(
        [Hardware.INSTRUCTIONS],
        wordcount,
        DATA
    )
    assert round(
        instruction_count / 10_000_000
    ) == 31, "wordcount()'s speed may have changed"

Now, if CPU instruction count changes sufficiently, the test will fail. You will then need to benchmark wordcount(), and:

1. If the benchmark suggests wordcount() is faster, or at least not slower, update the assertion with a new number.
2. If it’s slower, fix the speed regression.

You can automate the manual hardcoding of numbers with snapshot testing, for example with the syrupy library.

The current assertion will only notice changes that are more than ~3%. And we really want to catch smaller changes, in order to reduce false negatives. The solution is to reduce the noise:

1. Set a PYTHONHASHSEED environment variable, so different runs use the same hash seed. This reduces randomness, most importantly in dictionary hashing.
2. Disable ASLR, a Linux security feature that can add noise to the performance, using the setarch command’s -R argument.

When I run the original test this way, the instruction counts have far less variation:

$ export PYTHONHASHSEED=3
$ setarch x86_64 -R pytest test_speed.py
...
FAILED AssertionError: assert 312713197 == 'FIXME'
$ setarch x86_64 -R pytest test_speed.py
...
FAILED AssertionError: assert 312713196 == 'FIXME'
$ setarch x86_64 -R pytest test_speed.py
...
FAILED AssertionError: assert 312713200 == 'FIXME'

That means I can change the test to be more sensitive:

    assert round(
        instruction_count / 100_000
    ) == 3127, "wordcount()'s speed may have changed"

You’ll also want to make sure you’re using the same version of Python, and the same build of Python. The Python build provided by RedHat is slower than the Python build provided by Ubuntu, for example. So different developers might get different results if they run different Python builds, even if they’re running on the same hardware and same Python version.

The easiest way to ensure a consistent Python build is to always install Python with uv’s managed Python feature, so you use the same build of Python everywhere.

Right now the wordcount() function is storing lowercase strings:

def wordcount(txt):
    # ...
        word = word.lower()
    # ...

What happens if we change this to storing uppercase strings?

def wordcount(txt):
    # ...
        word = word.upper()
    # ...

I run the test, and:

$ setarch x86_64 -R pytest test_speed.py
...
AssertionError: wordcount()'s speed may have changed
assert 3125 == 3127
 +  where 3125 = round((312541319 / 100000))

Apparently this change has some impact on the number of CPU instructions used; whether it’s faster (or slower) in practice is something you’d have to check with a real benchmark.

Using real CPU instruction counts has some downsides:

• You can’t access CPU counters in some virtualized CI environments, for example GitHub Actions. In practice this is fine, you can simply skip these tests when running in CI; we’re already assuming you have benchmarks in CI, and those will be more informative.
• Different CPUs will give different results, in particular insofar as your code uses things like SIMD. For example, NumPy will use SIMD CPU instructions like AVX2 if available, and other instructions if AVX2 is not available.
• Macs have a completely different CPU architecture than PCs, using ARM instead of x86_64. That means very different CPU instruction counts.

To deal with different CPU features and architecture, I would suggest recording different CPU instruction counts for different categories (e.g. ARM, and then x86-64-v1, v2, v3, and v4), and then asserting against the one relevant to your local computer. Implementation is left as an exercise for a sufficiently-enthusiastic reader. Or, you can switch to the Cachegrind mechanism, which will be consistent across hardware (but currently won’t run on modern ARM Macs).

I have to admit that this particular idea is rather speculative. It depends on a specific situation:

1. You already have benchmarks.
2. Cachegrind-based benchmarks aren’t sufficiently accurate.
3. You want to catch speed regressions early, in developer-run local tests.

And there is a risk of false positives making these sort of tests too annoying to use, and I might be missing something about consistency of instruction counts across CPUs when not using Cachegrind, etc..

Nonetheless, I suspect that for some people this particular technique might be helpful. If that’s you, do try it out, and let me know how it goes.