Getting Started with BenchmarkDotNet

Photo by Kolleen Gladden on Unsplash

How do you know whether code is performant?

As a developer I’ve written a lot of code that I felt was performant.

But the problem is I have rarely tested to see whether it was. Or whether there were better alternatives?

As soon something met requirements and there weren’t major problems with it, I’d create a PR.

Luckily there’s a better way. You can actually take the time to benchmark your code.

Luckily libraries like BenchmarkDotNet exist to help make that task a lot easier.

So lets take a closer look on how to start using BenchmarkDotNet to benchmark your code.

Overview

BenchmarkDotNet is very simple to start using. All you have to do is:

1. Add nuget package
2. Add Benchmark attributes to items
3. Create a BenchmarkRunner
4. Run in Release mode.

I’m going to use the dotnet cli and start from scratch so you can see how it’s done beginning to end.

I’m using a Powershell environment to execute my commands, but the commands should be very similar if you’re on Mac or Linux.

Create Project Structure

Ensure you have dotnet core installed on your machine, I’m running 2.2. Lets use the dotnet cli to get everything done. You can also create replicate the same project using Visual Studio or Visual Studio Code.

A copy of the finished project can be found at this repo so you can check if you run into any issues.

Open up powershell and lets create a new folder and our solution:

> mkdir SpeedyStuff
> cd .\SpeedyStuff\
> dotnet new sln --name SpeedyStuff

Then create a console project inside of it:

> mkdir Benchmarker
> cd .\Benchmarker\
> dotnet new console

After those two steps here is a snapshot of our project structure:

SpeedyStuff\
--SpeedyStuff.sln
--Benchmarker\
----Benchmarker.csproj
----Program.cs
----obj\

Lets move back up and add a project reference into the solution.

> cd ..
> dotnet sln .\SpeedyStuff.sln add .\Benchmarker\Benchmarker.csproj
> dotnet build
> dotnet run

Running it gives an error because we didn’t specify the project to run, so lets do that and run again.

> dotnet run -p .\Benchmarker\Benchmarker.csproj
Hello World!

We’ve got Hello World printing to the console! Small victories! Now lets add some benchmarking and get testing.

Benchmarking

Lets add the BenchmarkDotNet nuget package to the Benchmarker project.

> cd .\Benchmarker\
> dotnet add package BenchmarkDotNet

This will automatically install the most recent version. At the time of writing it’s version 0.11.5.

Lets open up Program.cs in that folder and replace with the following. It’s a simplified version of the benchmark found on the BenchmarkDotNet website.

Return to the console and lets run your first benchmark.

> cd ..
> dotnet run -p .\Benchmarker\Benchmarker.csproj

Error.

When benchmarking always remember to run in Release mode. Code is compiled differently for Debug versus Release mode. The C# compiler performs optimizes in Release mode that aren’t available in debug mode .If you want to take an accurate measure of performance always use Release mode.

Lets specify the release mode and run a benchmark.

> dotnet run -p .\Benchmarker\Benchmarker.csproj -c Release

That finished running in about 30 seconds on my machine. A lot of information was printed to the console, you scroll up to view everything, but lets examine the table.

This simple table gives us a pretty good overview of the benchmark. Taking a look, it seems like MD5 is about twice as fast of the SHA256 method.

MD5 and SHA256 are hash functions widely used for security. MD5 outputs a 128 bit hash and SHA256 outputs a 256 bit hash. The bit difference means that SHA256’s hash is more secure but it takes longer to compute. So our first benchmark lines up with our understanding of what we are benchmarking.

Lets add the [MemoryDiagnoser] attribute to the class and run this again. The table will now include some memory information.

Now the table has several more columns detailing memory information. It describes how often different garbage collection generations run and how many total bytes were allocated during the run.

The last column shows us that SHA256 allocated about ~40% more memory. The Gen 0 column also shows us that the Garbage Collector ran more frequently during the SHA256 example.

So SHA256 runs slower and uses more memory.

What can we take away from this Benchmark? There’s a tradeoff between security and performance when it comes to hashing functions.

If you need security, always choose SHA256. But if you are just looking for a hashing function, maybe MD5 would fit your needs and give you better performance.

Benchmarking Collections

Everyone uses collections all the time. This simple example will show us some implementations differences between an array and a list.

We have four benchmarks this time. Each adding elements to a different collection. The first collection is a basic array. The second collection is a list specifying an initial capacity. The third collection is a list with the default initial capacity. And the last collection is a HashSet.

We are also adding a second value in the Params tag to run with our Benchmark. This will allow us to see what happens as higher values of N are used.

Run that and a table similar to the one below should print to your console.

That Benchmark took ~3 minutes on my local machine.

Examining the results, its easy to see that across the board using an Array gave us the best performance, followed by ListA, then ListB, the way behind is HashSet.

Array and ListA had the same memory footprint for both Ns.

Time wise, Array ran faster than ListA, around twice as fast for 1000 and ~40% faster for 1,000,000.

The ListB is where things get interesting. It had the worst memory footprint and time for both Ns.

Why was ListA’s performance so much better than ListB? They’re the same class, the only difference is we gave ListA an size when creating it. Why does that make such a huge difference in our performance test?

Collection Details

In C#, an array is a collection that occupies contiguous memory locations. Navigating and inserting into an array are very fast, constant time actually.

The List class is a wrapper class around an array. It provides easier ways to interact with an array but with a tradeoff in performance.

This comes from the fact that an array always has a fixed length. It can never change.

But a List can grow dynamically. It’s this subtle difference that is the root of our performance differences.

For ListB we didn’t specify the length of the List and it worked just fine. It grows to handle whatever size is needed.

The way C# handles this is whenever the first element is added to the List is creates an array with an initial capacity of 4. When a 5th element is added, it creates a new array with double the capacity of the old one, copies everything over, adds the 5th element and returns.

It does this every time it runs out of space, creates a new array with double the capacity, copies everything over, adds the new element and returns.

When the above runs it prints out the following:

count/capacity: 0/0
count/capacity: 1/4
count/capacity: 2/4
count/capacity: 3/4
count/capacity: 4/4
count/capacity: 5/8
count/capacity: 6/8
count/capacity: 7/8
count/capacity: 8/8
count/capacity: 9/16
count/capacity: 10/16

The count is how many elements are in the array and the capacity is how many elements the array can hold.

When it reaches its capacity, it doubles. Default value is 4, which is doubled to 8, which is doubled to 16.

That’s why ListB runs slower and uses so much more memory. Every time it runs out of capacity it has to create a new array and copy everything over.

ListA avoids that problem because we create the list with an initial value. We know how big the array is going to be. During out benchmark ListA never needs to resize.

That’s why ListA’s memory footprint is very similar to the Array. ListA uses an array internally and Array is just a plain array. They will always have very familiar memory footprints as long as the List doesn't need to resize.

If ListA uses an array why does it run slower? Even though it uses an array internally, there’s still extra checks and steps it takes when adding items to the array. Over the course of a long performance run those steps add up.

It’s worth noting that in this example the Array was clearly the best collection to use. But we only Benchmarked one small aspect of Array v. List.

The are many reasons to use a List v. Array. This benchmark only shows if you’re only adding elements to a collection, use an Array.

HashSet Performance

Due to a commenter asking, I’ve updated this benchmark to also include a HashSet. As you probably saw in the results above, performance of the HashSet was pretty terrible compared to the array and lists. To figure out why lets dive into the source code of the HashSet. When you call the Add method on a HashSet it eventually makes it way down to this method.

Lets compare that to the List add method.

As you can see the HashSet performs a lot more work than a List does. That boils down to the differences between the two data structures. HashSet only holds unique elements. So every time something is added it checks to see whether it exists or not. A List doesn’t care about uniqueness, it will just throw it in the list and continue.

Because uniqueness of elements is not something that we care about, using a HashSet is overkill for this benchmark.

Which is another reason knowing data structures is important. Given a particular problem, you’ll be able to choose the most appropriate data structure. Something that only has the features you want and doesn’t kill your performance, as HashSet did in this case.

Benchmarking Is Hard, Use BenchmarkDotNet

These two examples might make it seem like benchmarking is fairly easy.

Certainly BenchmarkDotNet makes it easier, but benchmarking itself is actually pretty hard problem to solve.

There are so many things that BenchmarkDotNet implements so you don’t have to. It has warmup runs. It runs each benchmark multiple times to get averages. It offers multiple runtimes to target.

It can provide a more detailed statistics around the benchmark. It has predefined exporters that allow you to see the results of your performance in a web page, pdf or R script.

It can benchmark F# or VB code as well!

Benchmarking is important and is the only way to know whether your code is performant.

Benchmarking should become another resource for you to better enhance your coding skills. You should become familiar with BenchmarkDotNet and it’s features.

Take some time to explore this library. Come up with your own examples. Instead of just guessing between two implementations measure it and let the numbers decide it.

If you’re using .NET, learn to use BecnhmarkDotNet!

Github Repo

I’m Morgan Kenyon. I’m a .NET developer working in the DFW area. I find C# a great language to use and it’s also backed by a great ecosystem, I love solving hard problems and want to continue talking about the tech I use. If you found this article helpful or thought provoking lets connect over LinkedIn!