Sanitizers as another line of defence

Last updated on 2026-07-01 | Edit this page

Overview

Questions

  • The tests all pass — so why does the program crash?
  • What classes of bug are invisible to unit tests?

Objectives

  • Explain what AddressSanitizer instrument at compile time.
  • Build the test executable with sanitizer instrumentation.
  • Observe a specific case that all unit tests miss
  • Describe the relationship between unit testing, coverage, and sanitizers as complementary tools

Our tests pass, our coverage is high, but are we bug free?

In short, we don’t know there are no bugs, but test coverage gives us increased confidence that at least most lines of code are exercised. Bugs are inevitable, and as we discussed earlier, if a bug does arises we could:

  1. Write a GoogleTest test case that exposes the bug, i.e. we construct the inputs/state and assert the the expected pass condition. Failure of the test then exposes the bug.
  2. Diagnose, edit, build, until the test passes.
  3. The bug is fixed and our test case stays in the suite as a regression test.

Can we give ourselves more warning of obvious problems that tests might not pick up though?

Introducing a deliberate bug

Let’s say we want to tidy up some of the internals of Histogram, and we decide to store the overflow counts in the last bin. We naively update fill to do this:

CPP 

void Histogram::fill(float x, float weight)
{
    ++n_entries_;
    int bin = static_cast<int>((x - x_min_) / bin_width_);
    if (x < x_min_)
    {
        ++n_underflow_;
        return;
    }
    if (x >= x_max_)
    {
        ++n_overflow_;
        // starting to refactor - store overflow in last element of counts_ vector
        counts_[bin] += 1.0f;
        return;
    }
    counts_[bin] += weight;
    value_sum_ += x;
    ++in_range_;
}

If we recompile, retest and rerun coverage, we will find:

  1. All the tests pass.
  2. The coverage remains high, and our line is executed.

Yet we have a genuine (albeit contrived) bug - we are writing to memory outside of the bounds of counts_ and neither testing or coverage has picked this up.

Callout

Yes, this example is contrived. In practice, bugs like this are more subtle and insidious, but the same principle applies: neither coverage or tests would neccesarily identify the issue.

Code Sanitizers

As with coverage, sanitizers instrument our code with detectors for various types of runtime errors:

  • Address: out of bounds reads/writes, leaks.
  • Threading: e.g. race conditions.
  • Undefined behaviour: e.g. integer overflow, divide-by-zero.

GCC and Clang provide these for us, and like we did for coverage, we need to add the needed compiler and linker flags (sanitizers usually come as a library the compiler will automatically add, but this means the flags also have to be applied at link time). As before, we can use a custom CMake build type to do this:

CMAKE 

...
# - C++ Standard setup
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

# - Coverage Build Type flags for C++ 
set(CMAKE_CXX_FLAGS_COVERAGE "-O0 --coverage")

# - Sanitizer Build Type flags for C++
set(CMAKE_CXX_FLAGS_SANITIZE "-O1 -g -fno-omit-frame-pointer -fsanitize=address")
...
Checklist
  1. -O1 -g: Sanitizers do introduce a performance penalty, so we use the lowest level of optimization. This isn’t significant for the tests we write, but is the recommended default. We add debugging so we can get line numbers etc, or to assist debugging when a problem is found.
  2. -fno-omit-frame-pointer: This gives a cleaner “stack trace” which we’ll see in a bit.
  3. -fsanitize=address: Enable the address sanitizer. Note that it must appear in both the compiler and linker flags. CMake handles this for us when we set flags like this, but some generators make need extra work (mostly Xcode).

We have deliberately only picked one sanitizer here for simplicity, and also because whilst GCC and Clang allow you to add multiple sanitizers -fsanitize=address,thread,... care is needed as some do not work well together. It’s best to start with one sanitizer per build type.

To use the “sanitized” build, all we need to do is build and test the project in the needed build mode:

BASH 

(ccptepp-test) $ cmake -GNinja -DCMAKE_BUILD_TYPE=Sanitize -S . -B build-sanitize
-- The C compiler identification is AppleClang 17.0.0.17000604
-- The CXX compiler identification is AppleClang 17.0.0.17000604
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found GTest: /tmp/ccptepp-test/.pixi/envs/default/lib/cmake/GTest/GTestConfig.cmake (found version "1.17.0")
-- Configuring done (0.9s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/ccptepp-test/build-sanitize
Callout

GoogleTest plays nicely with sanitizers (or vice versa), so you don’t need to worry about false positives here.

When we now run the tests, we should get the following, long failure:

BASH 

(ccptepp-test) $ ctest -R Hist --output-on-failure --test-dir build-sanitize
Test project /tmp/ccptepp-test/build-sanitize
    Start 2: TestHistogram
1/1 Test #2: TestHistogram ....................Subprocess aborted***Exception:   0.39 sec
Running main() from /Users/runner/miniforge3/conda-bld/gtest-split_1748319995326/work/googletest/src/gtest_main.cc
[==========] Running 21 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 8 tests from HistogramConstruction
[ RUN      ] HistogramConstruction.ValidParametersDoNotThrow
[       OK ] HistogramConstruction.ValidParametersDoNotThrow (0 ms)
[ RUN      ] HistogramConstruction.NegativeBinsThrows
[       OK ] HistogramConstruction.NegativeBinsThrows (0 ms)
[ RUN      ] HistogramConstruction.ZeroBinsThrows
[       OK ] HistogramConstruction.ZeroBinsThrows (0 ms)
[ RUN      ] HistogramConstruction.IncorrectRangeThrows
[       OK ] HistogramConstruction.IncorrectRangeThrows (0 ms)
[ RUN      ] HistogramConstruction.BinCountsHasCorrectSize
[       OK ] HistogramConstruction.BinCountsHasCorrectSize (0 ms)
[ RUN      ] HistogramConstruction.AllBinsInitiallyZero
[       OK ] HistogramConstruction.AllBinsInitiallyZero (0 ms)
[ RUN      ] HistogramConstruction.BinEdgesHasCorrectSize
[       OK ] HistogramConstruction.BinEdgesHasCorrectSize (0 ms)
[ RUN      ] HistogramConstruction.BinEdgesHaveCorrectExtremes
[       OK ] HistogramConstruction.BinEdgesHaveCorrectExtremes (0 ms)
[----------] 8 tests from HistogramConstruction (0 ms total)

[----------] 8 tests from HistogramFillTest
[ RUN      ] HistogramFillTest.Mean
[       OK ] HistogramFillTest.Mean (0 ms)
[ RUN      ] HistogramFillTest.SingleFillIncreasesCorrectBin
[       OK ] HistogramFillTest.SingleFillIncreasesCorrectBin (0 ms)
[ RUN      ] HistogramFillTest.SingleFillLeavesOtherBinsZero
[       OK ] HistogramFillTest.SingleFillLeavesOtherBinsZero (0 ms)
[ RUN      ] HistogramFillTest.ValueBelowXMinIsUnderflow
[       OK ] HistogramFillTest.ValueBelowXMinIsUnderflow (0 ms)
[ RUN      ] HistogramFillTest.MultipleWeightedFillsAccumulate
[       OK ] HistogramFillTest.MultipleWeightedFillsAccumulate (0 ms)
[ RUN      ] HistogramFillTest.NEntriesCountsAllFillsRegardlessOfWeight
[       OK ] HistogramFillTest.NEntriesCountsAllFillsRegardlessOfWeight (0 ms)
[ RUN      ] HistogramFillTest.MeanOfSymmetricFillsIsNearCentre
[       OK ] HistogramFillTest.MeanOfSymmetricFillsIsNearCentre (0 ms)
[ RUN      ] HistogramFillTest.MeanExcludesUnderflowValues
[       OK ] HistogramFillTest.MeanExcludesUnderflowValues (0 ms)
[----------] 8 tests from HistogramFillTest (0 ms total)

[----------] 5 tests from LinearHistogramTest
[ RUN      ] LinearHistogramTest.TotalEntryCount
=================================================================
==61793==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6040000021bc at pc 0x00010091a27c bp 0x00016f4fe810 sp 0x00016f4fe808
READ of size 4 at 0x6040000021bc thread T0
    #0 0x00010091a278 in Histogram::fill(float, float) histogram.cpp:27
    #1 0x000100a62290 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0xb4 (libgtest.1.17.0.dylib:arm64+0x22290)
    #2 0x000100a620d4 in testing::Test::Run()+0x80 (libgtest.1.17.0.dylib:arm64+0x220d4)
    #3 0x000100a63648 in testing::TestInfo::Run()+0x160 (libgtest.1.17.0.dylib:arm64+0x23648)
    #4 0x000100a648f4 in testing::TestSuite::Run()+0x3a4 (libgtest.1.17.0.dylib:arm64+0x248f4)
    #5 0x000100a7661c in testing::internal::UnitTestImpl::RunAllTests()+0x6d8 (libgtest.1.17.0.dylib:arm64+0x3661c)
    #6 0x000100a75db0 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0xb4 (libgtest.1.17.0.dylib:arm64+0x35db0)
    #7 0x000100a75ca8 in testing::UnitTest::Run()+0x88 (libgtest.1.17.0.dylib:arm64+0x35ca8)
    #8 0x000100967e80 in main+0x50 (libgtest_main.1.17.0.dylib:arm64+0x3e80)
    #9 0x000194d4eb94  (<unknown module>)

0x6040000021bc is located 4 bytes after 40-byte region [0x604000002190,0x6040000021b8)
allocated by thread T0 here:
    #0 0x000101107428 in _Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4b428)
    #1 0x000100919c6c in std::__1::vector<float, std::__1::allocator<float>>::assign(unsigned long, float const&) vector.h:1076
    #2 0x000100919ac0 in Histogram::Histogram(int, float, float) histogram.cpp:11
    #3 0x00010090dabc in testing::internal::TestFactoryImpl<LinearHistogramTest_TotalEntryCount_Test>::CreateTest() gtest-internal.h:448
    #4 0x000100a63a40 in testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*)+0xb4 (libgtest.1.17.0.dylib:arm64+0x23a40)
    #5 0x000100a6362c in testing::TestInfo::Run()+0x144 (libgtest.1.17.0.dylib:arm64+0x2362c)
    #6 0x000100a648f4 in testing::TestSuite::Run()+0x3a4 (libgtest.1.17.0.dylib:arm64+0x248f4)
    #7 0x000100a7661c in testing::internal::UnitTestImpl::RunAllTests()+0x6d8 (libgtest.1.17.0.dylib:arm64+0x3661c)
    #8 0x000100a75db0 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0xb4 (libgtest.1.17.0.dylib:arm64+0x35db0)
    #9 0x000100a75ca8 in testing::UnitTest::Run()+0x88 (libgtest.1.17.0.dylib:arm64+0x35ca8)
    #10 0x000100967e80 in main+0x50 (libgtest_main.1.17.0.dylib:arm64+0x3e80)
    #11 0x000194d4eb94  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow histogram.cpp:27 in Histogram::fill(float, float)
Shadow bytes around the buggy address:
  0x604000001f00: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x604000001f80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x604000002000: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x604000002080: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x604000002100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
=>0x604000002180: fa fa 00 00 00 00 00[fa]fa fa 00 00 00 00 00 fa
  0x604000002200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x604000002280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x604000002300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x604000002380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x604000002400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==61793==ABORTING


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.40 sec

The following tests FAILED:
	  2 - TestHistogram (Subprocess aborted)
Errors while running CTest

The good news is that we have an error, which is what we wanted, but how to make sense of the output? It’s scarier than it looks as the santizer has printed:

BASH 

SUMMARY: AddressSanitizer: heap-buffer-overflow histogram.cpp:27 in Histogram::fill(float, float)

That alone is sufficient to pin point the error in our simple case, but if we needed further triage, we get a full stack trace of the fault, starting at

==61793==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6040000021bc at pc 0x00010091a27c bp 0x00016f4fe810 sp 0x00016f4fe808
READ of size 4 at 0x6040000021bc thread T0
    #0 0x00010091a278 in Histogram::fill(float, float) histogram.cpp:27

Usually the first (zeroth) stack frame contains the exact source, but if the error is dependent
on previous calls, you have that information to aid triage.

Hopefully, using documented specifications, writing good tests, and ensuring they cover the code well will prevent a high fraction of problems occuring. Using sanitizers provides one extra layer of defence (largely against ourselves!).

Key Points
  • Unit tests check that your code does what you intended; sanitizers check for errors your intentions did not anticipate
  • A test suite that is green and fully covered can still contain memory errors and undefined behaviour
  • AddressSanitizer detects out-of-bounds memory access and use-after-free at runtime — errors that produce no compiler warning and may crash only rarely in production
  • Sanitizers diagnose bugs that already exist; a well-chosen test prevents their reintroduction
  • No single tool is sufficient — unit tests, coverage measurement, and sanitizers answer different questions and catch different bugs; together they give you the best practical assurance that your code is correct