Module 2: Memory, Pointers & Machine Representation: Case Studies
These case studies make pointer bugs and representation bugs visible before they become folklore.
Case Study 1: Use-After-Free In A Cache
Scenario: A cache returns a pointer to an entry. Another path evicts and frees the entry. The caller later dereferences the dangling pointer.
Source anchor: Clang's AddressSanitizer documents detection of use-after-free and related memory errors.
Module concepts: heap, ownership, dangling pointer, ASan.
Wrong Approach
"The pointer value still looks valid."
Better Approach
Define ownership:
borrowed pointer:
valid only while cache lock/lease held
owned copy:
caller must free
handle/id:
lookup each use
Tradeoff Table
| Choice | Gain | Cost |
|---|---|---|
| Return raw internal pointer | Fast access | Lifetime is easy to violate |
| Borrowed-pointer contract | Clearer ownership rules | Requires lock or lease discipline |
| Copy or handle-based API | Safer lifetime boundary | More lookup or allocation overhead |
Failure Mode
Tests pass under light usage, but eviction on another path frees the object and later reads become intermittent crashes or corrupt results.
Required Artifact
Write an ASan reproduction and an ownership contract for the cache API.
Project / Capstone Connection
Use this ownership language in any allocator, cache, or shared-buffer subsystem built later in systems work.
Case Study 2: Buffer Overflow Detected By Valgrind
Scenario: A parser allocates bytes and writes + 1 to append a terminator.
Source anchor: Valgrind Memcheck documents how it reports invalid reads, writes, and leaks in C and C++ programs.
Module concepts: heap allocation, bounds, terminator, Valgrind.
Wrong Approach
Forget that strings need room for '\0'.
Better Approach
Allocate and test explicitly:
char *s = malloc(len + 1);
memcpy(s, input, len);
s[len] = '\0';
Tradeoff Table
| Choice | Gain | Cost |
|---|---|---|
| Allocate exact payload length | Minimal memory use | No room for terminator |
Allocate len + 1 | Correct C-string layout | Must reason about overflow on size math |
| Higher-level buffer helper | Reusable safety | More abstraction and API surface |
Failure Mode
The extra terminator write lands one byte past the allocation, producing small corruptions that may surface only under tooling or unrelated code changes.
Required Artifact
Run or mock a Memcheck report and explain the invalid write.
Project / Capstone Connection
Apply this allocation rule to parsers, tokenizers, and serialization helpers that materialize C strings.
Case Study 3: Endianness Breaks Binary File Format
Scenario: A program writes an integer directly with fwrite(&x, sizeof x, 1, f). The file is read on another architecture and values are wrong.
Source anchor: Machine representation and serialization require a byte-order contract; network byte order functions provide a practical reference. See byteorder(3).
Module concepts: endianness, serialization, integer width.
Wrong Approach
"Bytes on my machine are the format."
Better Approach
Define format:
field width:
uint32_t
byte order:
big-endian/network order
version:
header magic + format version
Tradeoff Table
| Choice | Gain | Cost |
|---|---|---|
| Dump native integer bytes | Fast to implement | Non-portable across architectures |
| Fixed-width explicit byte order | Portable and testable | Requires encode/decode code |
| Text serialization | Human-readable | Larger and slower |
Failure Mode
Files look correct on the authoring machine, then decode into nonsensical values or version mismatches on another architecture.
Required Artifact
Write a binary format spec with byte offsets and endianness.
Project / Capstone Connection
Use this format discipline for binary logs, caches, and protocol exercises later in the program.
Case Study 4: Struct Padding Over The Wire
Scenario: A network protocol sends sizeof(struct Message) bytes. Another compiler lays out padding differently.
Source anchor: C object representation and alignment rules make raw structs unsafe as portable wire formats.
Module concepts: struct layout, padding, alignment, packed formats.
Wrong Approach
Use raw in-memory structs as protocol messages.
Better Approach
Serialize fields explicitly:
offset 0: uint16_t type
offset 2: uint32_t length
offset 6: payload bytes
Tradeoff Table
| Choice | Gain | Cost |
|---|---|---|
| Send raw struct bytes | Very little code | Padding, alignment, and ABI coupling |
| Explicit field serialization | Stable wire format | More encode/decode work |
| Packed struct shortcuts | Smaller apparent diff | Can still hide portability and alignment issues |
Failure Mode
One build inserts padding or changes alignment, so peer systems misread fields even though each side's local tests pass.
Required Artifact
Draw struct layout with padding and a separate wire layout.
Project / Capstone Connection
Carry this distinction into socket protocols, file headers, and IPC message formats.
Case Study 5: Floating-Point Equality In Money-Like Calculation
Scenario: A program sums decimal prices in double and checks equality against an expected total.
Source anchor: IEEE 754 floating point representation explains rounding and special values; use compiler or runtime examples to inspect binary approximation.
Module concepts: floating point, rounding, binary representation, fixed-point.
Wrong Approach
Compare decimal money with exact double equality.
Better Approach
Use integer cents or decimal type:
int64_t cents = 1299;
For measurement values, compare within tolerance.
Tradeoff Table
| Choice | Gain | Cost |
|---|---|---|
double equality for money | Simple code | Rounding mismatch and unstable comparisons |
| Integer cents | Exact for fixed decimal scale | Requires explicit scaling and formatting |
| Tolerance-based compare | Works for measurements | Not appropriate for exact financial totals |
Failure Mode
An apparently exact total fails equality checks because decimal fractions such as 0.1 are represented approximately in binary.
Required Artifact
Write a representation note showing why 0.1 + 0.2 is not exact in binary floating point.
Project / Capstone Connection
Use this choice framework when later projects need counters, pricing-like values, or scientific measurements with different precision rules.
Source Map
| Source | Use it for |
|---|---|
| AddressSanitizer | detecting use-after-free and memory errors |
| Valgrind Memcheck | invalid reads/writes and leaks |
| byteorder(3) | network byte order |
| cppreference fixed-width integers | integer widths |
Completion Standard
- At least three artifacts are completed.
- At least one artifact includes ASan or Valgrind evidence.
- At least one artifact defines a byte-level format.