Skip to main content

mmap and Memory-Mapped Files

What This Concept Is​

mmap lets you map a file (or raw anonymous memory) into the virtual address space of your process. After the call, a pointer into the returned address range reads and writes the file's bytes directly, with no explicit read or write calls.

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

The common invocations you will see:

  • MAP_PRIVATE | PROT_READ -- a read-only view of a file. Reads from the pointer page-fault in the file's bytes lazily, cached by the kernel.
  • MAP_SHARED | PROT_READ | PROT_WRITE -- a read/write view. Stores through the pointer eventually get written back to the file, and other processes that map the same file with MAP_SHARED see them.
  • MAP_PRIVATE | PROT_READ | PROT_WRITE -- copy-on-write. Reads come from the file; the first write duplicates the page and leaves the file untouched.
  • MAP_ANONYMOUS | PROT_READ | PROT_WRITE, fd = -1 -- not a file at all. Give me length bytes of zero-initialized memory. This is how malloc implementations get large regions.

Undo with munmap(addr, length).

Why It Matters Here​

mmap is the real primitive underneath a great deal of the system. Your executable is mmap'd into your process. malloc for large allocations calls mmap. Shared libraries are mmap'd and shared. Databases like SQLite and LMDB use mmap to avoid a userspace buffer. The kernel's page cache is already holding the file's bytes; mmap just hands you a pointer to them.

Operationally, mmap trades syscalls for page faults. A read-heavy program that uses read does a syscall per block; the same program on an mmap'd file does zero after the initial mmap -- but the first access to each page triggers a page fault, which does roughly the same work. The win comes when the data is re-accessed (cached) or when multiple processes share the mapping.

Concrete Example​

A word counter that mmaps the whole file:

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int main(int argc, char **argv) {
    int fd = open(argv[1], O_RDONLY);
    struct stat st; fstat(fd, &st);
    if (st.st_size == 0) { printf("0\n"); return 0; }

    char *p = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
    if (p == MAP_FAILED) { perror("mmap"); return 1; }
    close(fd);  /* the mapping keeps a reference; fd not needed */

    size_t words = 0;
    int in_word = 0;
    for (off_t i = 0; i < st.st_size; i++) {
        int c = (unsigned char)p[i];
        int ws = (c == ' ' || c == '\t' || c == '\n');
        if (!ws && !in_word) { words++; in_word = 1; }
        else if (ws)         { in_word = 0; }
    }
    printf("%zu\n", words);
    munmap(p, st.st_size);
    return 0;
}

Note what is missing: there is no read loop and no user-space buffer. We treat the file as an array p[0..st.st_size), and the kernel reads pages in as we touch them.

Common Confusion / Misconception​

"mmap-ing a 4 GB file allocates 4 GB of RAM." It does not. mmap reserves 4 GB of virtual address space (cheap on 64-bit systems) and registers "when a page in this range faults, fetch it from this file." Physical RAM is used only for pages the process actually touches, and the kernel can evict them under pressure.

"MAP_PRIVATE with PROT_WRITE modifies the file." It does not. MAP_PRIVATE is copy-on-write: the first write duplicates the page so the change stays private. Only MAP_SHARED propagates writes back to the file.

"After mmap, I should keep the fd open." You do not need to. The mapping holds its own reference to the underlying file object. Closing the fd after mmap is the standard pattern and frees the fd slot.

Another trap: changes to a MAP_SHARED mapping are not immediately on disk. msync(addr, len, MS_SYNC) forces them. Otherwise the kernel flushes them lazily.

How To Use It​

Good use cases for mmap:

  1. Random-access reads of a large file (an index, a database).
  2. Multiple processes sharing a large read-only dataset.
  3. Read-only binary formats you want to parse as a C struct: struct header *h = (struct header*)p.
  4. Anonymous mappings as the backend for your own allocator or for shared memory.

Bad use cases:

  1. Streaming I/O from one end to another (read/write is simpler and faster).
  2. Files that other processes might truncate -- accessing a mapped page past the current file size raises SIGBUS.
  3. Files on network filesystems where consistency semantics are weak.

Check Yourself​

  1. Why does mmap-ing a 4 GB file not require 4 GB of RAM?
  2. What is the difference between MAP_SHARED and MAP_PRIVATE for a writable mapping?
  3. When can you safely close the fd passed to mmap?

Mini Drill or Application​

Extend the word counter above. Do all four:

  1. Run it on /usr/share/dict/words (or any multi-MB text file). Compare its runtime and syscall count (strace -c) against the read-based wc from Concept 5.
  2. Change MAP_PRIVATE to MAP_SHARED, add PROT_WRITE, and uppercase every letter through the pointer. Verify the file on disk is modified.
  3. mmap a 1 GB region with MAP_ANONYMOUS, then only touch the first 4 KB. Check ps's RSS column. Explain why it is tiny.
  4. In one sentence, state why truncating an mmap'd file with ftruncate to a smaller size can segfault the mapper.

Read This Only If Stuck​