Back to course

Where the Kernel Put Everything

25 minadvancedModule 3: The LayoutContinues from img-vdso
25 min · Module 3: The Layout · Advanced

Why This Matters

The ELF course showed a picture of a process address space: stack at the top, heap below it, a gap, the program image at the bottom. That picture is right, and it is a diagram, and diagrams do not have addresses in them.

This concept replaces the diagram with the real thing, and the real thing has three properties the diagram cannot show:

Each of those is a small thing. Together they are the difference between a mental model of memory and an accurate one, and getting them wrong is how a program ends up with a null-pointer bug that only appears sometimes.

A Simple Model

/proc/<pid>/maps is the kernel’s own list. One line per mapped region, and the format is worth reading once properly because every column is a fact about a decision the kernel made:

7f81f7609000-7f81f7621000 rw-p 00000000 00:00 0 [anon] | | | | | | | | | | | | | +-- the PATH. square | | | | | | brackets = the kernel | | | | | | made this, no file | | | | | +-- inode | | | | +-- device, MAJOR:MINOR in hex | | | +-- file OFFSET this region starts at | | +-- PERMISSIONS. four letters: | | r read w write x execute p shared | +-- the ADDRESS RANGE +-- the kernel's table of its own decisions the address range is the decision. everything else is bookkeeping.

Two columns carry more meaning than they look like they do.

r-xp versus rw-p is the PT_LOAD flags from your file, verbatim. The trailing p is MAP_PRIVATE, which is the default for everything a loader maps, and it is why a library’s data is copied if you write to it. The segment-types concept read these four flags out of a file; here they are as the kernel applied them, and the application is where GNU_RELRO becomes visible — a region that shows up as r--p because the loader asked for read-only after finishing its writes.

The p is the whole copy-on-write story in one character. Not s, which is MAP_SHARED, and which you will see on exactly one region — the one that maps libc’s data so that a printf internal counter is shared with every process using that library. The relocations course introduced COW as a cost PIE pays; this is where you can see which regions chose it and which did not.

Measured, in a Real Process

$ cd courses/img/assets/samples && ./build_samples.sh 2>&1 | sed -n '/I7/,$p'
     the map of a process that IS the process asking:
  5bd505393000-5bd505394000 r--p  00000000 .../samples/maps
  5bd505394000-5bd505395000 r-xp  00001000 .../samples/maps
  5bd505395000-5bd505396000 r--p  00002000 .../samples/maps
  5bd505396000-5bd505397000 r--p  00002000 .../samples/maps
  5bd505397000-5bd505398000 rw-p  00003000 .../samples/maps
  5bd520ccd000-5bd520cee000 rw-p  00000000 [heap]
  7f81f7400000-7f81f7428000 r--p  00000000 /usr/lib/.../libc.so.6
  7f81f7428000-7f81f75c0000 r-xp  00028000 /usr/lib/.../libc.so.6
  7f81f7609000-7f81f7621000 rw-p  00000000 [anon]

     the regions that matter, and who owns each:
  \[stack\]    7ffff8299000-7ffff82bb000
  \[heap\]     5c17e1e83000-5c17e1ea4000
  \[vdso\]     71e4c0d45000-71e4c0d47000
  ld-linux     7881a3b73000-7881a3b74000

Now the floor, which is the measurement people are surprised by. mmap will not map below 0x10000, and it reports that as EPERM rather than ENOMEM:

$ ./build_samples.sh 2>&1 | sed -n '/I5/,/I6/p'
    FIXED  0x100000 = 0x100000             ok
    FIXED   0x10000 = 0x10000              ok
    FIXED    0x8000 = 0xffffffffffffffff   Operation not permitted
    FIXED    0x1000 = 0xffffffffffffffff   Operation not permitted
    hint   0x100000 = 0x7763b1205000       ok
    hint    0x10000 = 0x7763b1204000       ok
    hint     0x8000 = 0x7763b1203000       ok
    hint     0x1000 = 0x7763b1202000       ok

Read the last four lines against the first four. The identical low addresses fail with MAP_FIXED and succeed without it. Nothing was rejected for being too low — the kernel simply put it somewhere else. That is not a quirk, that is the contract:

mmap(addr, len, prot, flags, fd, off) addr != NULL a HINT. "I would prefer here." the kernel may ignore it, and usually will if it cannot honour it. you must CHECK the return. MAP_FIXED a DEMAND. "Put it exactly here, or fail." it will overwrite whatever was there. MAP_FIXED_NOREPLACE the safe version: honour it if free, fail with EEXIST if occupied. added in Linux 4.17 precisely because plain MAP_FIXED is dangerous.

And the reason for the floor is not arbitrary — it is the oldest attack in the book. A null-pointer dereference that only writes is often not exploitable; one that also maps something at address zero turns a bug into arbitrary code execution. Refusing to map below 64 KB removes the bottom of that technique by construction, and it is why a “can I map page zero?” experiment gives EPERM rather than succeeding. The address is not reserved; nothing can be put there, which is a stronger guarantee.

Two features of the map above are worth reading off directly. The program image is five lines, not three, and two of them are r--p with the same offset 0x2000. That is section-to-segment padding: the file has a .rodata and a .eh_frame with different alignments, and the gap between them is mapped as a separate read-only region because the PT_LOADs demand it. The linker script course produced that padding; here is why it is visible at all, and it is visible only because the kernel honours the alignment rather than rounding the permissions up.

And the heap is nowhere near the program. It is at 0x5c17e1e83000 while the binary is at 0x5bd505393000 — a gap of tens of terabytes. The diagram showed the heap above the program image; the truth is that the heap is wherever brk was first called, which is a decision, and the diagram is a cartoon of it.

The Trap That Cost a Day

Here is the single most misleading thing in this area, and this course shipped a version of it before fixing it.

$ awk '{print $1, $NF}' /proc/self/maps | head -3
  75481304a000-75481304b000
  7ce75aa1c000-7ce75aa1d000  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
  7ce75aa1d000-7ce75aa4c000  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
  $ echo "that looks completely reasonable."

Those addresses are not the shell’s. They are awk’s./proc/self means “the process doing the reading”, and in a pipeline the process doing the reading is awk. The output is a perfectly valid address space — a PIE binary, a heap, a loader, everything in the right places — and it is the wrong process’s. Nothing in the output says so.

The fix is to have a program read its own map, which is three lines:

$ cat > maps.c <<'EOF'
  #include <stdio.h>
  int main(void) {
      FILE *f = fopen("/proc/self/maps", "r");
      char l[1024];
      while (fgets(l, sizeof l, f)) fputs(l, stdout);
      fclose(f); return 0;
  }
  EOF
$ clang -O1 -o maps maps.c && ./maps | head -2
  5bd505393000-5bd505394000 r--p 00000000 .../samples/maps
  5bd505394000-5bd505395000 r-xp 00001000 .../samples/maps

Now the paths say maps, because they are this process’s. The general rule is short and worth memorising: in a shell pipeline, /proc/self is the last stage, not the first.cat /proc/self/maps is the shell’s own map; awk … /proc/self/maps is awk’s; grep proc /proc/self/maps is grep’s. Every one of them a different process, every one of them a plausible-looking answer.

This is why the artifact in this course holds its child process still. imgdump blocks on stdin precisely so the parent can read /proc/<pid>/maps and /proc/<pid>/cmdline of a process it knows the state of. The path is explicit there, which is the whole fix — /proc/1234/maps cannot be ambiguous, and /proc/self/maps is only correct when the reader is the subject.

Try It Yourself

$ cd courses/img/assets/samples
$ ./build_samples.sh 2>&1 | sed -n '/I5/,$p'
$ python3 crosscheck.py 2>&1 | sed -n '/G5/,$p'
$ ./maps | head -30

Then measure the parts the build script does not:

  1. Run ./maps twenty times and collect the
     first line's address. Plot it. Then run
     with setarch -R (which disables ASLR) and
     plot that. (That is the whole of ASLR, in
     two curves, and setarch is the switch. The
     relocations course measured WHY PIE needs a
     base; this measures the base MOVING.)

  2. Find the MAP_SHARED region in ./maps output.
     There is exactly one, and it maps libc's
     data. Now count how many regions are r--p,
     r-xp, rw-p. Then read readelf -lW on the
     same libc and count its PT_LOADs. (The
     numbers will not match, and the reason is
     alignment: each LOAD becomes several
     mappings. This is the padding effect from
     the concept, measured.)

  3. mmap a region, write to it, and watch it
     become [anon] with a file offset of 0. Then
     mmap a FILE and watch it get the file's
     path and a non-zero offset. Those two lines
     in /proc/maps are the difference between
     "memory" and "a window onto a file", and it
     is the distinction the whole file-backed
     paging mechanism exists to make.

  4. cat /proc/self/maps | head -1
     grep proc /proc/self/maps | head -1
     awk 'END{print}' /proc/self/maps
     Three commands, three different processes,
     three different answers to "where is the
     stack". Confirm with the PID each one
     reports. (The first is bash, the second is
     grep, the third is awk. None is the shell
     you typed into if you piped.)

  5. Read /proc/self/limits and find the stack
     size limit. Compare it to the size of the
     [stack] mapping. Now write a recursive
     function that overflows it and watch the
     address it die at -- it should be the end
     of the stack mapping, not a random place.
     (The kernel put a guard at the boundary.
     That guard is why the crash is a clean
     SIGSEGV and not silent corruption.)

Exercise 1 is the measurement that ties this concept to the relocations course’s randomisation concept and closes the security thread. Twenty curves versus one flat line is what “the base is randomized” means when you stop describing it and start plotting it. And setarch -R is the switch, which makes the mechanism concrete: it is one flag in one program, and it turns a defence off completely. A defence you can disable with a flag on the same machine is a defence against a class of attacker, not against a determined one, and knowing which is which is the whole point of measuring it.

Exercise 5 has the best payoff. The stack grows down until it hits the guard, and then it dies exactly there — not wherever the overflow wandered, not at a corrupted pointer. That is the kernel enforcing a boundary it chose, and it is the difference between a crash that is diagnosable and a security problem. It is also the reason the stack is mapped the way it is: the growth direction is a convention the ABI picked, and the kernel honours it so that overflow becomes a fault instead of a corruption.

Connect

This concept is the load-bearing correction to a diagram the ELF course drew and a fact the linker script course produced. The PT_LOAD rows in a program header table are requests, and this is where they are granted. The linker wrote down that a segment should be readable, executable at p_vaddr, with p_align of 0x1000; the kernel read that table, chose a base, and produced five mappings where the file asked for three. The extra two are alignment padding, and they are visible in /proc/<pid>/maps as duplicate r--p lines at the same offset. Nothing in the file mentions them. They exist only because the kernel honoured the alignment.

The connection to the dynamic linking course’s load-order concept is a mechanism made visible. That course measured the order the loader mapped libraries in — breadth-first, from DT_NEEDED. This is what each of those steps looks like from outside: a new run of r--p, r-xp, rw-p lines appearing in the map, in the order the loader chose, at addresses the kernel picked. The loader decided the order; the kernel decided the addresses. Two different decisions, two different tools, and /proc/<pid>/maps is the single place both of them are written down.

The MAP_FIXED result connects to the dlopen concept and to a security idea the dyn course could not reach. MAP_FIXED is the reason a plugin system can be a security problem: load a library whose code you did not audit, and it can map a page over anything in your address space, including the GOT. The dyn course covered -z now and RELRO, which close the window by making the GOT read-only — and MAP_FIXED_NOREPLACE is the same instinct applied to mapping itself, added to Linux in 4.17 because plain MAP_FIXED had proven too sharp an edge. Both are the same lesson: an interface that can silently do something surprising will eventually do it on purpose.

And the trap connects to everything, because it is a lesson about method rather than about memory. A measurement that reads /proc/self in a pipeline is not a slightly wrong measurement — it is a confident, well-formatted, entirely wrong one, with a PIE binary and a heap and a loader all in the right places. It is exactly the shape of a correct answer. Every crosscheck in this course asserts against a value obtained by a different route for this reason: AT_ENTRY from the kernel versus e_entry from the file, AT_SYSINFO_EHDR against a mapping in the same process, AT_EXECFN against /proc/<pid>/cmdline. A checker that reads the same source twice will agree with a wrong answer, and agreeing is not the same as being right.

render_lesson_js(&mut page)