Back to course

Rebuilding the Image Reader

27 minadvancedModule 4: Rebuild ItContinues from img-place
27 min · Module 4: Rebuild It · Advanced

Why This Matters

Five concepts, one artifact. The chain this collection has been building runs from a compiler’s output to something running, and every step so far has ended at a boundary: the object file stops at the archive, the linker stops at the file, the loader stops at the process. This is the step where the boundary disappears, because the thing being parsed is not a file at all — it is a running process, and the kernel is the only thing that wrote it down.

So the artifact is a reader, and the discipline around it is the thing worth building. A parser that prints values is a demo. A parser that cross-checks every value against an independent source is a tool, and the difference is the entire difference between “I think this is right” and “this is right, and here is why you should believe it.”

Two programs, both under 500 lines, and together they are the whole course:

$ cd courses/img/assets/samples
$ wc -l imgdump.c stackwalk.py build_samples.sh crosscheck.py
  132 imgdump.c
  430 stackwalk.py
  330 build_samples.sh
  240 crosscheck.py
$ python3 stackwalk.py 2>&1 | tail -3
  18/18 checks passed
  ALL CHECKS PASS

A Simple Model

The division of labour, and it is the shape of every parser worth writing:

imgdump.c 132 lines, in C the only thing that CAN read the raw initial stack, because it runs BEFORE anything else in the process and can ask the kernel where its own stack is. it does not parse. it hands over bytes and holds still. stackwalk.py 430 lines, in Python all the parsing. no libc, no readelf, no /proc/self -- just the bytes and a struct.unpack_from. and then the checking. build_samples.sh the specimens, written out by the script itself, so there is exactly one source of truth. crosscheck.py re-derives every claim in the course from a fresh run, so a claim that rots is a FAILING EXIT CODE, not a lie in a markdown file.

The split is not arbitrary and it is the reason the thing works. A program cannot fully parse its own initial stack — not because it is hard, but because by the time your code runs, libc has consumed part of the frame. So there has to be a handoff: something inside the process that captures the bytes, and something outside that does the thinking. The boundary between those two things is the architecture of the tool, and it is forced by the subject matter rather than chosen for elegance.

And the stdin block is what makes the checking sound. Without it, imgdump would print its bytes and exit, and the parent would read /proc/<pid>/maps of a pid that might already be gone — or worse, might have been recycled, so the parent reads a different process’s map and compares it to the wrong stack. With the block, the process is provably alive and provably idle for exactly as long as the checks take.

parent child ------ ----- spawn imgdump -------------------> execve read STACK header + bytes <------ print, fflush read /proc/PID/maps <-------- BLOCKED on read /proc/PID/cmdline <-------- stdin, having parse the bytes executed nothing CHECK one against the other write "go\n" ------------------> fgets returns waitpid ----------------------> exit 0 three observations, one instant. that is what makes the comparison mean something.

Eighteen Checks, and Why Each One Earns Its Place

A check that re-reads the same source as the code under test verifies nothing. Every check below compares two things obtained by different routes, and the pairs are chosen so that a bug in one cannot hide a bug in the other.

KERNEL vs FILE AT_PHNUM == e_phnum 14 == 14 AT_PHENT == e_phentsize 56 == 56 AT_ENTRY-AT_PHDR == e_entry-e_phoff AT_PHDR -> maps -> file offset == e_phoff AT_ENTRY -> maps -> file offset == e_entry KERNEL vs KERNEL (same instant) AT_SYSINFO_EHDR == base of [vdso] AT_EXECFN == /proc/PID/cmdline[0] AT_RANDOM inside the bytes we hold KERNEL vs RUNTIME AT_PAGESZ == sysconf(_SC_PAGESIZE) AT_UID == getuid() KERNEL vs THE KERNEL'S OWN FILE the child read its own startstack from /proc/self/stat; the parent read the same field from /proc/PID/stat. equal.

The two “kernel vs kernel” checks are the ones that took the most work and they are the best. Comparing AT_SYSINFO_EHDR against a vDSO mapping read in a different process is impossible, because ASLR puts the vDSO somewhere different in each — that was a real mistake here, documented in research.md. And the startstack check is the cheapest and strongest of all: the child computed an address from its own /proc/self/stat, the parent computed the same field from /proc/<pid>/stat, and they agree. Two different readers, one number, zero shared state.

Three checks are deliberately weak, and it is worth being explicit about which and why. AT_SECURE == 0, AT_UID == getuid() and AT_MINSIGSTKSZ is plausible all pass on every normal run, and they are there as regression checks rather than as proofs — they would catch a parser that had started reading the wrong array, which is a real failure mode, and they cost three lines. A check you understand is worth more than a check that is merely impressive, and three honest weak checks beat one that looks strong and proves nothing.

The Part Worth Copying

Not the auxv walk — that is in the previous concepts. This is the verification pattern, and it generalises to every parser in this collection.

$ python3 stackwalk.py 2>&1 | sed -n '/KERNEL DESC/,$p'
  [ok  ] AT_PHENT == e_phentsize in the file  -- auxv 56, file 56
  [ok  ] AT_PHNUM == e_phnum in the file  -- auxv 14, file 14
  [ok  ] AT_ENTRY - AT_PHDR == e_entry - e_phoff  -- 0x10f0 == 0x10f0
  [ok  ] AT_ENTRY lands in an r-x region  -- 0x5a9b10f0f130 in .../imgdump
  [ok  ] AT_ENTRY maps to the file offset of e_entry  -- 0x1130, 0x1130
  [ok  ] AT_PHDR lands in a region of our own file
  [ok  ] AT_PHDR maps back to e_phoff  -- 0x40 vs 0x40
  [ok  ] AT_BASE lands in the dynamic linker, not in us
  [ok  ] ld.so is ET_DYN, so it must be given a base  -- e_type 3
  [ok  ] AT_PAGESZ == the system page size  -- auxv 4096, sysconf 4096
  [ok  ] AT_RANDOM points into the initial stack we hold
  [ok  ] AT_RANDOM and AT_PLATFORM are different places on the stack
  [ok  ] AT_EXECFN names the file the kernel actually ran
  [ok  ] AT_SYSINFO_EHDR == the base of the [vdso] mapping
  [ok  ] AT_SECURE == 0 (not a setuid run)  -- value 0
  [ok  ] AT_UID == getuid()  -- auxv 1000, getuid 1000
  [ok  ] AT_MINSIGSTKSZ is a SIZE and stays plausible  -- 3376 bytes
  18/18 checks passed

Line three is the one to build first in any parser like this. Two numbers that must be equal for a reason that is not “they came from the same place”. The left is the kernel’s translation of the file into memory; the right is the file’s own layout. If the kernel were doing anything other than loading this file at one base, they would differ — and if a future kernel changed how it builds the process image, this is the line that would catch it.

Lines four and five close the loop in the other direction, and they are the ones that make “the kernel is telling the truth” a stronger statement than it looks. Take the kernel’s AT_ENTRY, find the mapping that contains it, compute its offset within that mapping, add the mapping’s file offset, and you have a file offset. That is a number the kernel never gave you, derived through two hops, and it must equal e_entry — which the script read from a completely different structure. Kernel → address → mapping → file offset → the number in the file.

And note what the script does not do: it never shells out to readelf. It parses the 64-byte ELF header itself, with struct.unpack_from. That is deliberate and it is the mission of this collection in one line — the tool is the oracle, and the format is the subject. A crosscheck that used readelf to verify readelf’s fields would prove that readelf is consistent with itself.

Finally, the harness that keeps all of it honest:

$ python3 crosscheck.py 2>&1 | tail -13
  G1   12 passed   0 failed      the auxv exists
  G2    5 passed   0 failed      auxv vs the ELF file
  G3    3 passed   0 failed      AT_BASE is another file
  G4    7 passed   0 failed      the vDSO is a real ELF
  G5    5 passed   0 failed      mmap_min_addr
  G6    3 passed   0 failed      demand paging
  G7    5 passed   0 failed      the process map
  G8    1 passed   0 failed      the artifact, end to end
  41/41 checks passed
  ALL CLAIMS HOLD

Group G8 runs stackwalk.py as a subprocess and requires exit 0. So the harness is not a parallel description of the artifact — it is a test of it, and the two cannot drift apart without a failure. This is the last piece, and it is the piece that makes the other five courses’ claims checkable too: a claim is not “measured” until something re-measures it.

Try It Yourself

$ cd courses/img/assets/samples
$ ./build_samples.sh                    # from empty
$ python3 stackwalk.py                 # the artifact
$ python3 stackwalk.py -v              # with the raw random bytes
$ python3 crosscheck.py                # all 41 claims
$ python3 stackwalk.py --dump          # decoded, no assertions

Then make it fail, which is the only real test of a checker:

  1. Make stackwalk.py lie. Change the
     AT_ENTRY - AT_PHDR check to compare
     against e_entry + e_phoff and run it.
     (The check fails, crosscheck reports 40/41
     and exits non-zero, and the build script
     shows the failure. A checker that has never
     failed is not known to work.)

  2. Make the KERNEL's data wrong instead.
     Add an AT_PHNUM of 15 to the raw bytes in
     imgdump's output before parsing. The check
     against e_phnum catches it immediately.
     (This is the check's real purpose: not to
     confirm the parser, but to catch a
     disagreement.)

  3. Now break the FILE side: corrupt byte 24
     of imgdump -- the low byte of e_entry --
     with dd. The auxv still says the old
     address, the file now says something else,
     and TWO checks fail: the difference check
     and the offset check. (Two failures for one
     corruption is the right answer, and seeing
     which ones is how you learn what each is
     actually testing.)

  4. Add a check stackwalk.py does not have:
     assert that AT_PHNUM * AT_PHENT + AT_PHDR
     does not overflow, and that the program
     header table is inside a read-only region.
     Then break a PT_LOAD's p_flags in the file
     to be writable and see whether the memory
     check catches it. (It will not -- /proc
     reports the KERNEL's permissions, not the
     file's. Write down what that means: the
     kernel is trusted here, and a parser that
     wants to verify the FILE must read the
     file.)

  5. Finally, run the whole thing under
     valgrind ./imgdump, and then under
     "setarch -R". Everything must still pass.
     (It will: none of the checks assume a
     particular address, only relationships. A
     check that depends on an absolute address
     is testing the randomiser, not the parser.)

Exercise 4 is the sharpest one in the course, and its answer is a real limitation rather than a triumph. /proc/<pid>/maps reports the permissions the kernel applied, which came from the file — so a check that compares the map against the file’s p_flags is not independent, it is circular. The honest resolution is that verifying the file requires reading the file, and the script does that separately with parse_elf64. Knowing which of your two sources is derived from which is the difference between a crosscheck and a tautology, and every claim in this course was chosen so that the two sides come from genuinely different places.

Exercise 5 is the last discipline and it is the one that keeps the tool useful. Every check is a relationship, so all eighteen survive setarch -R and a valgrind run without modification. A parser that only works at one address, or only outside an instrumentation tool, is a parser that works on your machine once. Being able to say “and it still passes with ASLR off” is a small thing that separates a script you ran from a tool you trust.

Connect

This is the concept that completes the chain, and the completion is worth stating plainly. The mission of this collection is parsing → a working executable, without depending on a tool that already knows the answer. This course ends with a program that reads the running process’s own description of itself, using no tool, and verifies every field against the file the kernel loaded. That is the same shape as reading a program header table in module one of the ELF course, with the file replaced by a live kernel and the header replaced by an array on a stack.

The forward dependency is exact. Rebuilding the Scope Yourself asks the reader to read DT_NEEDED out of a running process and walk the loader’s search order — and the first thing that exercise needs is AT_PHDR, which is what this artifact produces. So the dyn course’s hardest exercise is this course’s starting point, and the two are one artifact at two layers. That is the chain working as the mission describes it: a course must neither require a link the reader has not been given, nor re-teach a neighbour.

The connection back to the relocations course’s central measurement is the deepest one. That course proved the same source needs one relocation per datum on x86-64 and two on AArch64, checked programmatically. This course shows why the difference exists: on AArch64 the address is a 64-bit quantity formed from two instructions far apart, while the kernel here hands the program a single already-computed base. The kernel’s choice of what to put in the auxv is what makes one relocation enough — if the kernel handed over a page number and an offset separately, every datum would need two. The ABI choices of two different machines fall out of the same design pressure, and this is the layer where you can see it.

Two connections to the platform itself, since this course ends the executable chain. The linker-script concepts taught ld --verbose as a file you can edit, and the PT_LOAD rows that script emits are the requests this course watched the kernel grant. And the Mach-O loader concept in the sibling course covers the same kernel job on a different kernel — dyld reads a load command instead of a program header table, and gets the same information by a different route, because the kernel still has to tell it where everything landed.

And the honest note about where this stops. There is no kernel source in this course, and the research.md says so. Every claim here is observational: the auxv was measured, the vDSO was read, the floor was found, the fault counts were counted. Why the kernel chose 64 KB as its floor is explained from the attack it prevents, not from the code that implements it. That distinction is the course’s own discipline applied to itself, and it is stated in the research log rather than hidden, because a claim that has not been verified should be visible as unverified rather than quietly smoothed over into prose.

render_lesson_js(&mut page)