Keyboard Shortcuts

Ctrl+K
Open search
Esc
Close search / dialog
↑
Back to top

Read the ABI Out of the Disassembly

25 minadvancedModule 2: The Second Description: Unwinding and the AuditContinues from x86-unwind
25 min · The x86-64 ABI

Why this matters

This is the concept that decides which of the other five you are allowed to believe, and it is the one that turns the course's argument from a set of claims into a method.

An ABI is not a description. It is a contract between two compilers, and a contract between two compilers is the one kind of thing in this whole course that can be checked mechanically. The specification says which register argument four goes in; the compiler emits bytes; the two can be compared without a human being in the loop. That is worth having for a reason that is easy to miss: the specification is a document nobody re-reads, and the compiler is a program that changes every quarter. A rule that has never been checked is a rule that is a memory.

And there is a second thing this concept settles, which the previous five could not. Four of them produced a number that somebody could have typed in. The alignment result is a SIGSEGV; the red-zone result is a word read back out of the slot a call overwrote; the varargs result is a bit pattern that is bit-for-bit a different call's argument. Those do not move between runs and they do not depend on a tolerance. The audit below is in the same category: 64 of 64 is 64 of 64 on a busier machine and a busier day.

The model: the corpus, and the two things that had to be fixed

The audit needs a corpus in which the argument assignment is a fact about the emitted code rather than a fact about the optimiser's mood. Two things had to be arranged, and both were learned by getting them wrong.

The optimiser deletes the experiment

The first corpus marked its functions noinline, which stops the inliner and nothing else. At -O2 the bodies are pure, so gcc constant-folded every one of them: the calls vanished, call_each became a run of mov $0x7, %edi; call abi_sink, and the register assignment was no longer anywhere in the program. A census would still have printed six numbers. The numbers would have been about nothing.

The attribute that works is noipa, which forbids interprocedural analysis of any kind. That is not a detail of this course's corpus; it is the general shape of the problem, and it is the reason a microbenchmark written as ordinary C so often measures the wrong thing.

And the argument has to be identifiable

Even with noipa, gcc knows the value of every argument in the caller, so it propagates it and the register each argument lands in becomes an implementation detail of the propagation. So each argument's source is its own volatile global, and the corpus is linked before it is read: in a .o every RIP-relative load reads [rip+0x0] and the identity of the argument is not in the file at all. In a linked executable, objdump prints the symbol in the comment:

$ objdump -d --no-show-raw-insn -M intel ./x2 | sed -n '/<c_f7>:/,/^$/p'
0000000000000420 <c_f7>:
  420:	endbr64
  424:	sub    rsp,0x10
  428:	mov    rax,QWORD PTR [rip+0x0]        # 42f <c_f7+0xf>
  42f:	mov    r9,QWORD PTR [rip+0x0]        # 436 <c_f7+0x16>
  436:	mov    r8,QWORD PTR [rip+0x0]        # 43d <c_f7+0x1d>
  43d:	mov    rcx,QWORD PTR [rip+0x0]        # 444 <c_f7+0x24>
  444:	push   rax
  445:	mov    rdx,QWORD PTR [rip+0x0]        # 44c <c_f7+0x2c>
  44c:	mov    rsi,QWORD PTR [rip+0x0]        # 453 <c_f7+0x33>
  453:	mov    rdi,QWORD PTR [rip+0x0]        # 45a <c_f7+0x3a>
  45a:	call   45f <c_f7+0x3f>
  

That is the linked form. The reader resolves each mov to a symbol and gets a mapping from argument to register, and the seventh argument is visible as a push rather than a register. Recovering that from a .o is not possible, and the first version of the reader produced sixteen rows of (no arguments) on every optimisation level for exactly that reason.

One caller per callee, and never count

The third fix is smaller and is the seventh time in this collection that a helper written for one shape has been used on another. The first caller was one function with sixteen calls in it, and the reader identified each call site by counting — because a call to an external function is a relocation and objdump cannot name it. Counting assumed the compiler kept the calls in source order, which is an assumption about the optimiser inside the instrument that is supposed to be auditing the optimiser. One function per call site removes the assumption, and call f4 is a call to a symbol objdump can name.

The reality: three results, and one that is a version claim

Result one: the audit passes, everywhere

$ ./abidump | sed -n '/THE AUDIT, PER OPTIMISATION LEVEL/,/^$/p'
  level   callers audited   match the specification   differ
  -O0                 16                       16         0
  -O1                 16                       16         0
  -O2                 16                       16         0
  -Os                 16                       16         0
  

Sixty-four of sixty-four, and the harness asserts it as a partition rather than as a number: at every level, matched plus differing must equal audited, and differing must be zero. A partition is the right shape because it is the thing that catches a reader which silently stopped looking — a reader that returned nothing would report 0 of 16 rather than 16 of 16, and the partition makes that visible.

Result two: a real library, and the flags that are not violations

$ ./abidump | sed -n '/^  LIBRARY |/,/^  CLAPPERCONTEXT/p'
  LIBRARY | libc.a | 6238972 bytes
  FUNCTIONS | 4561 functions, and the register census is:
  CENSUS | rbx | 1670 | 1598
  CENSUS | rbp | 2555 | 2503
  CENSUS | r12 | 1242 | 1193
  CENSUS | r13 | 981 | 945
  CENSUS | r14 | 885 | 850
  CENSUS | r15 | 627 | 600
  CLOBBERFLAGS | 271 | 94 functions
  CLOBBERKIND | leaf | 26 | nonleaf | 245
  CLAPPERCONTEXT | 97 | 174
  

4,561 functions, and the second column of the census is the one that makes the flags column mean anything: 1,670 functions write%rbx and the checker examined 1,670 real cases. A zero in a violation column is only meaningful next to a non-zero in a census column, and that is the single most transferable sentence in this section.

The flags decompose, and the decomposition is the result. 26 of the 271 are in leaf functions, which may clobber a callee-saved register if the compiler knows no caller had a live value there. 97 are in functions whose contract is to install a saved register context and which are therefore required to destroy those registers. Both exemptions are real and neither is checkable from inside one function, which is the honest limit of an ABI linter and the reason a reader who reports all 271 as bugs is reporting the rule.

And they are not disjoint, which is the one piece of arithmetic in this course that was wrong before it was written. Three of the 271 are in functions that are both leaves and context restorers — setjmp and its relatives are exactly that shape — so the flags that are neither exemption total 151, not the 148 that 271 − 26 − 97 gives. The linter now reports the overlap as its own row so the subtraction cannot be performed by accident, and the artifact's tenth retraction is about this arithmetic rather than about glibc.

Result three, and it is a version claim rather than a fact

The brief for this course asked for at least one case where the compiler disagrees with the specification, or else a demonstration that it does not on this version. The honest answer is the second, and it must be labelled as the second. On gcc 15.2.0 / binutils 2.46, reading 4,561 functions of glibc's static library and a 16-call-site corpus at four optimisation levels, this course found no case where a non-leaf function writes a callee-saved register without saving it, and no case where the emitted argument assignment differs from the specification.

That is a version claim, not a law, and the difference is worth being pedantic about. It says what a checker saw on one machine on one day with one compiler. A new compiler version, a different distribution's build flags, or a function compiled by a hand-written assembly file are all outside it. What makes it worth having anyway is that it is cheap to re-run: ./build_samples.sh and the claim is re-checked in about twenty seconds, which is the property that a hand-inspection of a specification does not have.

THE CLAIM, STATED SO IT CAN BE FALSIFIED On gcc 15.2.0 and binutils 2.46, the System V AMD64 argument assignment emitted by this compiler for 16 call sites at -O0, -O1, -O2 and -Os matches the specification in 64 of 64 cases, and no non-leaf function among 4,561 in glibc's static library writes a callee-saved register without saving it. To falsify it: change the compiler, or add a hand-written callee that breaks the rule, or run the linter over a library built by a different distribution.

Worked: the harness, and the retraction that only a harness could catch

The harness for this course is 143 checks in nine groups, and it asserts structure exactly and timings as orderings — which on this machine means it asserts almost no numbers at all, because the results that matter here are faults, bit patterns and words of disassembly, and those do not move between runs.

It runs against a fresh run rather than against the recorded output, so “the claims still hold on this machine today” is a statement about the machine rather than a tautology about a checked-in file. The recorded output ships anyway, so the claims are checkable on a machine that never ran the benchmark.

And here is the retraction, which is the best bug in the course

The linter reported zero violations across 9,255 real functions of libc, libm, the dynamic loader, libcrypto and /bin/ls. Every one of those zeroes was hollow. The expression that normalises a register name had its ternary the wrong way round, so "r12" came back as "e12", nothing was ever recognised as a callee-saved register, and the check compared nothing against anything. The number looked perfect because it was empty, and the only reason it was empty was a bug.

$ ./abidump | sed -n '/THE POSITIVE CONTROL/,/flags everything/p'
  THE POSITIVE CONTROL, without which every zero above is hollow:

    FLAGGED   ct_clobber_r12
    FLAGGED   ct_clobber_r13
    FLAGGED   ct_clobber_rbx
    CLOBBERFLAGS | 3 | 3 functions
    ALIGNCALLS | 5 | 1   <- and the one intended alignment violation
    ct_ok_leaf and ct_ok_leaf4 are conforming and are NOT in that
    list, which is the other half of the control: a linter that
    flags everything is caught as loudly as one that flags nothing.
  

control.s exists because of that, and it is the most transferable thing in the course. Its every function is a violation on purpose, in a named way, and the harness requires the linter to flag all three of them and not to flag the two conforming ones. The second half matters as much as the first: a linter that flags everything is caught just as loudly as one that flags nothing, and both look like a working linter on a page of output.

The control was itself written twice, and the first version is worth recording because it fails in a way nobody expects. Written in C, the compiler repaired two of the three violations: it pushed %rbx around the movq on its own, and it deleted the leaq that was supposed to misalign the call. The compiler repairs what it can — which is the whole point of the file, stated differently, and the reason a positive control for an ABI checker has to be written in a language the compiler does not repair.

And the retraction that is about a checker, not about code

The linter's second half tracks %rsp and checks the 16-byte alignment at every call site, and it reported 775 of 9,980 call sites misaligned. The first draft of this course was going to print that as a finding: glibc keeps the stack aligned at 92 % of its call sites, and the rest is a list of bugs.

That number is a fact about the checker and not about glibc, and the artifact says so where it prints it. The tracker loses %rsp at pushfq, at enter, and at any write it does not model, and a checker that cannot account for a mismatch is not evidence of a mismatch. It is the seventh time in this collection that the same lesson has cost something, and the general form is the one worth carrying: when a checker finds a great many of something, the first question is not how many it found but how many it looked at. The census column exists precisely so that question has an answer.

Apply it

  1. Reproduce the whole thing.cd courses/x86abi/assets/samples && ./build_samples.sh. (Expect 143/143, and expect the harness to run against a fresh run so that the result is about your machine. If a group fails, the group name tells you which claim moved: A is the instrument, C is the audit, E is a fault, H is the linter. A failure in A means the machine is too noisy for the floor and everything after it is suspect.)
  2. Break the oracle and confirm it fails by name. Change one register in SPEC in mkabi.py and re-run. (Expect one row to report DIFFERS at all four levels, and the row to be the one you broke. Then change the audit rather than the oracle — make caller_map return nothing — and expect the partition check to fail rather than the total to fall silently, because matched plus differing must still equal audited.)
  3. Write a positive control for a checker you already have. Take any validation script you wrote for a compiler, a linker, a protocol or a parser, and build a file that must be rejected. (Expect the checker to pass on the control immediately, which means it was already right, and expect the experience of the linter's 9,255 hollow zeroes to make you doubt it: how would you know? The answer is a file built to be caught, and a second one built to be passed, and both must be in the repository next to the checker.)
  4. Add a violation to the linter's corpus and see whether it is one. Write an assembly function that clobbers %r12 without saving it, put it in control.s, and run. (Expect the flag, the function count to rise by one, and the census row for r12 to rise by one. Then write the same function as a leaf and note that the flag is the same while the legality is different — the checker's verdict is about the code in front of it and the exemption is about the code it cannot see.)
  5. Audit something that is not x86-64. Take the method, not the rules: choose a specification, transcribe it into a checker, build a corpus where the thing under test cannot be optimised away, and compare. (Expect the interesting part to be entirely in the corpus and the specification — the two things a course about an ABI teaches that generalise to any interface between two programs. Expect also to hit the limit this course hit: an interface with no independent account of what the output should be cannot be audited the way a register assignment can, and knowing which kind you are holding is the transferable skill.)

Connect

Backwards, this concept consumes all five. The calling convention supplied the oracle. The stack frame and the saved registers supplied the fault experiments and the census the linter reports. The varargs concept supplied the retraction that a self-describing convention is not a checked one. The unwinding concept supplied the limit — the part of the ABI that cannot be audited this way, and the reason being able to say which part cannot is part of the result.

Outward, the harness is the seventh instance of a pattern this collection has now established seven times, and the two nearest precedents are the most instructive. The memory course's verification concept is where a check whose threshold came from the recorded run was shown to fail on two other recorded runs of the same experiment on the same machine, and the fix was to assert the shape and not the value. The privilege course's harness concept is where a row was found below its own control and the subtraction that produced a negative cost was retracted rather than kept. Both are the same lesson this course's R6 is about from the other end: a checker that cannot account for a mismatch is not evidence of a mismatch.

And forward, into the rest of the section. The fourth course in this series is about the machine's privileged side, where the syscall instruction this course measured as a curiosity is the only door in, and where the argument registers are the same six minus one. And the assembly course that precedes this one in the section taught the reading this concept depends on: there is no way to recover an argument assignment from a disassembly without knowing that the destination is the operand on the left.

render_lesson_js(&mut page)