Keyboard Shortcuts

Ctrl+K
Open search
Esc
Close search / dialog
↑
Back to top

The One Place the ABI Describes Itself

25 minadvancedModule 1: The Frame: What a Call Owes WhomContinues from x86-saved
25 min · The x86-64 ABI

Why this matters

Every other rule in this ABI is implicit. The callee knows how many arguments there are because the caller and the callee were compiled from the same declaration, so the two of them share something — a prototype — and the registers are just the places they agreed to put them. Nobody has to be told anything at run time.

A variadic function breaks that completely. printf does not know how many arguments it was called with, or what they are, and neither does the code that walks the list. Something has to carry the information, and the ABI's answer is a small block of stack that the callee fills in before it can read anything, plus one integer in %rax that says how much of it is worth filling in.

That is the only self-describing convention in the whole of x86-64, and it is worth looking at closely for two reasons. It is the mechanism behind the single most-used function in the C library. And it is the one place where a caller can be wrong in a way that produces a plausible number that nobody passed rather than a crash — which makes it the best demonstration in this course of what a contract violation actually looks like when the contract is about meaning rather than about memory.

The model: 176 bytes and one integer

The register save area is a fixed 176-byte block, and the division is not arbitrary: 48 bytes for the six integer registers at 8 bytes each, and 128 bytes for the eight SSE registers at 16 bytes each.

$ ./abidump | sed -n '/1B\./,/^  the two sequences/p' | tail -6
  the return value             rax; rdx for a second integer; xmm0
                               for a double; xmm0:xmm1 for a long double
  the stack alignment constant 16 bytes
  %rsp at a callee's entry     8 (mod 16), because `call` pushed 8
  the red zone                 128 bytes below %rsp, LEAF functions
  the varargs save area        176 bytes = 48 integer + 128 SSE
  the initial gp_offset        8
  the initial fp_offset        48
  the vector count in %al      0 .. 8
  

And here is the shape of the whole mechanism, which is a two-field cursor and a pointer:

THE va_list, IN THE TERMS THE ABI DEFINES gp_offset where to read the NEXT integer argument, as a byte offset into the register save area. Starts at 8, because slot 0 is the register the named parameter already took. fp_offset the same for SSE registers. Starts at 48, which is where the FP half begins. overflow_arg_area where the 7th and later arguments live, on the stack. reg_save_area the 176-byte block itself. And in a register the ABI does not put in the struct: %al, the number of vector registers the CALLER used.

So a variadic callee does two things. It fills in the save area — six integer registers unconditionally, and the SSE registers only if %al is non-zero. And it sets the two cursors to 8 and 48. Then va_arg is arithmetic: read the field, add the size of the type, write the field back.

Here is a real one, compiled at -O0 so that every store is still in the listing, and read back:

$ gcc -O0 -S -o - abidump.c && sed -n '/^abi_take2:/,/^\.L190:/p' abidump.s
abi_take2:
	.cfi_startproc
	endbr64
	pushq	%rbp
	movq	%rsp, %rbp
	subq	$256, %rsp
	movq	%rdi, -248(%rbp)        -> the NAMED parameter, elsewhere
	movq	%rsi, -168(%rbp)        -> the first vararg register
	movq	%rdx, -160(%rbp)
	movq	%rcx, -152(%rbp)
	movq	%r8, -144(%rbp)
	movq	%r9, -136(%rbp)         -> the sixth, and the LAST
	testb	%al, %al               -> %al: is it ZERO?
	je	.L190                  -> if so, do not spill xmm at all
	movaps	%xmm0, -128(%rbp)     -> the FP half begins here
	movaps	%xmm1, -112(%rbp)
	movaps	%xmm2, -96(%rbp)
	movaps	%xmm3, -80(%rbp)
	movaps	%xmm4, -64(%rbp)
	movaps	%xmm5, -48(%rbp)
	movaps	%xmm6, -32(%rbp)
	movaps	%xmm7, -16(%rbp)      -> eight of them, 16 bytes each
.L190:
	movq	%fs:40, %rax           -> the stack protector canary,
	movq	%rax, -184(%rbp)          which is a DIFFERENT convention
	movl	$8, -208(%rbp)        -> gp_offset  = 8
	movl	$48, -204(%rbp)       -> fp_offset  = 48
	leaq	16(%rbp), %rax        -> overflow_arg_area
	movq	%rax, -200(%rbp)
	leaq	-176(%rbp), %rax      -> reg_save_area, the 176 bytes
	movq	%rax, -192(%rbp)
	movl	-204(%rbp), %eax
	cmpl	$175, %eax            -> will the next 16 bytes fit?
	ja	.L191                    -> no: read the stack instead
  

Three things to read out of that, and one of them is a surprise.

The reality: a caller that lies about %al

Here is the experiment. The caller puts two doubles in %xmm0 and %xmm1, and then puts a chosen value in %al — the number of vector registers it claims to have used. The consumer is an ordinary variadic C function using va_start and va_arg, so the whole thing is compiled by gcc and the prologue above is the code that runs.

$ ./abidump | sed -n '/The caller claims a number/,/arm 4/p'
  honest, arm 1: 1.5 and 2.5
    al=2  a=0x3ff8000000000000 (1.500000)  b=0x4004000000000000 (2.500000)
  honest, arm 2: 101.25 and 202.5 -- these are now what the save
                area holds, and arm 3 is where it stops mattering
                what the caller put in xmm0 and xmm1.
    al=2  a=0x4059500000000000 (101.250000)  b=0x4069500000000000 (202.500000)
  LYING, arm 3: the caller puts 3.5 and 4.5 in xmm0 and xmm1 and
               then says it used NO vector registers:
    al=0  a=0x4059500000000000 (101.250000)  b=0x000000000000000a (0.000000)
  al=1, arm 4: the same lie, understated rather than zero:
    al=1  a=0x400c000000000000 (3.500000)  b=0x4012000000000000 (4.500000)
  

Read arm 3 again, because the first number is the finding. The caller passed 3.5 and 4.5. The consumer returned 0x4059500000000000, which is 101.25 — exactly what arm 2 passed, bit for bit, and not what this arm passed. The second came back 0x000000000000000a, which is a fragment of a pointer and not a number anybody passed to anything.

WHAT ACTUALLY HAPPENED IN ARM 3 NOT: the arguments were corrupted in transit. YES: the arguments were never looked at. `al` was 0, so the callee skipped all eight `movaps` stores. The 176-byte save area was therefore never written, and `va_arg(ap, double)` read whatever was already in those 128 bytes -- which was the last HONEST call's data, and beyond it, stack. The value 0x4059500000000000 is not a corrupted 3.5. It is a perfectly intact 101.25 that was never on its way anywhere.

That is a much better failure than a crash, and it is the reason this concept is on a page about contracts. A crash tells you where to look. A plausible number tells you nothing, and it will be printed into a log, compared against a tolerance, and believed.

And it is not only your own code that reads it

The same experiment through glibc's printf, which is a variadic consumer and obeys the same rule:

$ ./abidump | sed -n '/AND THE SAME FAILURE/,/wrong answer,/p'
glibc printf, al=2: 1.500000 2.500000
glibc printf, al=2: 101.250000 202.500000
  and now the same call with al=0, having been passed 3.5 and 4.5:
glibc printf, al=0: 101.250000 0.000000

  A number nobody passed, printed as though somebody had.  That is not
  a slow path and not a portability warning: it is a wrong answer,
  produced by a caller that broke one instruction's contract.
  

So the mechanism is not an academic corner of the convention. It is the code path of every printf call with a floating-point argument in it, and the reason %al exists at all.

And arm 4 is the surprise, and it is a correction

The specification says %al is the number of vector registers used. Arm 4 claims one and passes two, which is a lie in the strict reading — and it works, because gcc tests %al for zero and spills all eight when it is non-zero. So on this compiler the field is a boolean and the specification's stricter reading would be a lie that happens to be harmless.

That is worth stating as what it is rather than as a contradiction: a self-describing convention is still a convention. The description is there, and nothing checks it. Any callee is entitled to use %al as an upper bound — to trust it and read only that many registers — and one that does will break on arm 4 while working perfectly on arm 3. The value of the mechanism is that it makes the common case safe; the cost is that the common case and the correct case are not the same case.

Worked: the save area, from the specification to the addresses

The layout deserves to be written out once, because it is short and everything about varargs is a consequence of it:

THE 176 BYTES, AT THE OFFSETS GCC ACTUALLY USES reg_save_area = -176(%rbp) in the listing above, so the offsets are measured from there: base + 0 .. 7 rdi's slot. UNUSED here, because rdi is the named parameter and already has a home at -248(%rbp). That is the whole reason gp_offset STARTS AT 8. base + 8 .. 15 rsi -> -168(%rbp) base + 16 .. 23 rdx -> -160(%rbp) base + 24 .. 31 rcx -> -152(%rbp) base + 32 .. 39 r8 -> -144(%rbp) base + 40 .. 47 r9 -> -136(%rbp) base + 48 .. 63 xmm0 -> -128(%rbp) --> 48 is base + 64 .. 79 xmm1 -> -112(%rbp) where fp_ base + 80 .. 95 xmm2 -> -96(%rbp) offset base + 96 .. 111 xmm3 -> -80(%rbp) starts base +112 .. 127 xmm4 -> -64(%rbp) base +128 .. 143 xmm5 -> -48(%rbp) base +144 .. 159 xmm6 -> -32(%rbp) base +160 .. 175 xmm7 -> -16(%rbp) 176 bytes total, and the last legal byte is base+175, which is the number the `cmpl $175` above compares against.

Two consequences fall out of the layout and both are worth stating as rules rather than as trivia.

The movaps stores are aligned because the area is 16-byte aligned, and the area is 16-byte aligned because of the stack alignment rule. That is why the two concepts are adjacent: a variadic function is the one function whose prologue depends on the caller's alignment, because the save area is a 16-byte-aligned object that the callee does not build — it inherits.

Saving eight XMM registers costs 128 bytes of store and, when %al is zero, is skipped entirely. The skip is the optimisation that made the mechanism cheap, and it is also the entire attack surface: a caller that says zero pays nothing and the callee reads nothing. An implementation that trusted %al as an exact count would save four stores in the common case where one float is passed, at the cost of a second pass over the save area for the rest. gcc's author chose the branch.

And the arithmetic of va_arg itself is worth one sentence, because it explains the cmpl $175 in the listing: before reading, the callee compares the SSE cursor against 175 — the last byte of the 176-byte area — and if the next 16-byte argument would not fit, it reads from overflow_arg_area instead. That is the boundary between “passed in a register” and “passed on the stack”, and it is the same boundary the first concept measured at stack=0x10. The rest of va_arg is the same idea: load the cursor, add the size of the type, store the cursor back, read through the pointer.

Apply it

  1. Make the lie yourself. Copy the VARARG_CALL macro out of abidump.c into your own file, change the al value, and watch which arms come back wrong. (Expect al=0 to lose the FP arguments and al=1 to keep them on gcc, and expect the values you get back to be the previous call's rather than zeroes. That second half is the part that makes it a security-shaped bug rather than a numerical one: the value is not garbage, it is someone else's data.)
  2. Find the guard in a real binary.objdump -d any variadic function and look for testb %al,%al above a block of eight movaps stores. (Expect to find it in printf's implementation, in snprintf, in every libc routine that takes a floating-point variadic argument, and in every program your compiler has ever produced that includes <stdarg.h>. Then change al to 1 in a hand-written caller and see the ones that trust the count fail and the ones that test for zero survive.)
  3. Work out the save-area offsets for a function whose first parameter is a double.(Expect gp_offset to start at 0 rather than 8, because the first integer register is still available, and fp_offset to start at 64 rather than 48. This is the clearest demonstration that the two cursors really are independent counters, and it is the same fact the first concept measured as mx in the disassembly.)
  4. Break the alignment underneath a variadic function and watch its own prologue fault. Call a variadic function through a hand-written caller that has deleted the subq $8, %rsp from its outgoing-argument area. (Expect a SIGSEGV inside the callee's movaps, before a single argument is read, and expect the fault address to be 8 bytes off a 16-byte boundary. That is the alignment fault wearing a different hat, and it is the reason the two concepts are in the same module.)
  5. Count the bytes. A variadic function with eight float arguments saves 128 bytes of SSE registers plus 48 of integer ones whether or not any integer was passed. (Expect 176 bytes of frame that a fixed-arity function with the same eight parameters would not have, because a fixed-arity function knows the types and puts the ninth-and-later arguments straight on the stack with no save area at all. The convention pays 176 bytes for the privilege of not knowing, and that is the trade the next ABI makes differently.)

Connect

Backwards, the calling convention is where the independence of the two register sequences was measured, and this concept is the only place in the ABI where that independence is load-bearing: the two cursors in the va_list are the same two counters, and a function whose first parameter is a double has an integer cursor starting at 0 and an SSE cursor starting at 64. The stack frame is where the movaps stores in that prologue get their alignment, and a reader who has read it will recognise -128(%rbp) as the first slot of a 16-byte-aligned object rather than as a coincidence of the frame size.

Forwards, the %al finding is the last retraction in this module and it is a general one: a convention that describes itself is not thereby checked. The unwinding concept is the other place in this ABI where the hardware is told something it could have worked out, and the parallel is exact — %al is information the caller has and the callee cannot derive, and the .eh_frame table is information the compiler has and the profiler cannot derive. Both are conventions rather than instructions, and both are paid for in bytes.

Outward, the neutral course that owns the why of dynamic dispatch across a language boundary. This concept is about how a C caller tells a C callee what it passed; it is not about what happens when the two sides were compiled by different compilers in different languages, and that is the dynamic-linking course's runtime-order concept and its interposition concept. The one thing worth carrying across is the shape of the failure: a boundary that disagrees about a signature does not crash, it returns a value from a different conversation.

render_lesson_js(&mut page)