Font: Spacing: Letters: Width:

SYSCALL and the Four Registers It Touches

25 minadvancedModule 1: The Machine's Privileged Side, as an Inventory

Why this matters

There is exactly one instruction in long mode that a user process can execute to reach the kernel, and it is one byte long. Everything else on the privileged side of the boundary refuses. That asymmetry — one instruction in, fifty instructions out — is the shape of the whole machine, and a reference that lists the privileged instructions without measuring that asymmetry is a list and not a reference.

syscall has a small, exact, checkable promise: on entry it writes the address of the following instruction into RCX and the RFLAGS it saved into R11, and the kernel entry path restores the rest on the way back. Two registers destroyed, and the sentence is worth having only if you know it is two and not fifteen. The measurement below puts thirteen markers in thirteen registers and reads them back across a real Linux system call.

The model: the convention is two rows long, and the ISA is the only part of it that is architecture

Print the convention first, because it is four lines and every one of them is a decision somebody made:

THE x86-64 SYSTEM CALL CONVENTION RAX the system call NUMBER going in, and the RETURN VALUE coming out RDI RSI RDX R10 R8 R9 the six arguments, and the fourth one is R10 and not RCX RCX DESTROYED: the address of the instruction after the `syscall` R11 DESTROYED: the RFLAGS saved on the way in CF NOT used on x86-64. A negative value in RAX IS the error. The clear-carry convention is i386 only. Of the twelve rows, TWO are the architecture. The other ten are LINUX, and `grep "system call number" ` in the Intel SDM finds nothing.

That split is the whole of why the fourth argument is R10. RCX is busy holding the return RIP, and the SysV function call convention had already given RCX to the fourth integer argument. The two conventions collided and Linux renumbered around it. The ABI course measured the function side — sixteen of sixteen callers at four optimisation levels put argument four in RCX — and this course measures the system-call side, so the contrast is two measurements rather than a story.

The reality: thirteen markers, one system call, and the two that had to be put first

Here is the whole table. A distinct marker goes into each of the thirteen registers other thanRAX and RCX, one syscall executes, and each register is read back in the same inline-assembly block before the compiler has had a chance to touch anything:

$ ./sysdump | sed -n '/^  SYSCALL | register/,/^  SYSCALL | r15/p'
  SYSCALL | register | marker in | marker out | verdict
  SYSCALL | rbx | 0xb000000000000010 | 0xb000000000000010 | SURVIVED
  SYSCALL | rdx | 0xb000000000000020 | 0xb000000000000020 | SURVIVED
  SYSCALL | rsi | 0xb000000000000030 | 0xb000000000000030 | SURVIVED
  SYSCALL | rdi | 0xb000000000000040 | 0xb000000000000040 | SURVIVED
  SYSCALL | rbp | 0xb000000000000050 | 0xb000000000000050 | SURVIVED
  SYSCALL | r8  | 0xb000000000000060 | 0xb000000000000060 | SURVIVED
  SYSCALL | r9  | 0xb000000000000070 | 0xb000000000000070 | SURVIVED
  SYSCALL | r10 | 0xb000000000000080 | 0xb000000000000080 | SURVIVED
  SYSCALL | r11 | 0xb000000000000090 | 0x0000000000000202 | CLOBBERED
  SYSCALL | r12 | 0xb0000000000000a0 | 0xb0000000000000a0 | SURVIVED
  SYSCALL | r13 | 0xb0000000000000b0 | 0xb0000000000000b0 | SURVIVED
  SYSCALL | r14 | 0xb0000000000000c0 | 0xb0000000000000c0 | SURVIVED
  SYSCALL | r15 | 0xb0000000000000d0 | 0xb0000000000000d0 | SURVIVED
  SYSCALL | arms 26 | survived 24 | clobbered 2

Two arms of the run, thirteen registers each, and exactly one register per run came back clobbered: R11, and its value is 0x202 — an RFLAGS word with bit 1 set, which is the bit the ISA requires to be set in every RFLAGS and is what makes the saved flags recognisable as flags. (Bit 2 is the parity flag, so the low three digits of this number depend on the parity of the process id that came back in RAX and the two recorded runs differ there. Quote the relation, not the word.)

Twelve of thirteen survive, across two arms. That is the ISA's promise and Linux's convention agreeing, and it is a measurement rather than a quotation because the markers are distinct in every digit and the compiler is not allowed to touch the registers the block names in its clobber list.
RCX is deliberately absent from the table. The ISA says it holds the return RIP and the ABI course measured that bit for bit. A row for RCX here would have said “clobbered” and taught nothing; a row reading “holds the return address” would have restated another course's measurement, which is the mistake this collection's harnesses exist to prevent.
The markers are pasted into the assembly text, not passed as operands. A marker passed as an "m" constraint and typed as 0x0b00000000000008 instead of 0x0b000000000000080 compiles, runs, and prints a verdict that is right for the wrong number — the two agree in their low fourteen digits. Two levels of stringification cost nothing.

Why there are three blocks and not one

The first version of this measurement asked for fourteen outputs in a single asm statement. It cannot be written: once RAX and RCX are clobbered there are fewer than fourteen allocatable registers, so gcc allocated one for a compiler value, and the block died with SIGBUS because it had overwritten the address of one of its own output slots.

$ ./sysdump | sed -n '/WHY THREE BLOCKS/,/overwritten/p'
  SYSCALL | WHY THREE BLOCKS AND NOT ONE, which is R3 in section 7:
  SYSCALL | there are fewer than fourteen allocatable registers once
  SYSCALL | rax and rcx are clobbered, so a fourteen-output asm block
  SYSCALL | cannot be written at all; the first version asked for one
  SYSCALL | and died with SIGBUS, because the compiler was holding
  SYSCALL | the ADDRESS of one of the fourteen output slots in a
  SYSCALL | register the body had already overwritten.

And that bug produced a wrong answer that was still wrong after the fix was not yet applied, which is the subject of retraction R4: a first table reported all fifteen registers clobbered, because the clobber list named six registers the body touched and said nothing about the other seven. A wrong result that agrees with a story you have already written down is the hardest kind to catch, and this one did.

THE GENERAL FORM, and it is the ninth time this collection has learned it. A clobber list is a CLAIM about what your assembly modifies. A register the COMPILER allocated is not one your assembly named. Three consequences, in order of how often they have cost somebody a day: 1. A register your body WRITES and does not name is a wrong result. 2. A register your body COUNTS with and does not name is a wrong result that only appears on the SECOND iteration. 3. A register gcc allocated to HOLD one of your outputs is a SIGBUS, and the symptom is a crash in a program whose every other failure mode is a number.

Worked: the number, the error, and the six arguments that were ignored

Three arms, and the middle one is the interesting one because the wrapper is one of the things under test:

$ ./sysdump | sed -n '/getpid, six arguments/,/caller cannot vary/p'
    getpid, six arguments, RAW         = 1037820
    and getpid() through libc          = 1037820   same ? YES
    syscall number 9999, RAW           = -38, i.e. -ENOSYS
    and the same call through libc     = -1 with errno 38
    A bad number is an ERROR RETURN, not a fault: the kernel
    checked the number at the boundary, returned -ENOSYS in RAX,
    and on x86-64 it did NOT set CF.  The clear-carry-on-error
    convention is the i386 one.  And the SIX arguments were
    accepted and ignored: getpid takes none, the kernel reads the
    number of arguments from the call SITE'S declared arity, and
    this call site is libc's, which is why a caller cannot vary
    the cost of a system call by passing more arguments.

The raw value and libc's agree, which is the control that proves the syscall really was a system call and not a nop that happened to leave the markers alone.
An unknown number is a return, not a fault.-38 in RAX is -ENOSYS; libc turns that into -1 and sets errno to 38. A first version of this file printed libc's -1 and wrote a sentence about -ENOSYS beside a number that was not it — the same failure as a transcribed hex digit, in a different column.
Carry flag: not set. The clear-carry-on-error convention is the i386 one and it does not apply in long mode, which is one of those facts that is quoted in a great many places and costs one instruction to check.
Six arguments to a zero-argument call, accepted. The kernel reads the number of arguments from the call site's declared arity, and the call site is inside libc, so a caller cannot make a system call dearer by passing more arguments. The privilege course deleted an entire experiment that tried to vary this, and this file says why in a sentence rather than repeating the experiment.

What this concept cannot see

Three MSRs decide where a system call goes and what is cleared on the way in: IA32_STAR, IA32_LSTAR and IA32_SFMASK. RDMSR is a #GP from ring 3 — measured, in the third concept — and /dev/cpu/0/msr is root-only, also measured, in the eighth. So the register convention above is a convention about registers, and the values in the three MSRs are a convention about a memory the process cannot read. The privilege course got the same information from the vDSO's bytes, which is a weaker source and says so.

Apply it

Reproduce the table before you read anything else.cd courses/x86sys/assets/samples && ./build_samples.sh, then read section 2D. (Expect twelve survivors and one clobbered, and the clobbered one to be r11. If a second register comes back clobbered, do not assume the CPU changed: look at whether your build has a seccomp filter, an audit rule, or a different kernel, because all three can rewrite the register file on the way through.)
Add a fourteenth register and watch the artifact die. Take the three asm blocks in sysdump.c and put all thirteen markers into one block, with only the six you name in the clobber list. (Expect a SIGBUS, not a number. Then name all thirteen in the clobber list and the compiler runs out of registers and tells you so. Both failures are the same lesson and the second one is louder.)
Measure the R11 bit that is not parity. The 0x202 above has bit 1 set and bit 2 clear. Write the marker for R11 as the saved flags and print bit 9, the interrupt-enable flag, instead of the whole word. (Expect bit 9 set, because IF is a caller's flag and the entry path preserves it. Then clear IF before the syscall and confirm the kernel has put it back, which is a different claim and needs a different measurement.)
Find the collision in code you did not write. Take any C file with a hand-written assembly callee that makes system calls and check, for each call, what the code assumes about RCX and R11. (Expect to find at least one place that keeps a live value in RCX across a system call because the author was thinking about a function call. The two conventions share five of their six argument registers, which is exactly why the sixth is the interesting one.)
Count how many system call numbers your program can name. If you are writing a backend, the question is not “can I issue a system call” but “which numbers, and what does each cost, and does the answer change if the kernel has a vDSO path for it.& Expect the answer to be a table with three columns — number, arguments, and whether there is a faster path — and expect the third column to be the one that decides the design.

Connect

Backwards, the ABI course's calling-convention concept measured the function side of this same collision, and the privilege course's mode-change concept measured the cost of getting here at all: the same function through a syscall and through the vDSO differed by more than an order of magnitude, and that ratio is the cost of a privilege change with everything else held constant. The ISA course is where the 0F 05 encoding lives.

Forwards. The instruction brings you in; the next concept is what happens when the CPU wants to send you back out, and it is a table of thirty-two vectors rather than a convention. And the two registers this page could not see — STAR and LSTAR — are MSRs, which is the subject of the fourth concept, and the reading that would settle them is a #GP, which is the subject of the third.

Outward. The syscall entry path is the one place where a compiler backend's register allocation and the kernel's register convention are the same table, and the collision between them is why the fourth argument is R10. The next course in this section is about the data path rather than the machine, and it inherits the movdqa trap from the eighth concept here as its standing example of a compiler strengthening an instruction you asked for by name.

Keyboard Shortcuts