Keyboard Shortcuts

Ctrl+K
Open search
Esc
Close search / dialog
↑
Back to top

What Ring 3 Can See, and What It Cannot

25 minadvancedModule 3: Time, and the Limits of What a User Process Can CountContinues from x86-pmu
25 min · The x86-64 Machine

Why this concept exists

Eight concepts of reference, and the reader is left with a set of tables about a machine they are not running on. This concept is the inventory, and it has a property none of the others has: every row in it was produced by this process on this machine, and every row is either a bit pattern read out of a register or a signal number that killed a forked child.

That is worth having on its own terms even if you will never read another word about the control registers. A user program that wants to know something about its own machine has a finite, enumerable list of things it can ask, and the list is shorter than most people assume, longer than most people guess, and the boundary of it is not where the documentation says.

The model: the harness, and the one bug that made it a different instrument

Every fault in this course is a signal, and the parent has to learn the signal's si_code and si_addr from a child. How it learns them is the whole design, and the first version got it wrong in the way that is most expensive to notice.

THE WRONG VERSION, and its output. static int code; /* a plain global */ static void handler(int sig, siginfo_t *si, void *uc) { code = si->si_code; /* runs in the CHILD */ _exit(3); } The parent then reads `code` after waitpid(). The handler ran in the child's address space. The parent's copy of `code` was never written. RESULT: si_code 0 for all fifty rows. And 0 is not an arbitrary wrong answer. 0 is a plausible one. Every row that should have said 128 said 0, and a table full of zeroes reads like a result -- "the kernel did not report a code for any of these" -- and would have been published.
THE RIGHT VERSION, which is three lines longer and cannot be got wrong by accident. static rec_t *R; /* mmap MAP_SHARED */ static void handler(int sig, siginfo_t *si, void *uc) { R->sig = sig; R->code = si->si_code; R->addr = si->si_addr; R->used = 1; _exit(3); } A shared page is the only channel that crosses a fork, and `used` is what tells the parent the child reached the handler at all rather than exiting normally by accident. A harness without a `used` flag cannot tell "the instruction returned" from "the instruction was deleted" -- and the divide-by-zero arm of section 4 was deleted by gcc, and the first version of this file would have reported it as a clean run.

The artifact prints both halves of that story in section 3, next to the table, because a harness failure is part of the instrument and a course that reports only its final table has hidden the part of the method that a reader most needs to check.

The reality: the inventory

First the can column. Ten instructions, each in its own forked child, each reported by name:

$ ./sysdump | grep '^  EDGEREAD'
  EDGEREAD | sidt                           | si_code 0 | readable
  EDGEREAD | sgdt                           | si_code 0 | readable
  EDGEREAD | sldt                           | si_code 0 | readable
  EDGEREAD | str                            | si_code 0 | readable
  EDGEREAD | smsw                           | si_code 0 | readable
  EDGEREAD | rdtsc                          | si_code 0 | readable
  EDGEREAD | rdtscp                         | si_code 0 | readable
  EDGEREAD | xgetbv                         | si_code 0 | readable
  EDGEREAD | rdfsbase                       | si_code 0 | readable
  EDGEREAD | wrfsbase                       | si_code 0 | readable
                

And the shape of the cannot column, which is the number and one more thing:

$ ./sysdump | grep '^  EDGE | rows'
  EDGE | rows 50 | faulted 39 | returned 11 | si_code 128 34 | other 5
                

Fifty instructions, thirty-nine faults, eleven returns — and the harness asserts the two sets are disjoint, because a partition is a claim a check can make and a count cannot.

Then the three absences, each proved rather than quoted

$ ./sysdump | sed -n '/6C\./,/XCR0 = /p'
  6C. THERE IS NO AVX-512, WHICH IS A DIFFERENT KIND OF ABSENCE,
      because it is a FEATURE BIT rather than a permission.

  ABSENT | CPUID.7.0:EBX AVX512F 0 DQ 0 CD 0 BW 0 VL 0 | ALL ZERO
  ABSENT | XCR0 = 0x0000000000000207, and the AVX-512 STATE bits of XCR0
  ABSENT | (5 opmask, 6 zmm_hi256, 7 hi16_zmm) are all 0, so even
  ABSENT | the state is not merely unsupported, it is not enabled.
  ABSENT | A feature bit is a FACT ABOUT THE CPU that any process
  ABSENT | can read with an unprivileged instruction, which is the
  ABSENT | opposite of a permission: the absence of AVX-512 is
  ABSENT | DECLARED BY THE HARDWARE, and the absence of the PMU is
  ABSENT | imposed by software.  Two absences, two different kinds.
                

That is the concept's whole argument in one paragraph, and it is a distinction rather than a number. An absence of hardware is declared by the hardware, in a register any process may read. An absence of access is imposed by software, in a file and a bit and a file mode. The first is a fact about the CPU and identical on every machine of that model. The second is a fact about this machine, today, as configured, and it can change without the CPU changing — which is why the performance-counter absence needed a fork to establish and the AVX-512 absence did not need a fork at all.

Worked: the ten retractions, and what each one cost

Every one is printed by the artifact and asserted as text by the harness, so a taken-back claim can be neither dropped nor edited into being right. They are worth reading as a list because the pattern of them is the argument for the whole collection:

$ ./sysdump | sed -n '/^   R1\./,/^   R3\./p'
   R1. "A ring-3 process cannot read the IDT because the IDT is
       privileged."  RETRACTED AND REPLACED.  The REGISTER is
       readable: SIDT returns base and limit from ring 3, and section
       2 prints them.  What cannot be read is one byte of the TABLE,
       and the reason is not privilege: the fault is si_code 1,
       SEGV_MAPERR, with si_addr EXACTLY the base SIDT printed.  A
       #GP would have been si_code 128 and si_addr 0, which is what
       every privileged instruction in section 3 produced.  The right
       sentence is a PAGE-TABLE sentence, and it predicts the address
       the first sentence could not.

   R2. "The 16-byte arm of the canonical sweep measures the
       canonical boundary."  RETRACTED.  It measured an ALIGNMENT
       requirement, because gcc compiled the __int128 load to
       `movdqa`, and the sweep then reported si_code 128 for every
       unaligned address at every offset, which looks like a rule and
       is not one.  Written in inline `movdqu` the same sweep gives
       2^47 - 15, the value the rule predicts, and section 5 prints
       BOTH arms side by side so that the retraction has numbers.
                

Two of the ten are not about the machine at all, and they are the two that a reader is most likely to repeat.

And the two that are about reading a document correctly, which is where a reference course lives or dies:

THE SHAPE OF ALL TEN, which is the argument for the whole collection and the reason the harness asserts them as text. R1 a claim about PRIVILEGE that was really about a MAPPING R2 a compiler quietly changed what was being measured R3 a harness asked for something the machine cannot provide R4 the harness produced a wrong ANSWER that agreed with a story R5 the right number read from the wrong field R6 a constant stated as a boundary when it is a function of one R7 an absence of ACCESS reported as an absence of HARDWARE R8 a field the KERNEL fills reported as a field the CPU fills R9 two different protections reported as distinguishable R10 two different bits reported as one because they share a number Not one of them is a mistake about how a computer works. Every single one is a mistake about what a NUMBER is.

Apply it

  1. Reproduce the inventory.cd courses/x86sys/assets/samples && ./build_samples.sh, then read section 3. (Expect 50 rows, 39 faults, 11 returns, and the two sets disjoint. If a row flips, the most likely causes in order are a seccomp filter, a hardened kernel with CR4.UMIP set, and a different vendor — and the first of those is worth chasing, because a seccomp filter that returns EPERM instead of a signal would break the harness's assumption that a refusal is a fault.)
  2. Break the shared-page harness. Replace the MAP_SHARED allocation with a plain global, exactly as the first version was, and re-run. (Expect every si_code to read 0 and the harness to fail on the #GP checks. Then read what the table now claims: “the kernel reported no code for any privileged instruction” is a sentence you could publish, and it is the sentence this bug produces. That is why the failure mode of a measurement harness is part of the measurement.)
  3. Count your own machine's inventory. Write a program that runs the same fifty instructions and reports which of them fault on your kernel. (Expect the privileged set to be stable across kernels of the same vintage and the SIDT/SGDT/STR set to be the part that varies, because it depends on CR4.UMIP and a policy decision. A list of what a user program can do is a list of a kernel's configuration, and it is a list that changes between distributions.)
  4. Check whether a claim you are making is a consequence or a reading. Take the last five technical claims you have written or said and ask, for each, “did I observe this, or did I infer it from something I observed?” (Expect most of them to be consequences, which is fine and is what most engineering is. Expect one or two to be readings dressed as consequences, and those are the ones that are wrong with a confidence that will not help you.)
  5. Prove an absence your own toolchain depends on. Take a tool you use every day and find the machine property it silently requires. (Expect the answer to be a CPUID bit, a /sys file, or a syscall return value — that is, a fact about the machine that the tool assumes rather than checks. Then check it, and if it is not there, say so. A tool that reports a number derived from a property it never verified is making a claim about the machine and has not said so.)

Connect, and the last thing

Backwards, this concept has no single parent; it is the sum of the first eight, and the table it prints is the argument that the sum is a list rather than a pile. The privilege course's harness concept is where the one measurement this course builds on — SIDT returns and the IDT does not — was taken, and the difference is that this course made that asymmetry into fifty instructions and then into a partition the harness can check.

Forwards, and there is a course after this one. The fourth course in this section is about the data path, and it inherits a specific debt from the eighth concept here: the AVX-512 feature bits that read as all zero are the same zero that course has to work around, and the movdqa trap that retraction R2 caught is the same class of problem as a misaligned vector store being a fault rather than a slowdown. A course that teaches a data path on a machine that lacks half the instructions in it has to be explicit about which half, and this course is where the explicit part lives.

And the last thing, which is the limits block and which is not a footnote. The artifact prints nine things it cannot tell you, each one naming the instrument that would settle it:

$ ./sysdump | sed -n '/^  LIMITS/,/second machine/p' | head -40
  LIMITS -- what this artifact cannot tell you, printed not footnoted,
  and each one names the instrument that would settle it:
    * ANY CYCLE COUNT.  There is no counter this process can open:
      RDPMC is a #GP, perf_event_open is EACCES, and the kernel
      reports perf_event_paranoid above.  Every duration in this file
      is a DURATION, and a duration bounds a count without measuring
      it.  Settled by perf_event_open with a paranoid setting below 4
      and a PMU passthrough from the host.
    * WHETHER CR0 TO CR4 HOLD THE VALUES THE KERNEL REPORTS.
      Section 2 prints arch_prctl's SMEP/SMAP/SHSTK bits and section
      3 measures that every read of CR0-CR4 and CR0-CR8 is a fault.
      A consequence is not a reading.  Settled by a CPL 0 program.
                

Each limit ends in a sentence of the form settled by X, and that is the part worth copying. A limit that says “this was not measured” is a dead end. A limit that says “a twenty-line program at CPL 0 would settle it” is a work item, and it tells a reader exactly what they are missing in order to do the work themselves. Four of the nine gaps in this course are properties of the instrument rather than of the subject, and the last one — whether a second machine agrees — is the one no amount of cleverness on one machine can fix.

The verdict the artifact prints is the sentence to keep:

What this artifact can support: an inventory of what a ring-3 process can and cannot learn about the machine it is running on, with every entry of the CANNOT column a signal number rather than a quotation, and every entry of the CAN column a bit pattern read out of a register on this CPU. What it cannot support: any claim about a control register's value, any cycle count, any exception's class, and any statement about a second machine. Four of those five are properties of the instrument, not of the subject.

That last clause is the course's real conclusion and it is worth more than any table here. Four of the five things this artifact cannot tell you are limitations of a program running at CPL 3, and one of them — a second machine — is a limitation of being a program at all. Every claim in this collection that is marked INFERRED, every mechanism that is named rather than measured, and every QUOTED row in every table is the same fact: the measurement decides what may be claimed, and a course that hides which of its numbers are measurements is teaching a reader to trust the wrong sentences.

render_lesson_js(&mut page)