Font: Spacing: Letters: Width:

The 48-Bit Split, LA57 and LAM

25 minadvancedModule 2: The Address Space, and Which Bytes of It ExistContinues from x86-debug

Why this matters

Half of the 64-bit address space on this machine is not an address. Not “is not mapped” — not an address, in the sense that the CPU refuses to form one and no page table is ever consulted. That is a different kind of unavailability from an unmapped page, and it is the reason a fault handler that reads si_addr before si_code reports a null-pointer dereference for what was actually an arithmetic mistake.

This concept re-derives the rule on five access widths where the privilege course derived it on three, and the extension earned its place: it found a wrong constant in the first draft, and the constant was one that half the documentation quotes. The rule is first rejected = 2^47 - (access size - 1), and the constant is 2^64 - 2^47 and not0xffffffff80000000.

The model: bit 47 chooses the half, and nothing else does

A linear address is canonical when bit 47 equals bit 63. That is the whole rule, and it has an immediate consequence that is easy to get wrong: the two halves are each 2⁴⁷ bytes wide, because the split is down the middle of a 2⁴⁸-bit space and 2⁴⁸/2 is 2⁴⁷. The upper half therefore begins at 2⁶⁴ − 2⁴⁷.

THE TWO HALVES, and the arithmetic that settles them. low 0x0000000000000000 .. 0x00007fffffffffff HOLE 0x0000800000000000 .. 0xffff7fffffffffff upper 0xffff800000000000 .. 0xffffffffffffffff The hole is not one number, it is a RANGE, and it is 2^64 - 2^48 bytes wide out of 2^64: 93.75% of the address space is not an address on this machine. And note where the upper half STARTS. It is 2^64 - 2^47. It is NOT 0xffffffff80000000, which is 2^31 bytes INSIDE it and which appears in a great deal of documentation and in the GDT base of every Linux kernel you have ever read about.

That last paragraph is the correction, and the reason it survived so long is worth naming: 0xffffffff80000000 is a plausible-looking address that falls in the upper half, so a claim that “addresses starting with 0xffffffff8 are kernel addresses” is true while the claim that “the upper half starts at 0xffffffff80000000” is false. Two claims, one address, and the difference between them is 2³¹ bytes of address space that a program could use and cannot.

And the check itself, because getting it wrong is the commonest way to be wrong about x86-64 addressing:

IS CANONICAL(a) = ((a >> 47) & 1) == ((a >> 63) & 1) NOT (a & 0xFFFF000000000000) == 0 or (a & 0xFFFF000000000000) == 0xFFFF000000000000 The second form is what half the code on the internet writes, and it calls 0x0000800000000000 canonical. It is not. That address has bit 47 = 1 and bit 63 = 0, so it is in the hole. The first version of the artifact's own helper had this bug, and the artifact prints the correct test beside it.

The reality: five widths, and the row that is the whole concept

The privilege course measured this with 1-, 4- and 8-byte reads. This course adds a 2-byte and a 16-byte arm, and the 2-byte arm is the one that matters: a 1-byte read fails nowhere before the line and a 2-byte read fails one byte before it, which is a difference no rule stated as “the boundary is 2⁴⁷” can survive.

$ ./sysdump | sed -n '/^  MATRIX | 2\^47  -3/,/^  MATRIX | 2\^47  +0/p'
  MATRIX | 2^47  -3 | 0x00007ffffffffffd | 1       1       128     128     128
  MATRIX | 2^47  -2 | 0x00007ffffffffffe | 1       1       128     128     128
  MATRIX | 2^47  -1 | 0x00007fffffffffff | 1       128     128     128     128
  MATRIX | 2^47  +0 | 0x0000800000000000 | 128     128     128     128     128     NON-canonical

Columns left to right: 1, 2, 4, 8 and 16 bytes. Read the top row. A 1-byte read at 2⁴⁷−3 succeeds. A 2-byte read at the same address succeeds. A 4-byte read fails, because the four bytes it would touch run from 2⁴⁷−3 to 2⁴⁷, and the last of them is past the line. Same address, four different outcomes, decided entirely by how many bytes the instruction reads.

Then the rule, derived by bisection independently for each width:

$ ./sysdump | sed -n '/^  BISE | width/,/first rejected/p'
  BISE | width | first rejected address | 2^47-(size-1) | holds
  BISE | 1-byte | 0x0000800000000000          | 0x0000800000000000 | YES
  BISE | 2-byte | 0x00007fffffffffff          | 0x00007fffffffffff | YES
  BISE | 4-byte | 0x00007ffffffffffd          | 0x00007ffffffffffd | YES
  BISE | 8-byte | 0x00007ffffffffff9          | 0x00007ffffffffff9 | YES
  BISE | 16-byte | 0x00007ffffffffff1          | 0x00007ffffffffff1 | YES
  BISE | the rule, on five widths: 5 of 5
  BISE | first rejected = 2^47 MINUS (access size - 1)

Five bisections, five matches, and the address column is the measurement rather than a restatement: each one converged by asking a forked child whether a given address was an address, twenty times per width, halving a 16 KiB window each step. The formula in the fourth column is the artifact computing 2⁴⁷−(size−1) and the third is what the hardware said, and the harness asserts they are the same number.

And the same rule in the other half

$ ./sysdump | sed -n '/^  BISEL | width/,/^  BISEL | 2\^64/p'
  BISEL | width | first address of the upper half | 2^64 - 2^47 | holds
  BISEL | 1-byte | 0xffff800000000000                 | 0xffff800000000000 | YES
  BISEL | 2-byte | 0xffff800000000000                 | 0xffff800000000000 | YES
  BISEL | 4-byte | 0xffff800000000000                 | 0xffff800000000000 | YES
  BISEL | 8-byte | 0xffff800000000000                 | 0xffff800000000000 | YES
  BISEL | 16-byte | 0xffff800000000000                 | 0xffff800000000000 | YES
  BISEL | 2^64 - 2^47 on 5 of 5 widths

The same address for all five widths, and that asymmetry is the other half of the result. The rule “first rejected = 2⁴⁷ − (size−1)” is about the end of a half, and each half has its own end. The start of a half is one address and does not depend on the access width, because a read that starts there goes upward into legal territory.

TWO RULES, NOT ONE, and the artifact measures both. the START of a half is one address and is width-independent the END of a half is width-dependent, and the first rejected access is the highest address whose byte range still fits A page that quotes only the second and calls it "the canonical boundary" has quoted half a rule and will be caught by the first access whose size is not the one it had in mind.

One asymmetry the artifact reports and does not explain

$ ./sysdump | sed -n '/^  WRAP | 16-byte/,/^  WRAP | 8-byte/p'
  WRAP | 16-byte read at 0xffffffffffffffff | si_code 1
  WRAP | 16-byte read at 0xfffffffffffffff0 | si_code 1
  WRAP | 8-byte read at 0xffffffffffffffff    | si_code 1
  WRAP | So the LOW half's end is range-checked, on five widths, and
  WRAP | the TOP of the space is not observed to be.  This file does
  WRAP | not say why, because the two candidate explanations -- that
  WRAP | the implementation compares the offset within the half and
  WRAP | the high half's offset arithmetic cannot overflow, or that
  WRAP | the top of the space is simply never used and the hardware
  WRAP | never agreed to define it -- need a CPUID bit or a manual
  WRAP | sentence this process has no way to check.  Named as an
  WRAP | OPEN QUESTION, which is a different thing from being wrong.

A 16-byte read whose range runs off the end of the address space and wraps comes back as a real address rather than a rejection. The artifact names this as an open question rather than a finding, and the reason is worth copying: there are two plausible mechanisms, this process has no instrument that distinguishes them, and a page that picks one and states it as a fact is exactly the page that has to be retracted later.

Worked: LA57 and LAM, both reported as zero, and the difference between them

Both are extensions that widen the address space, and the CPU reports both in the same breath:

$ ./sysdump | grep -E 'LA57|LAM '
  CPUID | 0x00000007 subleaf 0 | ECX | LA57 0 RDPID 1 KeyLocker 0
  CPUID | 0x00000007 subleaf 1 | EAX 0x00000000 | LAM 0 MOVRS 0
  CPUID | 0x80000008 | EAX | physical address bits = 48
  CPUID | 0x80000008 | EAX | five-level paging (LA57) = 0
  CPUID | 0x80000008 | EAX | bits 13:12, the linear-address field = 3
  CPUID | 0x80000008 | EAX | of which 0 means 48-bit and 1 means 57-bit;
  CPUID | 0x80000008 | EAX | values 2 and 3 are RESERVED and this machine
  CPUID | 0x80000008 | EAX | reports 3.  A first version of this file read
  CPUID | 0x80000008 | EAX | that field as a COUNT OF BITS and printed 51.

Three readings, and they are three different kinds of statement about the same machine.

LA57 = 0 in 0x80000008:EAX bit 21 and in 7.0:ECX bit 16. Five-level paging is not present. This is a fact about the CPU, read with an unprivileged instruction by any process on earth.
LAM = 0 in 7.1:EAX bit 26. Linear address masking is not present. Also a fact about the CPU. And the subleaf matters: a first version of this file read subleaf 0ECX bit 26 — the same number, a different register, a different subleaf — got zero, and reported LAM as unsupported for a reason that had nothing to do with LAM. It happened to be right and it was still wrong, which is retraction R5.
Bits 13:12 of the same EAX word read 3, and the manual says 0 means 48-bit, 1 means 57-bit, and 2 and 3 are reserved. So the field is a reserved value. And a first version of the artifact read it as a count of bits and printed “51 bits”, which is a plausible-looking number with nothing behind it. The right reading is that the field is reserved and this part puts 3 in it, and the bits that software actually consult — bit 21 and 7.0:ECX bit 16 — both say 48. Three fields in one register, and the one that is easiest to read wrong is the one that is reserved.

AND HERE IS WHY A REPORT IS NOT A READING. LA57 = 0 in CPUID says the SILICON has no fifth level. It does NOT say what CR4.LA57 is set to on this machine, and reading CR4 is a fault. The distinction matters because the two questions have different answers on a machine that supports five-level paging: a kernel may leave CR4.LA57 clear on a part that could do it, and then everything above 2^48 is unavailable for a reason that has nothing to do with the hardware. The number you want is the second one and the number you can read is the first one.

And a fourth reading, the only one about the kernel's intent rather than the metal:

$ ./sysdump | sed -n '/PRCTL | ARCH_GET_XCOMP/,/cannot check it/p'
  PRCTL | ARCH_GET_XCOMP_PERM rc=0 | perms 0x207
  PRCTL | bit0 SMEP=1 bit1 SMAP=1 bit2 SHSTK=1
  PRCTL | that is the KERNEL'S REPORT and not a measurement of the
  PRCTL | CR4 bits, which are privileged and unreadable.  A kernel
  PRCTL | can report anything here and this process cannot check it.

Apply it

Reproduce the five-width matrix.cd courses/x86sys/assets/samples && ./build_samples.sh, then read section 5. (Expect 20 rows by 5 columns, five bisections, and 5 of 5. If a column comes out constant the whole way down, you have written the 16-byte arm in C rather than in movdqu and the compiler has handed you an alignment requirement — see exercise 2.)
Break the 16-byte arm on purpose. Change the movdqu in rd16 to movdqa and re-run. (Expect every unaligned address in the column to report 128, at every offset, which is a table with a pattern and no rule behind it. Then run objdump -d sysdump | sed -n '/<rd16>/,/ret/p' and find the instruction that explains it. A measurement harness that is not read is not evidence, and this is the fourth time in this course that a compiler has quietly changed what was being measured.)
Find the constant in the documentation. Grep a kernel source tree and a set of manuals for 0xffffffff80000000 as a claimed upper-half base. (Expect to find it described as “the kernel half”, which is a statement about where the kernel is mapped and is true, next to statements that treat it as the canonical boundary, which is false. The two get confused because they are the same number and only one of them is a rule.)
Write the canonicality test three ways and compare. The bit-equality test, the masked-ranges test, and the “is it below 2⁴⁷ or at least 2⁶⁴−2⁴⁷” test. (Expect the first to be correct and the second to accept 0x0000800000000000. A bound that is one bit wrong in the middle is a bound that lets a program build a pointer the hardware will refuse, which is a bug that survives testing because the test only ever used valid pointers.)
Decide what your compiler should do about it. If you are writing a backend, the interesting question is not how to test canonicity but when it is worth checking at all. (Expect the answer to be “when a pointer is truncated to 32 bits, and on every address computed from untrusted input”, and expect the second of those to be a real security boundary rather than a portability measure. A 64-bit machine with a hole in the middle of its address space has to treat that hole as unaddressable, and the only way to get that wrong is to test the wrong bits.)

Connect

Backwards, this concept extends rather than repeats: the privilege course's canonical-addressing concept derived the same rule on three widths and this one adds two. The retraction is the credit, and the credit is in the artifact in the form the plan asks for — the neutral course owns the principle and the per-architecture course owns the width table. The memory course's address-space concept is where the 48-bit decision was made and what it cost; this page is what the number is.

Forwards. The next concept asks what happens to a legal address: four index fields, fourteen flag bits, and the two the hardware sets. The connection is the hole in the middle of the address space and the top-level structure that would have to cover it — five-level paging exists because the canonical form grew to 57 bits, and every entry in every table gained a level. And the pagemap measurement in the next concept is the only place in this course where a user process observes a page-table entry directly.

Outward. The same idea on AArch64 is a 48-bit VA with top-byte-ignore, where the “hole” is the top byte rather than the top half, and a tagged-pointer scheme in a low byte rather than a high one. The comparison belongs to the memory course; what this course owns is the x86-64 table, and the reason the constant is wrong so often is that the number people quote is a placement rather than a rule, and only one of those two is architecture.

Keyboard Shortcuts