Keyboard Shortcuts

Ctrl+K
Open search
Esc
Close search / dialog
↑
Back to top

A Constant Is a Rotate and a Run Length

27 minadvancedModule 2: The Encoding: What the Bytes AreContinues from a64-encoding
27 min · AArch64: Encoding From The Ground Up

Why this matters

The last concept ended with a row that reads imm16   001fffe0   bits[20:5]   movz w0, #0x0000 and a note that a logical immediate is not a number. This concept is that note, and it is the most interesting encoding in the whole architecture: a twelve-bit field that is not a twelve-bit number, and which reaches more distinct constants than any other twelve-bit field in the instruction set.

The reason a reader needs this is not curiosity. It is that and, orr, eor and bic — the four operations a compiler reaches for constantly when it is masking, clearing and setting bits — have no literal immediate at all on this architecture. They have a pattern immediate. Every mask you have ever written in C is either expressible in one instruction or is assembled out of three or four, and which one is a property of the pattern rather than the value.

The model: one run of ones, repeated, rotated

The shape, in full: the constant is built from ONE run of consecutive ones, repeated to fill the register, and rotated. That is all. So the encodable constants are exactly the periodic bit patterns, and the set is tiny — tiny in the way a filter is small, not in the way an optimisation is small.

$ ./a64dec.py --section=7 | sed -n '/=== 7A/,/reach more values/p'
=== 7A. HOW MUCH OF THE CONSTANT SPACE THE ENCODING REACHES

  width    encodable  the whole space           share
  32       1,302     4,294,967,296             3.031e-07
  64       5,334     18,446,744,073,709,551,616 2.892e-16

  1,302 is the count the manual quotes and this file agrees with
  it, which is worth saying plainly: the number is DERIVED here by
  enumerating the encoding -- six element sizes, at most 64 run
  lengths, at most 64 rotations -- and not read out of the manual.
                

1,302 of 4,294,967,296. That is three parts in ten million. And it is more values than the add/sub immediate reaches, which is 4,095 — so the instruction with the smallest literal field in the architecture reaches more constants than any other twelve-bit field, including the one whose whole job is a number. It reaches them by not being one.

What that looks like in the bits

$ ./a64dec.py --section=7 | sed -n '/=== 7D/,/repeat that element 1 times/p'
=== 7D. WHAT THE SHAPE LOOKS LIKE, DERIVED

  92401000  and      x0, x0, #0x1f
      0x92401000   and      x0, x0, #0x1f
      class field bits[28:25] = 1001  ->  Data Processing -- Immediate
      . N = bit[22] = 1 with sf = 1: the element is 64 bits wide or the top
        half needed filling
      . LOGICAL IMMEDIATE: len = the position of the highest set bit of
        N:NOT(imms) = 0b1111011, so esize = 1 << 6 = 64 bits
      . LOGICAL IMMEDIATE: S = imms & 0b111111 = 4 is the WIDTH of the run
        of ones (S+1 = 5 of them); R = immr & 0b111111 = 0 is how far RIGHT
        that run is rotated inside the element
      . LOGICAL IMMEDIATE: a run of 5 ones is 0b11111; rotate it right by 0
        within 64 bits and you get 0b11111 = 0x1f
      . LOGICAL IMMEDIATE: repeat that element 1 times: the 64-bit pattern
        is 0x000000000000001f
                

Three fields and they are not a number:

That is the entire encoding: an element width, a run width, and a rotation. Everything the twelve bits name, they name because of those three numbers.

Three ways to be sure, and the third is the one that counts

First: this file against itself. Every encodable 64-bit constant, encoded and decoded back through the field-recovery path, which asks the decoder to recover the element size from the imms FIELD rather than being handed the encoder's answer:

$ ./a64dec.py --section=7 | sed -n '/=== 7B/,/proves only/p'
=== 7B. THE ROUND TRIP, THIS ENCODER TO THIS DECODER

  5334 encodable 64-bit constants; 0 round-trip failures
                

The first version of this test handed the decoder the encoder's own imms, which made it a tautology — the encoder chose imms, the decoder was handed imms, and the two agreed about a constant neither had examined. It printed the same line. A tautological round trip reads exactly like a passing one in the output, and that is worth internalising before the next two checks.

Second: the two halves disagree about the field order, which is the whole risk. The decoder takes (N, imms, immr, width) and the encoder returns (N, immr, imms). The first version of the round trip passed the encoder's output straight through, so it handed the rotate amount to the decoder's imms — and 94 of the 5,334 constants came back as something else. The names are now spelled out at the call site so the two orders cannot be confused again, and the count of failures is the evidence that they were.

Third, and this is the one that is not self-referential: a real assembler. All 1,302 encodable 32-bit constants, assembled by clang into a real object file, every word read back by this decoder and compared against the disassembler's own name:

$ ./a64dec.py --section=7 | sed -n '/=== 7C/,/disagreements/p'
  asked the assembler for all 1302 encodable 32-bit constants and
  read every word back with THIS decoder:
    agreed:                    1302
    disagreed:                 0
                

1,302 agreements and 0 disagreements, between two pieces of software written by different people that cannot see each other. That is the only one of the three checks that could have failed for a reason this file does not control, and it is the reason the other two exist.

The reserved space, found by asking

Now the other half, and it is the half a compiler cares about. Everything above measured what the encoding allows; a mask is a case where the encoding has no room, and the only honest way to find the boundary is to walk into it and see whether the assembler lets you back.

$ ./a64dec.py --section=7 | sed -n '/=== 7E/,/all but the top bit/p'
=== 7E. THE RESERVED SPACE, FOUND BY ASKING

  constant      shape                     verdict    what came back
  0x0           32-bit all zeros          REFUSED    expected compatible register
                                                       or logical immediate
  0xffffffff    32-bit all ones, 32-bit   REFUSED    (same diagnostic)
  0xdeadbeef    32-bit no run of two      REFUSED    (same diagnostic)
  0x12345678    32-bit no period at all   REFUSED    (same diagnostic)
  0x80000001    32-bit ONE bit set        ACCEPTED   andw0,w0,#0x80000001
  0x7           3-bit all ones, 3 bits    ACCEPTED   andw0,w0,#0x7

  And the same constants at 64-bit width, where the answer CHANGES
  for some and not others:

  constant              shape                verdict    what came back
  0xffffffffffffffff    all ones, 64-bit     REFUSED    expected compatible register
                                                       or logical immediate
  0x7fffffffffffffff    all but the top bit  ACCEPTED   andx0,x0,#0x7fffffffffffffff
  0xfffffffffffffffe    all but the bot bit  ACCEPTED
  0xf0f0f0f0f0f0f0f0    a 4-bit run          ACCEPTED
                

Read the first and the last rows together, because they are the two facts that matter.

One: the all-zeros and the all-ones are not encodable, at either width, and the diagnostic is the same one for all eight logical operations. The artifact asked and, orr, eor, ands, orn, eon, bic and bics each for #0xffffffff at 32 bits and each refused with the same message, and then each again for the 64-bit all-ones. That is a property of the shared immediate field and not of any one operation — which is why the diagnostic names the field rather than the instruction. And those two constants are exactly the ones a compiler most wants: a mask of “all but the low 32 bits” is not one instruction.

Two: the 64-bit all-ones is refused, and one bit short of it is accepted. This is the retraction, and it reverses the first draft of this page, which said the opposite — that the all-ones is acceptable at 64 bits and reserved at 32. The boundary is not a width. It is the shape: one bit short of all-ones is a legal run of 63 ones in an element of size 64, and all-ones is not a run at all. So a decoder that models the immediate as a scaled number gets this wrong in the specific direction of accepting a constant the assembler refuses, which is the more dangerous of the two — a decoder that invents an instruction produces output that looks right.

The corollary is a compiler's fact, and it is where the next section starts. A mask of all-ones is mov x0, #-1 — 0x92800000, one word — followed by an and. Not because the logical immediate is small, but because there is a different instruction for the one constant it cannot hold.

And the other two immediates, for contrast

AArch64 has three ways to put a constant in an instruction and each has a different reach. This table is the boundaries, found by asking:

$ ./a64dec.py --section=6 | sed -n '/6A. add\/sub/,/6B. move wide/p'
  asked for                     word        verdict   what came back
  add w0, w1, #0                11000020    accepted  add      w0, w1, #0x0
  add w0, w1, #0xfff            113ffc20    accepted  add      w0, w1, #0xfff
  add w0, w1, #0x1000           11400420    accepted  add      w0, w1, #0x1, lsl #12
  add w0, w1, #0x1, lsl #12     11400420    accepted  add      w0, w1, #0x1, lsl #12
  add w0, w1, #0xfff, lsl #12   117ffc20    accepted  add      w0, w1, #0xfff, lsl #12
  add w0, w1, #0x1001           --          REFUSED   expected compatible register,
                                                      symbol or integer in range
  add w0, w1, #0x1000000        --          REFUSED   (same)
  add w0, w1, #0x1234           --          REFUSED   (same)
                

The reach is three isolated values, not a range. The field itself, the field shifted left by twelve, and zero. #0x1000 is accepted and it is not a wider immediate — it is the value 1 with the scale field set, which is why concept 1 measured #0x1000 and #1, lsl #12 as the same 32 bits. #0x1001 is refused, because 0x1001 is not reachable as a twelve-bit field in any of the three states the scale admits. No formula of the form “0 to N times a power of four” describes it, and a reader who has internalised the x86-64 imm8-or-imm32 rule expects a contiguous range with a scale and gets three isolated values.

The same shape appears three more times in the architecture, and all four are worth knowing because the trap is always the same: a scaled field is not a contiguous range.

Worked: what a wide constant costs

Section 6 established that a 12-bit immediate reaches three isolated values and section 7 that the logical immediate is not a number at all. So what does a compiler do with a constant like 0x123456789abcdef0?

It builds it out of MOVZ and MOVK. MOVZ writes one 16-bit field and zeroes the rest; MOVK writes one 16-bit field and leaves the others alone. Four fields of sixteen bits, so four instructions for a 64-bit constant — sixteen bytes, four words, for one C literal — unless a field is zero, in which case it is skipped.

$ ./a64dec.py --section=8 | sed -n '/width  constant/,/why it is/p'
  width  constant             insns  what the assembler chose
  32     0x00001234           1      movz w0, #0x1234
  32     0x12345678           2      movz w0, #0x5678, movk w0, #0x1234, lsl #16
  32     0xffffffff           1      movn w0, #0x0
  64     0x123456789abcdef0   4      movz x0, #0xdef0, movk x0, #0x9abc, lsl #16,
                                    movk x0, #0x5678, lsl #32, movk x0, #0x1234, lsl #48
  64     0x0001000000000001   2      movz x0, #0x1, movk x0, #0x1, lsl #48
  64     0x7fffffffffffffff   4      movz x0, #0xffff, movk ... lsl #16, lsl #32, lsl #48
  64     0xffffffffffffffff   1      movn x0, #0x0
  64     0x123456789abcdef1   4      movz x0, #0xdef1, movk ... (four, again)
                

Now read the table, because it is not ordered by size and the first draft of this page got that wrong in the most instructive way available.

So the rule is not big number. It is pattern, which is the same rule section 7A measured from the other side: the cheapest 64-bit constant is the one with no variety in it at all, and the one twelve-bit field that is not a number is the one that reaches the most values. Both are the same design decision: the encoding pays for reach in bits, and it pays in variety.

And here is the trap inside the move-wide group, which the artifact found and which R7 is about. movn w0, #0x8000 and movn w0, #0xffff are the same mnemonic, and the printer prints different things:

$ ./a64dec.py --section=6 | sed -n '/6B. move wide/,/6C. opc/p'
  movn w0, #0xffff     129fffe0   accepted   movn w0, #0xffff
  movn w0, #0x8000     12900000   accepted   mov      w0, #-0x8001
  movn w0, #0          12800000   accepted   mov      w0, #-0x1
  movz w0, #0          52800000   accepted   mov      w0, #0x0
                

The field holds the MAGNITUDE and the register ends up holding the complement. Which of the two a printer shows is a choice about what it thinks a reader wants, so the encoding is unambiguous and the mnemonic is not: movn w0, #0x8000 prints as mov w0, #-0x8001 while movn w0, #0xffff prints as movn w0, #0xffff, by the same tool, on the same instruction group. And mov w0, #0x0 and mov w0, #-0x1 are the same mnemonic with different immediates and different words — 0x52800000 and 0x12800000. One is “all zeros in a 32-bit register” and the other is “all ones”, and the difference is invisible in the mnemonic.

Apply it

  1. Write the encoder yourself and check it against the assembler. For each element size 2, 4, 8, 16, 32, 64, take each run width 1 to esize-1, take each rotation 0 to esize-1, build the pattern, and ask clang to assemble and w0, w0, #pattern. (Expect 1,302 accepted of the 2,048 candidates at 32 bits, and expect the rejections to cluster on exactly two constants — all-zeros and all-ones — plus everything with no period. Then check your count against the artifact's and expect them to agree, because both are enumerating the same thing; the interesting part is that the two excluded constants are the two you wanted.)
  2. Find the boundary by bisection, not by arithmetic. Do the same for add w0, w1, #imm. (Expect the accepted set to be three disjoint ranges — 0; 0 to 4095; and 0, 4096, 8192 up to 0xfff000 — 8,192 values in all, and expect the largest to be 0xfff000. A reader who computes “12 bits times a scale of 4096” and concludes the range is 0 to 0xfff000 is right by luck: #0x1001 is inside that envelope and is refused.)
  3. Make the field-order bug happen on purpose. In a64dec.py, in section 7B's loop, swap the two arguments to decode_bit_masks so the encoder's immr is handed to the decoder's imms. (Expect 94 of the 5,334 constants to come back wrong rather than all 5,334, and expect the 1,302-constant assembler round trip in 7C to still pass. That asymmetry is the lesson: the self-consistent check is the one that broke, and the check against a third party is the one that did not notice — so neither is sufficient and you need both.)

Connect

Backwards, concept 2 measured imms and immr as bits[15:10] and bits[21:16] and warned that the mask is a lower bound; this page is the reason that warning is not pedantry, because a mask that stopped at bit 14 would still look like a valid imms. Concept 1 measured the scaled add-immediate as the same 32 bits as #1, lsl #12, and this page is why that is a trap rather than a curiosity.

Forwards, concept 4 is about a twelve-bit field that is not a number either — the condition, and what cset does to it — and the two pages have the same shape: an encoding that spends a small field on a restricted set and pays for the restriction in the reader's head rather than in the bits.Concept 5 is where normalisation rule 7 — the sign of an immediate, which three printers disagree about — is found by the cross-check, and it is nine of the eleven failures.

Outward, this is the concept a compiler backend is written against. A backend that assumes a logical immediate is a number will emit masks clang refuses to assemble, and one that assumes MOVN's field is a value rather than a magnitude will compute complements wrong in a way that only shows on inputs where the value has its top bit set. Both are silent, and neither has a test that finds it.

render_lesson_js(&mut page)