RISC-V Atomics and the Vector Extension

The Reservation, and Why the Retry Is the Instruction

26 minadvancedModule 1: The Reservation, and the Fence That Defines the Model
26 min · Concept 1 of 4 · module: atomic · RISC-V Atomics and the Vector Extension

The trap, stated before the count

On x86-64 the atomicity of an ordinary arithmetic instruction depends on a prefix that is not part of the instruction. Eighteen instructions accept LOCK, the memory form of xchg is implicitly locked with or without it, and a programmer who omits it gets no diagnostic at all — the instruction still assembles, still runs, and is simply not atomic. That is the kind of thing a course about the other architecture has to say out loud before it shows you anything, because you are about to read a page full of counts that are all zero and zero is a number that hides.

RISC-V has no such thing. There is no prefix, there is no flag, and there is no implicit lock on any instruction. The atomics are a third instruction group under their own opcode, and each one is atomic because of what it is.

Now the part that makes the contrast a measurement rather than a claim, and it is the reason this page exists. “Zero against eighteen” is a number a reader should be able to check on both sides, so the artifact counts both sides in the same session, with the same tools:

$ python3 rvat.py --run | grep -A 1 'THE COUNT:'
  THE COUNT: 0 instructions in the RISC-V corpus take a lock prefix.

$ python3 rvat.py --run | grep 'lock-prefixed instructions assembled'
  22 lock-prefixed instructions assembled, and 0 unprefixed
  line in the file.  The DISTINCT mnemonics among them are 22:
                

Zero against twenty-two. And notice what was asked: the RISC-V count is nine probes including the literal word lock, and the answer is the assembler’s unknown token — not a bad operand, not a warning:

  REFUSED     lock add a0, a1, a2              unexpected token
  REFUSED     lock lw a0, 0(a1)                unexpected token

    `lock` is not a mnemonic, a modifier or a reserved word on
    this target.  It is a token the assembler does not know, and
    the diagnostic is `unexpected token`.
                

That detail is load-bearing, and it is the difference between a claim about an architecture and a claim about a tool. The assembler refusing lock is a fact about the assembler. The claim needs to be about the architecture, and the measurement that carries it is the other count: how many instructions are atomic with no prefix at all.

So the zero is not printed alone. Nine operations plus a reservation pair, at two widths and four orderings, are atomic with no prefix whatsoever. A zero printed beside nothing is an absence of features; a zero printed beside that number is a design decision, and it costs the programmer the ordinary-operator shortcut — because compare-exchange, which is two words of x86 assembly with a lock on it, becomes a lr/sc pair inside a loop, and there is no cmpxchg in the base A extension at all. That is the cost, and a course that only advertises is not a reference.

And the structural reason the two lists differ in length is not the length. It is that x86-64 made atomicity a property of an existing instruction and RISC-V made it a category of instruction. A property and a category are different objects, and the difference shows up in what happens when you forget: on x86-64 forgetting is silent, and on RISC-V forgetting is not possible, because there is nothing to forget. The ordinary add is not atomic and never claims to be; the atomic is a different instruction with a different name, visible in a disassembly and checkable in a review.

One opcode, eleven names, twenty-one holes

MEASURED-ON-BYTES. The A extension is one opcode, 0101111, and inside it a five-bit funct5 at inst[31:27]. Eleven of the thirty-two values are named. Then a three-bit funct3 of which only two are AMO widths — 010 is a word, 011 a doubleword. The word comes before the name in the table, because the word is the thing you have to check:

$ python3 rvat.py --run | sed -n '/THE ELEVEN OPERATIONS AT BOTH/,+6p'
  operation  word        assembled  funct5  funct3  aq  rl  bytes
  amoadd     0x00b6252f  amoadd.w   00000   2       0   0   4
  amoadd     0x00b6352f  amoadd.d   00000   3       0   0   4
  amoswap    0x08b6252f  amoswap.w  00001   2       0   0   4
  amoswap    0x08b6352f  amoswap.d  00001   3       0   0   4
  amoxor     0x20b6252f  amoxor.w   00100   2       0   0   4
                

Eleven distinct operations, eleven distinct funct5 values, two widths, four bytes each — and the holes are printed rather than omitted, because a table with a gap and a table with a hole are different tables and a reader who has only ever seen one of them cannot tell which they are looking at:

$ python3 rvat.py --run | grep -A 4 'ELEVEN NAMES AND TWENTY-ONE HOLES'
  11 named of 32, so 21 unnamed, and 2 of the eleven are the
  reservation pair -- which is why the A extension is not "eleven
  atomics" but "nine atomics and a two-instruction construct that no
  single instruction is".
                

Two things fall out of that count, and neither is a coincidence. First, the eleven values are not evenly spaced — they sit in a stride-4 block, with the four bitwise operations at 0, 4, 8 and 12 and the four ordering comparisons at 16, 20, 24 and 28. A reader who expects eleven evenly spaced values writes a decoder with eleven cases where three would do. Second, the reservation pair sits at 2 and 3, inside the first block, next to the arithmetic pair: the two instructions that are not atomic operations are the two that implement a construct, and the encoder put them where a construct belongs.

The width is one bit, and that is the whole of the width story

Here is the next result, and it is a result you can check with a pencil. Take an instruction and its .d twin: same opcode, same operands, same everything except one bit.

$ python3 rvat.py --run | sed -n '/THE WIDTH IS ONE BIT/,+4p'
  the pair               XOR         bits that moved  reading
  amoadd.w / amoadd.d    0x00001000  1                bit 12 of funct3
  amoswap.w / amoswap.d  0x00001000  1                bit 12 of funct3
  ... and twelve rows in all ...

  12 of 12 pairs XOR to exactly 0x1000.
                

Twelve pairs, every one exactly0x1000, one bit, bit 12 of funct3. The .w/.d suffixes look like a type and they are a single bit, and a field with two names is not a type — it is a bit with two spellings. That is worth knowing for a specific reason: a bit with two names is a bit a decoder can read with one table, and it is the same shape as fadd.s against fadd.d in the AArch64 material, which is also one bit.

The same claim arrives from the other direction on this course’s landing page, and the two are the same fact. When the A extension is absent the compiler falls back to __atomic_compare_exchange_4 — and there the width is a template parameter in a symbol name. One bit in the hardware path, a number in a name in the fallback path. A type that is a number in one place and a bit in another is two representations of one decision, and the decision is made by the compiler, not by the programmer.

The consequence for the encoding is worth one sentence, because it is the reason a RISC-V atomic is four bytes and an x86-64 locked instruction is one to four plus something the decoder has to recognise before it can know the length: on RISC-V the ordering bits and the width are ordinary fields of an ordinary word. A suffix is not a type, a prefix is not a word, and both of those facts are visible in a byte count.

aq and rl: two adjacent bits, and the XOR is the proof

The next two bits down, inst[26] and inst[25], are the ordering. The measurement is not a reading of a manual; it is a subtraction. Set aq on an instruction that did not have it, and the difference is a single bit. Do it at both widths, on all three kinds of instruction, and print every result:

$ python3 rvat.py --run | grep 'THE SET OF'
  THE SET OF aq XORs OVER THE WHOLE CORPUS: 0x04000000
  THE SET OF rl XORs OVER THE WHOLE CORPUS: 0x02000000
  THE SET OF aqrl XORs OVER THE WHOLE CORPUS: 0x06000000
                

One number for aq across every operation, every width, all three instruction kinds. One for rl. One for the pair. aq is bit 26, rl is bit 25, they are adjacent, and that is the whole encoding. No prefix, no second instruction, two of thirty-two bits.

And the three are additive, which is the cheapest possible proof that they are two fields rather than one field with three values — because a decoder can recover a two-bit field by OR-ing two single-bit masks and cannot recover a single field with four values that way. Additivity is not a confirmation of the positions; it is the reason they are fields.

What the two bits buy is quoted, and the quotation is the point rather than the padding. Chapter 12 gives the four C11 semantics: acquire on aq alone, release on rl alone, sequentially consistent on both. So four semantics, two bits. And the other two architectures in this collection reach the same four by different mechanisms: AArch64 spends two opcodes (ldar and stlr are not ldr and str), and x86-64 spends nothing at all. RISC-V spends two bits where AArch64 spends two opcodes and x86-64 spends nothing, and the reason is that it did not want a fourth instruction group.

And the same four semantics are reachable with a fence instead, which is the thing that makes the model different in kind:

A RISC-V atomic is a fence with the unnecessary part removed, and the two bits are where the removal is recorded. Chapter 12 says the FENCE R, RW instruction suffices for acquire and FENCE RW, W for release “as compared to AMOs with the corresponding aq or rl bit set” — that is, the bits save you the ordering you did not ask for. A reader who only ever reads the fence form will not notice the bits exist; a reader who only ever reads the bits will not know what relation between two sets of operations the fence form is expressing. Both sides have to be measured, which is why this course has a page for the fence at all.

lr and sc: the retry is the instruction

MEASURED-ON-BYTES for the encodings, QUOTED for the reservation rules, and the division is the boundary of the whole concept rather than a convenience.

x86-64 has a compare-and-swap. cmpxchg with a lock prefix is one instruction that either swaps or does not, and a loop around it is a performance habit. RISC-V has no compare-and-swap in the base A extension at all. What it has is two instructions that must be used in pairs and that report failure in a register — and a pair that reports failure to a caller who must branch is not a pair, it is a loop. The retry is not syntax around the instruction; it is what the instruction is.

$ python3 rvat.py --run | sed -n '/THE LOOP, MEASURED/,+9p'
  word        mnemonic   operands                     bytes
  0x0005869b  sext.w     a3, a1                       4
  0x160525af  lr.w.aqrl  a1, (a0)                     4
  0xfed59ce3  bne        a1, a3, 0x40 <amo_cas_loop>  4
  0x1ac5272f  sc.w.rl    a4, a2, (a0)                 4
  0xfe071ae3  bnez       a4, 0x44 <amo_cas_loop+0x4>  4
  0x00068513  mv         a0, a3                       4
  0x00008067  ret                                     4

  7 instructions: 1 lr, 1 sc, 2 branches, and the rest is the
  comparison of the loaded value against the expected one.
                

Seven instructions, one lr, one sc, and the branch after the sc is the retry. The compiler did not choose a compare-and-swap, because there is not one to choose. The __sync spelling costs the compiler one extra instruction per attempt for a reason that has nothing to do with the hardware — it takes no expected-out pointer, so the compiler cannot learn the memory word’s value from a failed attempt and has to reload it. Same hardware, same instruction pair, two instructions per attempt against one, and the difference is entirely in the calling convention of the source language.

And the asymmetry in the suffixes is not a typo. The lr carries both bits and the sc carries only rl, and chapter 12 says why: software should not set rl on an LR unless aq is also set, nor aq on an SC unless rl is. So a two-bit field admits four combinations and only three are meaningful on each instruction — a field whose validity depends on a neighbour field, which a decoder that reads it without reading which instruction it is on will name as four things that are two.

The constraints, and the ones that are quoted

lr requires rs2 = 0 and the assembler enforces it. That is a measurement, and the refusal is the cheapest possible proof that a constraint is real:

  lr.w a0, a1, (a2)          REFUSED  lr with a THIRD operand
                                  expected '(' or optional integer offset
  sc.w a0, (a1)              REFUSED  sc with only TWO operands
  amoadd.w a0, (a1)          REFUSED  an AMO with only TWO operands
  lr.w.t a0, (a1)            REFUSED  a MASKED lr, which does not exist
  amoadd.b a0, a1, (a2)      REFUSED  a BYTE AMO, which is Zabha
                

And the diagnostic is about spelling while the cause is about semantics. The assembler is not saying you wrote the syntax wrong; it is saying the instruction has no rs2 to give. That is a general lesson about assemblers and it costs nothing to learn here. The Zabha row is a different kind of good error: the diagnostic names the letter that would make it work, which is a better message than “unknown instruction” and a reminder that the ISA is a set of documents.

What may sit between the halves, and what may not

That is quoted, and the rule is what the calling convention has to protect: a hart can hold only one reservation at a time, so a second lr invalidates the first, and a function call in the middle of a pair kills it. The mitigation is in the manual too — a store-conditional to a scratch word, to forcibly invalidate a reservation during a context switch. And the eventuality guarantee is what stops the loop from being an unbounded one: a constrained sequence of LR/SC pairs, with nothing else between the halves, must eventually make progress.

And none of that has been observed. No reservation is ever held on this host. There is no hart here to hold one, no other hart to invalidate it, and no scheduler to preempt the thread between the halves. What is measured is that clang emits an lr.w.aqrl and an sc.w.rl with a branch between them — a fact about a compiler, not a fact about an atomic. That line is in the artifact at the end of the section, in the header, and as limit 3, and it is the sentence this whole course is built to keep honest.

Try it

  1. XOR the widths yourself.(In amo_objdump.txt, take the amoadd.w line and the amoadd.d line and subtract. You should get 0x1000 and nothing else. Now do the same for amoswap and for lr, and ask what a decoder that hard-coded the XOR instead of reading funct3 would do about an instruction pair this course does not contain.)
  2. Find the constraint the assembler knows and you do not.(Run llvm-mc-21 --target=riscv64 -assemble on lr.w a0, a1, (a2) and read the diagnostic slowly. It is a syntax message. Now write down what would have to be true of the hardware for that message to be about syntax, and notice that nothing would. Then try amoadd.b and notice the opposite: a diagnostic that tells you the extension you are missing.)
  3. Write the loop by hand and see what the compiler will not do.(Put a __atomic_compare_exchange_n in a file, compile it at -march=rv64ima and at -march=rv64im, and diff the two disassemblies. One is four instructions and a branch. The other is a call into a library. Ask what the compiler would have to be able to do to emit a compare-and-swap here — and then check whether RISC-V has such an instruction.)

Connect

What a race is and what “atomic” has to mean for one is in the neutral course on SMP, and the ordering half is smp-ordering — neither of which is re-taught here. The same subject on the two machines this collection can actually run is x86-atomics, where the prefix is the mechanism and forgetting it is silent, and a64-atomic, where acquire and release are access modes rather than two bits. The section’s spine — why there are no condition codes at all — is rv-noflags, and it is the reason the sc result had to go into a register: on an architecture with no flags, the only place a result can land is a register somebody has to check.

Next: fence, and an Ordering Model Defined in Terms of It — because the two bits you have just measured are, in the manual's own words, a fence with the unnecessary part removed, and the next page measures the other side of that sentence.