The instruction in the last concept makes two promises. Atomicity — the read-modify-write happens as if no other thread were looking at that location — is what the last three concepts have been about. Ordering — nothing moves across the instruction that would let another core observe the stores on either side in the wrong sequence — is the second promise, and on x86-64 it comes attached to the first whether or not anyone wanted it.
This concept separates them, and its real subject is not the fence. Its subject is the difference between the two questions you can ask about a fence, and the fact that this course can only ask one of them. You can ask how long it takes. You cannot ask whether it is correct — and that is not a limitation of this artifact or of this machine. It is a property of what a timing measurement is, and the argument for it takes about four lines and is worth more than the number.
There is a reason to care beyond tidiness. Almost everything in concurrent programming that goes wrong is an ordering bug: a lock-free algorithm that reads a half-updated structure, a message that arrives before its payload, a flag that is set before the data it describes. Those bugs cannot be found by measuring how fast anything runs, and a course that measures a fence's duration and moves on is quietly implying that the two are connected. They are not. This one is about the gap, and the gap is named in the artifact rather than left in the margins.
The lesson generalises past concurrency. A duration is not a guarantee applies to every guarantee that is expressed as “never happens” rather than as a property of an output: that a compiler will not delete a volatile access, that a page will not be freed twice, that an overflow will not wrap. Timing tells you about cost. It tells you nothing about correctness, and where the two are conflated you get a green test suite.
The loop is the smallest message-passing pattern there is, and it is deliberately the weakest one:
$ ./smpbench | sed -n '/iterations, store data/,/The two loops COMPUTE/p'
2000000 iterations, store data / store flag / load flag:
no fence 1.3 ns/iter
mfence 34.3 ns/iter 25.86x
for (long i = 0; i < N; i++) {
g_data = i;
{ with fence: } __asm__ volatile("mfence" ::: "memory");
g_flag = 1;
g_sink = g_flag;
if (g_sink == 12345 || g_data == -1) g_flag = 0;
}
One thread stores a value, then stores a flag saying the value is there, then loads the flag back. What the fence orders is the two stores against each other — it says a reader that sees the flag set will also see the data. Without a fence on a machine that reorders stores, the reader can see flag = 1 and then read a stale g_data, which is a torn message: the code says the payload is ready and the payload is not.
Three details decide whether this loop measures what it claims, and all three are things that have gone wrong in earlier courses:
The third point is the one that makes this a controlled experiment rather than two unrelated loops. A common version of this measurement puts the fence in one arm and nothing in the other, and the compiler's freedom to move the volatile stores is then a difference between the arms on top of the fence. Here both arms carry a memory clobber, so the compiler is held still on both sides and the fence is the only variable left.
And note the timing method: this section uses clock_gettime(CLOCK_MONOTONIC), not the TSC, and divides by the iteration count to get a nanosecond figure. The result is an absolute time rather than a ratio, which is a deliberate change from the rest of the artifact — and it is allowed here because the number being claimed is a comparison between two numbers in the same loop, not a comparison with anything on another machine.
$ ./smpbench | sed -n '/The two loops COMPUTE/,/That is why the fence/p'
The two loops COMPUTE THE SAME ANSWER. The fence cannot change
what this program computes on x86-64, and it is not cheap. That is
not an argument against the fence: it is an argument for knowing WHY
you have one. x86-64's memory model is already strong enough that
an ordinary store then an ordinary load cannot be reordered past
each other. A weaker architecture -- and a compiler that does not
know which architecture it is generating for -- needs the fence to
make that true, and the SAME SOURCE needs it on one and not on the
other. That is why the fence is in the source and the cost is not.
34.3 ns against 1.3 ns: the fence costs more than twenty-five times the loop it sits in, and it changes nothing about what the loop computes. On this machine the memory model is already strong enough that an ordinary store followed by an ordinary load cannot be reordered past each other, so the fence is pure overhead here.
Read that sentence twice, because the 34.3 and the twenty-five times are different numbers and only one of them is the finding. The nanoseconds are stable; the multiple is not. Across the three recorded runs mfence came out at 34.3, 33.6 and 34.9 ns/iter, while the unfenced loop moved between 1.1 and 1.3 ns/iter — so the factor moved across a range of about 26 to 31 while the thing being factored did not. That is the same pattern as the atomic section, and it has the same lesson: quote the stable number, and treat any ratio whose denominator is a plain loop on a busy guest as a description of the day you ran it.
So the fence is not free, and the fence is not wrong. What is wrong is a course — or a compiler, or a reviewer — that reaches either conclusion from the number alone. The correct reading of this table is: on x86-64 the explicit fence is redundant, and it is expensive; therefore a redundant fence in the source is a performance bug and a missing fence on a weaker target is a correctness bug, and both statements are about a different machine.
That is why the fence is in the source and the cost is not.
Put that the other way round and it is a rule for portable code: write the fence, and expect the compiler to remove it where the target does not need it. An explicit atomic_thread_fence(seq_cst) in the source is documentation and insurance; the cost is paid only on the targets where the insurance is required. A hand-rolled asm volatile("mfence") in portable source is not that, because it is a x86 instruction spelled into the middle of a program that may be compiled for something else — and on AArch64 and RISC-V there is no mfence at all, because there the fence is expressed by the access's own mode or by chosen edges.
This is the part of the artifact that is worth more than the part above it, so it is printed as a list rather than a paragraph:
Each of those four is a different kind of not-knowing and they are worth separating, because only the first is fundamental.
The first is a logical impossibility, not a missing instrument. Consider the two hypotheses: (a) the fence works, and the loop runs in the same time with and without it; (b) the fence was compiled to nothing at all, and the loop runs in the same time with and without it. Both predict the observation. A measurement cannot distinguish two hypotheses that make identical predictions, and buying a performance counter would not help, because the counter measures the same observable. You could make the fence observable — by removing it on a weakly-ordered target and watching the program produce wrong answers — but then you are measuring a different program on a different machine, which is precisely what this course says it is not doing. Establishing correctness of an ordering requires either a formal model or a bug, not a stopwatch.
The second is a language problem, not a hardware one. A fence written as a compiler intrinsic carries the target's semantics. A fence written as an x86 instruction is an x86 instruction, and a compiler targeting AArch64 has no way to know the author meant “order my stores” rather than “emmit mfence, which does not exist here”. The volatile and the clobber in this artifact's loop make the question moot because the experiment is single-target by construction; in real portable code the intrinsic is the entire answer.
The third is an upper bound and the artifact labels it as one. 34.3 ns/iter is a claim about what the source asked for. It is not a claim about what the CPU executed, because a compiler is permitted to delete a redundant fence and on x86 it usually does. This is a small thing in isolation and a large thing in a course: a measurement of your own source is a measurement of a proposal, not of the machine.
The fourth is the interesting refusal. “The order in which two cores observed X” is a property of a program that would require pinning an observation onto a specific event in each core's history, with no atomicity on the act of reading the other core's history. The privilege course made the same refusal from the other side — a ring-3 process cannot see the saved instruction pointer — and the two refusals are the same shape: the quantity is real, it is what the hardware actually did, and it is not reachable from user mode.
A course that cannot measure the second most important thing in
concurrent programming should say so here rather than quoting the
architecture's ordering table as though it were a result. The
ordering table is in the ISA course's future and in the privileged
specification; it is a CONTRACT, and this is an observation.
asm volatile("" ::: "memory") from the unfenced arm only, so the compiler is free to reorder the volatile stores there but not in the other arm. (Expect the unfenced arm to get faster, not slower — you have just measured the compiler, not the CPU. If the two arms come out the same, check the assembly before you conclude anything: the volatile stores are both required to exist and the reordering freedom on this target is small, so the honest expectation is “little or no change” and the value of the exercise is seeing which.)objdump -d a binary that uses std::atomic with seq_cst, on x86-64 and, if you can, on AArch64. (Expect the x86 seq_cst store to be a plain xchg and the relaxed one to be a plain mov — the same instruction, because the compiler has nothing to choose between. Expect the AArch64 relaxed add to be ldadd-family and the seq_cst one to be surrounded by explicit acquire/release or barrier instructions. This is the concrete form of R5, and seeing it in a disassembly is worth more than reading about it twice.)Backwards, the atomic concept established that the lock prefix costs 2.54× a plain store with nothing to lock against, and attributed it to the barrier the prefix implies. This concept is the reason that attribution is right, and the reason it cannot be verified from a timing table. Read them together: the previous one measured a price for a guarantee, and this one says the guarantee itself is unmeasurable here. A course that quoted 2.54× without this section would be leaving a reader to assume the number was a measurement of correctness. It is not, and it is not.
Forward, the NUMA concept takes the largest thing this course cannot do — a second memory domain — and asks whether the distinction is still worth teaching when the phenomenon is absent. The answer it gives is yes, and the reason is the same one that applies here: a distinction you cannot measure the second half of is still a distinction, and naming it is not the same as claiming it. Then the three-architectures concept says where the fences actually differ, and the harness concept explains why a check can assert that this section said it cannot do these four things, and cannot assert that it can.
Outward, two links, one inward-looking and one outward. Inward: the privilege course's harness contains the same rule in its most literal form — it asserts the presence of the sentence that says the vDSO has no sigreturn trampoline on this kernel, and asserts the reason guessing would have been wrong. An absence is asserted as an absence, with a reason, and never as a decoder that prints a plausible number. Outward: this section is the one a compiler author will be asked about in a design review, and the defensible answer is the four-item list above rather than a citation of a benchmark. “I can tell you what the fence costs; I cannot tell you it is correct, and neither can anyone with a stopwatch” is a complete and correct answer, and it is much rarer than it should be.