The manifest’s completion criterion for this course is not “read it”. It is: reproduce the 155 checks, then break the decoder’s length arithmetic on purpose and make the group that catches it fail by name. That sentence is the argument for this concept, so it is worth reading as engineering rather than as course admin.
The harness is crosscheck.py, 155 checks in nine groups, and it does not re-measure anything. It reads the recorded output of the artifact and asks whether the claims are still there. The interesting thing is not that the checks exist but what they are willing to assert, and that is decided by a single number the artifact prints before it makes any claim at all:
$ ./x86dec | sed -n '/estimator =/,/WHAT A BENCHMARK/p'
estimator = min-of-5, 2000000 iterations, two bodies:
a POINTER CHASE, 64 KiB of nodes, L2-resident 6.994 ticks/op
an ARITHMETIC increment loop 0.661 ticks/op
And the two numbers are a factor of ten apart, which is not noise: an arithmetic loop is bounded by the core clock and the TSC is fixed-rate, so a clock ramp reads as noise that is really the clock. A pointer chase spends its time waiting on a load and a clock change does not shorten it. What a benchmark’s noise usually is tells you which part of the machine the benchmark is actually measuring, and the artifact quotes the chase and prints the other one so the gap is on the page.
So: a harness for a measurement with a 230 % spread cannot assert values, and one that tries will fail on a busier machine and teach its reader to ignore it — which is exactly how the harnesses of the four preceding courses broke. The arithmetic of a threshold wide enough to catch a real change is also wide enough to swallow the change.
Rule 1 is the one a reader is most likely to think is pedantic, so here is what it costs. Without it the harness would assert that the ADD chain ratio is 3.65×, and it would fail on a machine where the same experiment gives 3.31× — a machine where the course is just as correct. A check like that does not protect the claim; it protects the author’s day. With rule 1 the check passes on all three recorded runs and would still pass on a machine whose ADD ratio is 2.6.
$ python3 crosscheck.py --- group A: what was measured before anything was measured --- group B: the ALU group, and the result that was not the one expected --- group C: the register-name decisions, read out of the machine --- group D: SETcc, CMOVcc, and a fault instead of a timing --- group E: LEA, and the claim about it that did not survive --- group F: four encodings, three readers --- group G: the three maps, read twice, and one reader's useless agreement --- group H: every retraction is still present --- group I: the limits, and the verdict crosscheck: 155 checks, 0 failed
Seven of the nine groups check the machine. The two that check the checking are the ones worth naming, and they exist because of the experiments in the two previous concepts.
Group G asserts the two-reader cross-check and then runs the control that proves the cross-check is necessary. Fifty-one corpus entries are decoded by the artifact’s own length function and compared against the bytes the assembler emitted, which objdump independently confirms. All 51 agree and none disagree. But before that, the artifact decodes the corpus twice through the same code path, gets 100 % agreement, and then poisons its own table and gets 100 % agreement again:
$ ./x86dec | sed -n '/CONTROL: decode/,/EVIDENCE/p'
CONTROL: decode every corpus entry TWICE through the same code path.
agreement: 51 of 51 -- 100%, and it means nothing at all.
with the first byte of entry 1 changed, it still agrees with
itself: 4 and 4, still 100%. SELF-AGREEMENT IS NOT EVIDENCE.
That is the cheapest demonstration in this collection of why a check that reads its own input twice proves nothing, and it is in the artifact’s own output rather than in a principle stated in prose. Group F is the other one, and it is the only group that asserts a fault: the guard-page result in the third concept, checked as the cmov died and the branch did not, plus the control that the page really was unreadable. A group that asserted “the cmov is correct” could not exist, because correctness of a branchless form is a claim about the compiler and not about the instruction; the group asserts the observation instead and names what it cannot say.
They are worth recording because they are the same disease the four preceding courses documented, and because not one of them was found by reading the harness. All of them were found by running it.
And the two helpers that were rewritten without having been failures — the row extractor and the map parser — are documented in the source with the reason, because the fourth time a collection-wide habit repeats is the time a reader stops believing it was an accident.
$ ./x86dec | sed -n '/map slots/,/neither/p' map slots legacy VEX legacy only VEX only both neither 0F 256 218 101 124 7 94 31 0F38 256 28 125 13 110 15 118 0F3A 256 2 54 1 53 1 201
768 slots, each written into a real file as real bytes and handed to a real decoder, twice: once with the legacy escape and once with a three-byte VEX in front of the same escape. 0F 3A is named in 2 of its 256 slots with the legacy escape and 54 of them with one byte in front. The draft of this course said the newer maps were nearly full, because the designers knew a two-byte map would be consumed and put a second escape in front of it. Measured, they are the emptiest part of the whole encoding in their legacy form, and the room is what VEX and then EVEX moved into.
And the counts are asserted as a partition, not as values: legacy-only + both == the legacy count, VEX-only + both == the VEX count, and all four columns sum to 256 for every map. That is the check that would have caught failure (c) above, and it is exact even though the numbers are not.
x86dec.c, delete the imm_bytes lookup from the VEX branch of decode_len, rebuild, run, re-check. (Expect the vpalignr and vperm2f128 rows to report a length one byte short, the section 4 cross-check to print the disagreements by name rather than as a count, and group F to go red on the “all modelled entries agree” check. Write down which check fired before you predict which would — the gap between the two is usually a check you did not know existed. Then put it back and confirm 155.)0F3A legacy count is 256. (Expect it to fail today, which is the point: the machine changed under it. Now write the version that is true — that 0F3A legacy is below 32 and that it is below its own VEX count — and confirm that it passes on all three recorded runs. You have just written the difference between a check that remembers a number and a check that remembers a property, and the second is the only one that survives contact with a new binutils.)check(g, "add chains at 3.65x", abs(A/B - 3.65) < 0.1) to group B. (Expect it to pass on x86dec.out and to fail immediately on run1.txt and run2.txt, where the same ratio is 3.90 and 3.31. All three files are the same machine and the same experiment and all three are shipped with this course. You have just written a check that fails on the course’s own evidence, which is the clearest possible demonstration that a remembered number is not a fact.)Backwards, this concept is the ledger for the other four. The first concept produced the two syntaxes and the lock diagnostic that the corpus exists to hold. The integer set produced the ALU shape that group B asserts and the bit patterns group C asserts exactly. The flags concept produced the guard-page result, which is the only assertion in the whole harness that is a fault rather than a duration. The prefix encodings produced the four encodings group F cross-checks and the 768 slots group G counts.
Outward, the neighbouring harnesses own the same rules one course along and reading all of them is how you learn them rather than memorise them. The privilege course's harness has four documented failures of its own and the rule that an absent message must be asserted as text. The multiprocessor course's has a retraction group, a noise-floor rule, and the pair of conditions that assert its central result as a shape. The vector course's has a group that could not run at all on two of three runs — a course whose extra evidence is produced by different code is a course whose extra evidence is not comparable with its first evidence — and this course ships three runs through one code path for exactly that reason.
And the last word belongs to the thing this course will not measure. perf_event_paranoid is 4, and there is no way from user mode on this machine to count a branch, a misprediction, an instruction or a cycle. Every number in the measurement sections is a duration, and a duration bounds a count without measuring it. That is not a footnote and it is not a limitation of the author’s effort: it is a property of the machine, it was proved rather than assumed, and it is asserted by group A so that a future reader cannot quietly forget it. The three courses that follow this one in the section — the ABI, the machine, the data path — all inherit that position and none of them improves on it.