The manifest’s completion criterion for this course is not “read it”. It is: reproduce the 131 checks, then restore the static initialisers on the control arm’s three pointers and make group D fail by name. That sentence is the whole argument for this concept, so it is worth reading as engineering rather than as course admin.
The harness is crosscheck.py, 131 checks in nine groups, and it does not re-measure anything. It reads the recorded output and asks whether the claims are still there. The interesting thing is not that the checks exist but what they are willing to assert, and that is decided by one number the artifact prints before it makes any claim at all:
$ ./simdbench | sed -n '/estimator =/,/^ The body/p' estimator = min-of-3, 11 of them: spread 25.1% (11.81 ns/iter) The body is the section 2 loop. This number is the reason the crosscheck asserts SHAPES and not values: a tolerance that wide can verify that 4 lanes beat 1 lane and cannot verify by how much.
25.1% on the recorded run, and 33.5% and 62.5% on the two others — the same body, the same machine, the same binary, minutes apart. So a harness for a measurement with a 25% spread cannot assert values with a tolerance, let alone one measured at 62%, and one that tries will fail on a busy machine and teach its reader to ignore it, which is exactly how the previous three courses’ harnesses broke. The floor is also the reason the artifact’s own prose calls the absolute figures unusable and quotes only ratios — the ratio moved under 10% across those same three runs while the absolute nanoseconds moved by nearly two.
There is a second reason this course needed a stricter harness than any of the four before it, and it is the sharpest methodological finding in the whole artifact. This is a measurement about a compiler. Every other course in this collection measured hardware. A compiler’s behaviour is a fact about a specific program, a specific version and a specific set of flags — and a timing is not a way of establishing it. So this harness has a third rule the previous four did not need, and it is the reason one of its nine groups reads bytes rather than numbers.
Rule 1 is the one a reader is most likely to think is pedantic, so here is what it costs and what it buys. Without rule 1 the harness would assert that the L1 speedup is 3.89× — and it would fail on run1.txt, where it is 3.91×, and on run2.txt, where it is 3.70×. Three of this course’s own recorded runs, same machine, same experiment. A check like that does not protect the claim; it protects the author’s afternoon. With rule 2 all three pass, and the check would still pass on a machine whose ratio were 2.0 or 6.0.
And rule 2 has a second job that is easy to miss: the shape catches edits the value could never catch. A value check notices the number changed. The shape check notices that the pointer column came back to 3.9× — which is exactly what happens the moment somebody restores the static initialisers, and which is the completion criterion for this course.
$ cd courses/simd/assets/samples && python3 crosscheck.py --- group A: what the machine says it can do, before anything is claimed --- group B: the register, as a table of integers --- group C: six arms, one loop, and the two shapes it asserts --- group D: the same 4-wide arm at five working-set sizes --- group E: a claim about a compiler, checked against the bytes --- group F: elementwise, independent, and the price of a reduction --- group G: alignment, the fault, the gather and the tail --- group H: every retraction is still present --- group I: the limits, and the verdict crosscheck: 131 checks, 0 failed
Five of the nine groups are checks about the checking rather than about the machine, and those are the ones worth naming.
Group A checks that the machine’s own capabilities were read before anything was claimed — not that AVX-512 exists, but that it does not, and that all five leaves were read. That is the assertion that licenses the whole of module 3 to be quoted, and it is checked rather than assumed because a course that quotes a 512-bit register while quietly measuring one would be making a claim about hardware it has not got.
Group B is the one group in this collection that can tell you which cell you broke. It asserts each register’s lane counts against a hard-coded table, and it asserts the doubling separately — so a table that got one cell wrong while keeping the doubling would still be caught, and the failure message names the register and prints the five numbers. That is possible because the lane table is arithmetic rather than a measurement, which is the whole reason it is worth having a group for.
Group C is the course in six checks, and two of them are shapes rather than values:
That seventh one is the check that would have caught R7, and it is worth a moment. The first version of this course expected the memory-operand arm to be faster — one fewer load has to be better — and wrote a check defending that. The measurement said slower, the explanation turned out to be about dependency length rather than operation count, and the check was rewritten to assert the thing the data shows. A check written from a claim rather than from a measurement is a check that launders the claim.
Group D is the course in four checks and it has the most interesting member. The three that assert the collapse are described above. The fourth is the control: the same source reached through double * globals must be far worse than through named arrays, by more than 4× at every size. That check is the one the manifest’s exercise turns red, and it is red for a reason a reader can see in the output.
Group E is the group the previous four courses did not need, and it is four checks over a 2×2. The mnemonics are printed, not just a verdict; the compiler version and the flags are recorded beside them; and the aliasing mechanism is asserted as a named string, because a number going the right way is not evidence about bytes. If a future edit restored the pointer form and the artifact stopped being vectorised, this group is the one that notices.
Groups H and I are about the artifact’s honesty. H asserts all eight retractions are still present as text — not that the claims are false, but that they are still written down, because a retraction nobody can see is a retraction that gets re-derived eventually, usually by the same person. I asserts seven limits and then one more check that the artifact names the disassembly as the thing that would settle the missing compiler claim. A limits section that says “we cannot tell whether the compiler vectorised” without saying why is a disclaimer; the same section saying “here are the bytes” is a checked fact.
All eight are printed by the artifact in section 8 and asserted as text. Five of the eight were found by running the artifact rather than reading it, and that ratio is the argument for the exercise below.
Read R7 and R8 together and there is one idea: a measurement of a control, and an inference from a timing, are both ways of not looking at the thing. That is why the disassembly is appended to the recorded output and why the harness reads it, and it is the third clause this course added to the two rules the previous four courses had already learned.
And read R1 and R4 together, because they are the same lesson from opposite directions. A wrong answer that is approximately wrong announces itself; one that is exactly wrong does not. The FMA bug produced a checksum of 3.3 where 32.0 was expected, and the vhaddpd bug produced exactly 2× the right answer — which looks like a perfectly good sum of a different set of numbers, and is stable across every re-run. Neither produced a suspicious timing. You cannot detect this class of error by measuring more carefully; you can only detect it by looking at the value. That is why every table in this artifact ends with a checksum and why rule 5 of the header exists.
simdbench.c, change static double *gA, *gB, *gC; to static double *gA = bA, *gB = bB, *gC = bC; and delete the three assignments at the top of sec_ceiling. Rebuild, run, rerun the harness. (Expect the pointer column to come back to about 3.9× and group D to fail on the control checks. Write down which checks failed before you predict which would — the gap between the two is usually a check you did not know existed. Then put the assignments back and confirm 131 again.)ns per 4 elem of the scalar reduction arm is greater than 100. (Expect it to pass, trivially. You have just written a check that can only fail on a machine this course does not claim to describe. Now delete the string you are matching on and watch the group go red: that it went red proves it reads the artifact, and that it cannot go red on any other machine proves it is not a check. Both facts are worth having written down, and this is the only way to learn which of your checks are which.)check(g, "the L1 speedup", abs(g4[0] - 3.89) < 0.01) to group D. (Expect it to pass on simdbench.out and to fail immediately on run1.txt and run2.txt, where it is 3.91 and 3.70. Both are the same machine and the same experiment and both are recorded as part of this course. You have just written a check that fails on the course’s own evidence, which is the clearest possible demonstration that a remembered number is not a fact.)arm_memop’s vfmadd132pd to vfmadd231pd — the exact bug that R1 records. (Expect the table to look completely normal and group C to fail on the checksum checks, which is the whole point of rule 5. Notice that no timing check fails: the ratio the broken arm produces is plausible. Now go and find the two other places in your own tools where a wrong-but-plausible value can pass.)Backwards, this concept is the ledger for the other six. The width concept produced the lane table that group B asserts as twenty-five exact integers, and the three-architectures deferral that group B also checks. The compiler concept produced R7 and R8, and group E is the disassembly check that exists because of them. The shapes concept produced the add-versus-multiply null and the 4.00× independence result, and the “these four bodies leave four different numbers” check. The reduction concept produced R4 and R5. The boundaries concept produced R6 and the three shapes in group G. The three-architectures concept is entirely quoted, and group I’s checks are that the artifact says so and nothing more.
Outward, the four immediately preceding harnesses own the same rules one course along, and reading them together is how you learn them rather than memorise them. The privilege course’s harness has the same structure plus four documented failures of its own, and its rule that an absent message must be asserted as text is the same rule as R1 here. The multiprocessor harness is the immediately preceding instance and its first documented failure is this course’s first: a check that asserted a difference the data did not show. The memory course’s has the retraction group and the noise-floor rule, and it is where the “measure the floor before you claim anything” discipline was established. The execution course’s has the same three and an oracle loop. Four harnesses, one shared cause: a check written to defend a claim rather than a measurement. And this course added a fifth rule to them, because it is the first in the collection to make a claim about a compiler, and the failure mode is new even though the discipline is old.
And the last word belongs to the two things this course will not measure. perf_event_paranoid is 4, so no instruction, uop, cycle, load, store or cache-line split can be counted here — every number in sections 2 to 6 is a duration, and a duration bounds a count without measuring it. And five CPUID leaves are zero, so no 512-bit register and no mask register was measured at all; every claim about one is a manual claim with a document. Those are not limitations of effort. They are properties of the machine, they were proved rather than assumed, and both are asserted by group A so that a future reader cannot quietly forget them.