Font: Spacing: Letters: Width:

The Harness, and the Six That Refused to Go Quietly

26 minadvancedModule 4: The ArtifactContinues from smp-three

Why this matters

The manifest’s completion criterion for this course is not “read it”. It is: reproduce the 73 checks, then remove the 64-byte stride from the private-line rows and make the placement group fail by name. That sentence is the whole argument for this concept, so it is worth reading it as a piece of engineering rather than as course admin.

The harness is crosscheck.py, 73 checks in eight groups, and it does not re-measure anything. It reads the recorded output of the artifact and asks whether the claims are still there. The interesting thing is not that the checks exist but what they are willing to assert, and that is decided by a single number the artifact prints before it makes any claim at all:

$ ./smpbench | sed -n '/estimator =/,/upper bound/p'
   estimator = min-of-3, 11 of them: spread  82.5%

   The body is a volatile increment, so this floor INCLUDES the core
   clock: an arithmetic loop is bounded by the frequency and the TSC
   is fixed-rate.  Every ratio in section 2 has the same clock on both
   sides, so the clock cancels -- but the floor here is an upper bound

82.5%, on the recorded run. Across the three recorded runs the same estimator measured 19.2%, 55.3% and 82.5% — the machine’s own noise floor is a factor of four apart between one run and the next, on the same body, on the same machine.

So: a harness for a measurement with an 18% spread can assert values with a tolerance. A harness for a measurement with an 82% spread cannot assert anything numeric, and one that tries will fail on a busy machine and teach its reader to ignore it — which is exactly how the previous three courses’ harnesses broke, in the previous three courses. The arithmetic of a threshold wide enough to catch a real change is also wide enough to swallow the change, and a check with a bare number in it is a check whose failure is noise.

There is a second reason this course needs a stricter harness than the memory course did, and the privilege course predicted it. This is a measurement about many cores. A course about one core has a quiet machine to itself; a course about six has eleven other logical CPUs to compete with, and the guest it is running on is doing its best to be busy. More cores is a worse floor, not a better one, and the whole shape of what can honestly be asserted follows from that one fact.

The model: five rules, each one learned by breaking it

1. EXACT FOR STRUCTURE, ORDERINGS FOR TIMING. Asserted exactly: the online CPU list, the sibling pairs, the mutual sibling relation, cores x threads-per-core == the CPU count, the agreement between the cache's sharing list and the topology's sibling list, the L1 set being a strict subset of the L3 set, the ORDER of the seven placement rows, and the presence of all six retractions. Asserted as orderings: which row is dearer than which. Never as values. 2. THE RESULT IS A SHAPE, NOT A NUMBER. The claim the course exists to make -- true sharing and false sharing cost the SAME -- is checked as a PAIR of conditions: 0.6 <= B/C <= 1.7 near 1 B/A > 3.0 and C/A > 3.0 both far above the floor Either alone is nearly contentless. Together they say "these two are alike and neither is cheap", which is the finding, and it survives a machine four times quieter than this one. 3. AN ABSENCE IS AN ABSENCE. No PMU (perf_event_paranoid == 4). No second NUMA node. No count of coherence traffic. Each is checked as a STATEMENT plus its REASON -- because the alternative is a decoder that prints a plausible number in the space where the number is unknown. 4. RETRACTIONS ARE TEXT. Six of them, and the harness asserts each is still a string in the artifact's output. Not "the claim is false" -- "the claim is still written down". A retraction nobody can see is a retraction that gets re-derived eventually, usually by the same person. 5. A CHECK THAT FAILS FOR AN INVISIBLE REASON IS WORSE THAN NO CHECK. If a reader cannot see why it went red, they will learn to ignore it, and an ignored check is worse than an absent one because it is counted. The corollary is the G/A check below: a check that can never fire is not a check either, and both failures look identical from the outside -- a green tick.

Rules 1 and 2 are the ones a reader is most likely to think are pedantic, so here is what they cost and what they buy. Without rule 1 the harness would assert that B/A is 13.05×, and it would fail on a machine where the same experiment gives 11.82× — a machine where the course is just as correct. A check like that does not protect the claim; it protects the author’s day. With rule 2, the check passes on all three recorded runs, whose B/C values were 1.19, 1.03 and 0.97, and it would still pass on a machine whose B/C is 0.6 or 1.7.

And rule 2 has a second job that is easy to miss: the shape catches edits that the value could never catch. A value check notices that the number changed. The shape check notices that the rows stopped being a cluster — which is what happens the moment somebody changes the stride in the private-line declaration, and which is precisely the completion criterion for this course.

The reality: eight groups, and what each one is for

$ cd courses/smp/assets/samples && python3 crosscheck.py
crosscheck: .../smpbench.out

  --- group A: what was measured before anything was measured
  --- group B: the topology, read twice from two kernel files
  --- group C: seven rows, and the one comparison the course is for
  --- group D: the cost the hardware imposes on one thread
  --- group E: a duration, and the correctness it is not
  --- group F: the distinction, measured by its absence
  --- group G: every retraction is still present
  --- group H: the limits, and the verdict

  crosscheck: 73 checks, 0 failed

Eight groups, and five of them are checks about the checking rather than about the machine. Those are the ones worth naming.

Group A checks that the noise floor was measured and PRINTED before anything was claimed — not that it is small, that it exists. If a future edit removed the floor measurement, every ratio check that depends on it would silently compare against a default and the harness would still go green. So the floor is itself a checked fact, and so is the statement of the consequence: durations are not counts, with the word LOWER BOUND in it.

Group B checks the two-reader cross-check, and it checks the verdict. Not merely that both strings appear, but that the artifact printed AGREE or DISAGREE and said which. A harness that only checked for the presence of the comparison would pass on an artifact that compared and silently swallowed the result — which is a comparison for show. It also checks the sibling table three separate ways, because a sibling table can be wrong in a way that is locally plausible: no CPU is its own sibling, no lookup returned -1, and every relation is mutual.

Group C is the course in four checks, and the pair of conditions from rule 2 is two of them:

A, D and E all within a factor of 4 of each other -- the no-sharing rows are a cluster B/A > 3 and C/A > 3 -- both genuinely-shared rows are far above it 0.6 <= B/C <= 1.7 -- THE RESULT: true and false sharing cost the SAME D < C/3 and D < B/3 -- and the table is not flat

The fourth of those is the one people forget, and it is load-bearing. Without it, a harness that passed on every row being identical to the floor would satisfy every “B and C are far above A” condition in the world, by making A enormous. A shape needs a contrast to be a shape, and D — a row that shares nothing and shares an L1 — is the contrast.

Group E checks that the section did not claim what it could not measure. The assertions are all negative: that the output contains That a fence is CORRECT, that it contains is also what you would see if the fence were a no-op, that it contains neither can any timing measurement, and that the fence count is called an upper bound. There is no check in group E that the fence is correct, because there cannot be — and the check that it says so is the strongest statement the group makes.

Group H checks the limits and names the instrument that would settle the missing ones. Seven limits are asserted as strings, and then one more check asserts that the artifact names the PMU as the thing that would have fixed half of them. A limits section that says “we cannot count coherence traffic” without saying why is a disclaimer; the same section saying “perf_event_paranoid is 4, so no user-space event counter exists” is a checked fact.

And one honest observation about this harness, because the failure mode it is guarding against is present inside it. The comment above the SMT check says Measured, D/A came out at 0.97. In the three recorded runs D/A is 1.07, 1.08 and 1.01, and 0.97 is one of the recorded B/C ratios. A transposed number in a comment is the same disease the checks exist to catch, and it is in the harness — which is the reason to print this rather than quietly fixing it: the checks police the artifact’s output and nothing polices the harness’s prose, so the prose has to be policed by a reader.

Worked: the four failures this harness had of its own

They are worth recording because they are the same disease the previous three courses documented, and because every one of them was found by running the harness rather than reading it.

a. A CHECK THAT ASSERTED A DIFFERENCE THE DATA DOES NOT SHOW. The first version of the SMT check said D and A must DIFFER, because the artifact's prose said SMT costs a modest penalty. The measurement said they were the same. The prose was wrong and the check was faithfully defending the prose. This is the worst kind of harness bug: the check was working exactly as designed, and what it was designed to defend turned out to be false. A check written from a claim rather than from a measurement is a check that launders the claim. b. A CHECK THAT WOULD FAIL FOR AN INVISIBLE REASON. Asserting G < A -- that the plain store is cheaper than the locked one -- looks obviously right and is exactly the wrong assertion to freeze. A and G are the same body with and without a `lock`, in a table of SEVEN interleaved arms on a busy guest with a floor this artifact itself measured at 82.5%. The check fails whenever the noise is unlucky, and a reader cannot see the reason from the failure. Replaced by closeness. Rule 5. c. AN EXTRACTOR THAT RETURNED ONLY THE FIRST CAPTURE GROUP. A helper written for numbers was used on a row that has a LABEL and a VALUE, so the label was handed to float() and raised. Same class of error as the hex extractor in the previous course: one helper, two incompatible jobs, and the failure surfaces as a crash rather than as a wrong answer, which is the good case. d. A SUBSTRING COMPARISON THAT UPPERCASED ONE SIDE ONLY. Checking for "IDENTICAL INSTRUCTION" against an UPPERCASED copy of the text, while checking for "Only the memory address changes" against the copy as written. The check then quietly cannot match a phrase whose case differs from the probe, and it will do so in a way that looks like a missing sentence rather than a broken test. The fix is one normalised copy for every substring assertion -- which also fixed the previous course's line-break trap, where the artifact's 78-column wrapping hid four sentences from the harness, three of them retractions. Numeric parses still run on the original, because collapsing whitespace there would join adjacent table columns.

Two helpers were also rewritten for the same underlying reason, without having been failures: the CPU-list expander now treats "0-11" as a range rather than splitting on commas, because splitting on commas called a twelve-CPU L3 the same size as a two-CPU L1; and the placement-row parser now returns both the label and the number.

None of the six would have been found by reading the harness. Five of them are checks that are perfectly well-formed and that fail, or cannot fail, for reasons a reader cannot see from the failure. Only the crash is loud. That is the argument for the exercise below, and the argument for not trusting a green build.

The six retractions, which are what group G exists for

R1 "The plain-store arm measured 1.75 ticks per operation." It MEASURED NOTHING. Written in C as a relaxed atomic store to a variable nothing ever read, the compiler proved the whole loop dead and removed it, and the result was a plausible-looking number for four million operations that did not happen. THE GENERAL FORM, WHICH RECURS EVERY COURSE: WHEN TWO THINGS MEASURE AS SIMILAR, THE FIRST HYPOTHESIS IS THAT YOUR OPTIMISER DELETED THE DIFFERENCE. R2 "Two threads on unspecified CPUs is a measurement." NO. The check is the MECHANISM, not a filter: every worker calls sched_getcpu(), and the repetition is discarded if either thread is anywhere but where it was pinned. Discarded rows are COUNTED and printed. R3 "False sharing is a 61.9x effect." The number is the memory course's and it is correct THERE. Not restated as a property of this machine's protocol, because the ratio depends on the body, the width and the clock -- and re-publishing it would be the remembered-number mistake this harness exists to prevent. R4 "An uncontended lock is expensive because of coherence." RETRACTED TO THE WRONG REASON. R5 "`lock` is one idea that could have been three." A lock-free algorithm written on x86-64 is correct partly BY ACCIDENT. R6 "A placement check is a filter you apply afterwards." NO. Applied afterwards it cannot tell you WHY a thread was in the wrong place, and on a loaded machine it silently converts every row into a discarded row while the run still looks like it worked.

Read R1 and R6 together and there is one idea: a check performed after the fact cannot distinguish “the thing did not happen” from “the thing happened elsewhere”. That is why the placement check happens inside the worker, why discarded rows are counted rather than dropped, and why the plain-store arm is written in assembly rather than in C. All three are the same rule discovered three times.

And read R3 carefully, because it is the one that protects this course from its own headline number. The memory course measured 61.9× and it is right there. It is not restated here as a property of this machine’s coherence protocol, because the ratio depends on the body, the width and the clock, and re-publishing another course’s number as this course’s result is precisely what a harness exists to prevent.

Apply it

Do the manifest’s exercise: remove the 64-byte stride and watch group C fail by name. In smpbench.c, change g_line[MAXCPUS][16] to g_line[MAXCPUS][1], rebuild, run the artifact, rerun the harness. (Expect group C to fail on the shape checks — the no-sharing rows stop being a cluster, and B/A and C/A stop both being far above the floor. Write down which checks failed before you predict which would, because the gap between the two is usually a check you did not know existed. Then put the stride back and confirm 73 again.)
Add a check that cannot fire, and confirm that it cannot fire. Assert that the recorded package column is 0 for every CPU. (Expect it to pass — every value is 0 on this machine, because there is one socket. You have just written a check that can only fail on a machine this course does not claim to describe. Now delete the string you are matching on and watch the whole group go red: the fact that it went red proves it reads the artifact, and the fact that it cannot go red on any other machine proves it is not a check. Both of those facts are worth having written down, and this is the only way to learn which of your checks are which.)
Freeze one number and see what it does. Add check(g, "B/A is 13.05x", abs(B/A - 13.05) < 0.1) to group C. (Expect it to pass on the recorded output and to fail immediately on run1.txt and run2.txt, where B/A is 12.34 and 11.82. Both of those runs are the same machine and the same experiment, and both are recorded as part of this course. You have just written a check that fails on the course's own evidence, which is the clearest possible demonstration that a remembered number is not a fact.)
Run it on a deliberately busy machine. Start eight spinning processes and rerun. (Expect groups A, B, F, G and H to pass unchanged, because they assert structure and text; group C's shape checks should still pass because the shape is robust; and one or two of the tighter bands may fail because the machine got louder. That asymmetry is the design working. If instead a whole group fails together, the harness has a bare number in it somewhere and you have found it.)
Port the rules, not the checks. Write down the five rules for your next project — a compiler, a runtime, a profiler — and mark which one you are most likely to break. (Expect rule 2. Any project whose output includes timings will eventually be tempted to assert a timing, and the moment it does, the check becomes a statement about the day it was written. The defence is the same one here: decide whether the claim is a value, a shape, or an agreement, and only the last two survive contact with a different machine.)

Connect

Backwards, this concept is the ledger for the other six. The topology concept produced the exact structural claims that group B asserts — the online list, the mutual sibling relation, the two-reader cross-check — and the debt to the memory course that group B also checks is present. The sharing concept produced the B/C result, and it is the one claim in the course that is asserted as a shape rather than a value; it is also where the stride the completion criterion removes lives. The atomic concept produced R4 and produced the CAS comparison whose first-draft story the group D checks keep in the record. The ordering concept produced the four things section 4 cannot do, and group E asserts the artifact said them. The NUMA concept produced the node gap that no group checks — rule 3 says an absence must be an absence, and this one is named in the output rather than in the harness. The three-architectures concept is entirely quoted, and group H’s last check is that it says so.

Outward, the two immediately preceding harnesses own the same rules one course along, and reading all three is how you learn them rather than memorise them. The memory course’s harness has a retraction group, a noise-floor rule, and a paragraph on the claim it refuses to make — which is that a factor was measured and a mechanism inferred, and the harness asserts the factor and the preconditions while asserting nothing about coherence states. The execution course’s has the same three and an oracle loop. The privilege course’s has four of its own documented failures and the rule that an absent-sigreturn message must be asserted as text. Three harnesses, four identical failure modes, one shared cause: a check written to defend a claim rather than a measurement.

And the last word belongs to the thing this course will not measure. perf_event_paranoid is 4. There is no way, from user mode on this machine, to count an invalidation, a snoop, a line transfer or a fence. Every number in this course is a duration, and a duration bounds a count without measuring it. That is not a footnote and it is not a limitation of the author’s effort: it is a property of the machine, it was proved rather than assumed, and it is asserted by group A so that a future reader cannot quietly forget it. A course that says what it cannot count, prints the instrument that would have let it count, and has a check for both is the standard every course in this collection holds itself to.

Keyboard Shortcuts