Every number in this course is a claim about a pair of cores, and a pair of cores means nothing until you can say which pair. Two threads on unspecified CPUs is not a measurement; it is a rumour with a number attached. The artifact puts it as the first of six rules in its own header, and the reason it learned it was expensive:
So this concept is where the pinning acquires a meaning. To say “two threads on cpu2 and cpu4” is not to name two integers. It is to say: two different physical cores, each with its own L1 and its own L2, both inside one package, both sitting on a shared L3 — and every experiment later in the course either depends on that being true or is invalid because it is not.
And this concept pays a debt. The memory course excluded every row where both threads landed on the same physical core, on the stated grounds that “two SMT siblings share L1 and L2 and are a different measurement”, and used the term without defining it. A sibling is a specific thing with a specific source file, and it takes four rows of /sys to define properly. Until it is defined, the memory course's exclusion has no defensible justification — only a plausible one — and a reader who cannot check a justification cannot apply it to their own machine.
There is also a bug in this section worth reading for its own sake, because it is the shape of a whole family of errors: the first version of this walk reported the machine correctly except for one CPU, and produced a table with a hole in it that looked perfectly well-formed.
The kernel exposes the machine's shape in several directories that different subsystems maintain, and they are maintained independently. What this concept reads:
The walk is: for each cpuN directory, read online, keep it if the file says 1, stop at the first cpuN whose directory is gone. Straightforward, and the first version of it had this shape:
CPU 0 has no online file at all. It cannot be offlined, so the kernel never writes one, and read returns -1 — indistinguishable, to that loop, from a CPU that is not there. The first version skipped it. That dropped one of twelve CPUs, put a hole in the sibling table, and made the artifact report “1 logical CPU per core” on a machine with two.
/* CPU 0 has NO online file: it cannot be offlined, so the kernel does
not write one. The first version of this walk SKIPPED IT for that
reason, which dropped one of twelve CPUs, made the sibling table
have a hole in it, and reported "1 logical CPU per core" on a
machine with two. A missing file is not a missing CPU. */
The fix is one clause: if the read fails, the directory still exists, and this is cpu0, then it is online by definition. The general rule is the same one the rest of this collection keeps arriving at, in a new costume: an absent file is not an absent fact, and the code that cannot tell those apart will report a confidently wrong number. This is why the harness asserts that CPU 0 is in the printed list.
Resolving “who is the other half of this core” means parsing a kernel CPU list, which is not the comma-separated list people assume. It is ranges: "0", "0-1", "0,8", "0-1,8-9".
Attempt 2 is the one to dwell on. It produced a complete, well-formatted table in which every CPU was confidently reported as the sibling of CPU 0. Nothing in the output was wrong-looking, and that is precisely what makes it the dangerous version — attempt 1 at least printed -1, which a reader would investigate.
$ ./smpbench | sed -n '/12 online CPUs/,/the caches/p'
12 online CPUs: 0 1 2 3 4 5 6 7 8 9 10 11
core_id, package, and the other half of each core:
cpu0 core_id=0 package=0 sibling of cpu0 is cpu1
cpu1 core_id=0 package=0 sibling of cpu1 is cpu0
cpu2 core_id=1 package=0 sibling of cpu2 is cpu3
cpu3 core_id=1 package=0 sibling of cpu3 is cpu2
cpu4 core_id=2 package=0 sibling of cpu4 is cpu5
cpu5 core_id=2 package=0 sibling of cpu5 is cpu4
cpu6 core_id=3 package=0 sibling of cpu6 is cpu7
cpu7 core_id=3 package=0 sibling of cpu7 is cpu6
cpu8 core_id=4 package=0 sibling of cpu8 is cpu9
cpu9 core_id=4 package=0 sibling of cpu9 is cpu8
cpu10 core_id=5 package=0 sibling of cpu10 is cpu11
cpu11 core_id=5 package=0 sibling of cpu11 is cpu10
=> 6 distinct physical cores, 2 logical CPUs per core
Six physical cores, two logical CPUs each, one package, and every sibling relation mutual: cpu2's sibling is cpu3 and cpu3's sibling is cpu2. The harness asserts all three of those properties separately, because a sibling table where each row is individually plausible and the relation is not mutual is exactly the shape of attempt 2, and it still prints fine.
The derivation is one multiplication and it has to be exact:
Now the part that turns a report into a check.
$ ./smpbench | sed -n '/the caches/,/THAT IS WHY THE HARNESS/p'
the caches, and which CPUs share each one:
L1 data 32 KiB shared by [0-1]
L1 names [0-1] = 2 CPU(s); the topology puts 2 of
them in one core_id group. AGREE
L1 instruction 32 KiB shared by [0-1]
L1 names [0-1] = 2 CPU(s); the topology puts 2 of
them in one core_id group. AGREE
L2 unified 512 KiB shared by [0-1]
L2 names [0-1] = 2 CPU(s); the topology puts 2 of
them in one core_id group. AGREE
L3 unified 16384 kiB shared by [0-11]
cross-check: the cache sharing lists and the topology's own
thread_siblings_list AGREE (3 of 3 levels agreed)
Two independent kernel files, one fact. That is what makes this a
check rather than an echo -- and it is why the harness can assert it
This is the definition the memory course owed, and it is a definition in the form of a cross-check. The L1 data cache's shared_cpu_list names [0-1]. The topology's thread_siblings_list names 0-1 for the same core. These are two files written by two different kernel subsystems, and they were written without any arrangement with each other — the cache subsystem does not consult the topology when it fills in a sharing list, and vice versa. That is what makes it a check rather than an echo: a program that read the same byte twice would agree with a wrong answer, and one that reads the same fact from two independent writers would have to be wrong in two places at once.
Three of three levels agreed. AGREE is printed per level rather than once at the end, because a single summary line tells you nothing about which level failed, and the next thing this course does is use the L1 and L2 sharing lists to decide what an SMT sibling physically is.
One structural fact falls out of the table and is worth stating because it is the reason row D later is a different experiment rather than a cheaper one. The L1 sharing set is a strict subset of the L3 sharing set — two CPUs at the first level, twelve at the third — and it is contained in it exactly as the sibling pairs are. Two CPUs that share an L1 share an L2 and an L3. Two CPUs that share only an L3 have two separate coherence conversations to run.
Q3 is the one to do, and the order matters: do it before you read attempt 2's history, so that you meet the loud failure first and then learn that the quiet one was worse. A check that has never been seen to fail is a check nobody has reason to believe, and group B's three sibling checks are the reason the topology in this course can be trusted at all.
cat /sys/devices/system/cpu/online; the core_id and physical_package_id of CPU 0; cat /sys/devices/system/cpu/cpu0/cache/index*/shared_cpu_list. Then predict which CPUs are siblings before you read thread_siblings_list. (Predicting first is what makes the exercise worth doing. Almost everyone predicts a contiguous pair or a stride; the answer is usually neither, and on machines with more than one socket the pair crosses the package boundary.)core_id values × logical CPUs per core must equal the count of online CPUs exactly. Then say why a machine that counted one core twice and missed another satisfies the equation anyway — and name the three additional properties that catch it.online file and explain why it is the most important CPU on the machine.(It is CPU 0, it cannot be offlined, and the kernel never writes the file. It is also the CPU the artifact reads the cache sharing lists from, which is why cpu0 appears in the paths rather than the CPU under test.)shared_cpu_list against thread_siblings_list, then hard-code one of the L1 lists to the L3 list and confirm your check goes red. (A cross-check that cannot fail is an echo, and the fastest way to find out which of yours is an echo is to feed it a wrong answer on purpose.)Forward, the next concept spends the whole topology on one table with seven rows, and the reason it can is here: with the pairs named and cross-checked, “only the memory address changed” becomes a statement a reader can verify rather than a claim about a benchmark. Every ratio in that table is a claim about the pair cpu2/cpu4 specifically, and if the pin fails the repetition is discarded and counted.
Backwards, the memory course's instrument concept is the direct ancestor of section 0 of this course's artifact: measure the clock, measure the noise floor, print the floor, and only then make a claim. The rule is not repeated here except to say it was followed — perf_event_paranoid is 4, which means this machine has no PMU, and that single fact decides the shape of all six remaining concepts. The memory course's sharing concept is the page whose “SMT siblings” this concept defines; reading them in that order is the honest way to see the debt being paid, because the definition is only interesting once you know what was excluded with it.
Outward, there are two neighbours worth reading side by side with this one. The execution course's instrument concept is what makes “the same clock on both sides, so the clock cancels” a rule rather than a hope — a ratio has the same core clock in numerator and denominator, which is the only reason any of these numbers is a number at all. And the privilege course's harness is the model for what this course's harness does with the topology: structural claims asserted exactly, because the online CPU list and the sibling pairs are exact facts about this machine and not about a machine in general.
One practical line to carry out of here. If you are writing anything that must be measured rather than guessed — a benchmark, a profiler, a scheduler, a runtime — shared_cpu_list is the file that tells you whether two CPUs you are about to pin to are the same core, and it is a different file from the one that tells you the core id. Reading one and assuming the other is a mistake with a factor of about two in it, and it is the mistake this course exists to make once and only once.