Font: Spacing: Letters: Width:

The Performance Monitoring Architecture

24 minadvancedModule 3: Time, and the Limits of What a User Process Can CountContinues from x86-paging

Why this matters, and why this concept is mostly reference

Every number in the previous seven concepts that you would want to compare — instructions retired, cycles elapsed, branches taken, cache misses, mispredictions — comes from the performance monitoring architecture. A processor that cannot count its own events cannot be understood by measurement, only by reading about it. That is the situation on this machine, and the honest thing is to say so at the top of the concept rather than to build five pages of numbers and then footnote that they came from a manual.

So this concept is reference material with the absence proved. The counters and events are quoted and marked. The absence is measured three ways, and the third of those ways is the one that made the course's original sentence wrong.

The model: architectural counters, and the vendor event lists on top of them

The architecture provides four things, and they are enough to build any profiler on any machine:

General-purpose counters — a small number of 64-bit registers (four on Intel, six on this AMD part) that count an event you select by writing a control register, not by a number in an instruction. The event is chosen by writing an event-select MSR, and the MSR number and the event number are vendor-specific.
Fixed counters — two or three counters wired to a fixed event. On Intel they count instructions retired, reference cycles and unhalted core cycles; on AMD the equivalent trio is the same idea with different names. These are the counters a profiler reaches for first precisely because they are architectural.
A way to read them — the RDPMC instruction, whose permissiveness is a single bit: CR4.PCE. Clear, and RDPMC is a #GP at any CPL above zero.
A way to get the count out of the kernel — the perf_event_open system call, which is Linux rather than architecture, and whose permissiveness is a separate number: /proc/sys/kernel/perf_event_paranoid.

AND THE ONE BIT THAT MATTERS MOST, which is not on this page. CR4.PCE, bit 8: "if 1, RDPMC can be executed at any privilege level". That is a design decision with a security argument on both sides and it goes the way this course's measurement shows it going. A counter that a user program can read is a side channel: BRANCH PREDICTOR, and shared-cache occupancy, and the timing of a dependent-load chain all become observable by counting. So RDPMC from ring 3 is a #GP, and the event-counting interface that remains is a system call that asks the kernel. The second question -- how often MAY a process count -- is a policy setting and not an architectural one, which is why the third concept's boundary table has RDPMC in it and this page has three different permission systems stacked on top of one another.

The vendor event lists are where a profiler's portability problems live, and the reason is not that they are long. It is that an event number is a claim about a stepping-level silicon part and the numbering has holes, renumberings and reserved ranges. On a given family the core events are stable; across a family boundary they are not. Any document that presents “the x86 performance events” as a list is presenting one vendor's list, and the more useful table is the one that says which vendor.

The reality: three ways in, all three refused, and the metal is present

This is the section that matters, and it is short:

$ ./sysdump | sed -n '/6B\./,/6C\./p' | head -14
  6B. THERE IS NO COUNTER THIS PROCESS CAN OPEN.  Three ways in,
      all three refused, and none of them is a statement about the
      silicon.

  ABSENT | RDPMC in a forked child | si_code 128 | signal 11 | KILLED
  ABSENT | perf_event_open, a NULL attr | -1 | errno 13 | EACCES
  ABSENT | open /dev/cpu/0/msr | DENIED | errno 13 | EACCES: root only
  ABSENT | and yet CPUID.80000008.EBX bit 6 core-PMU = 1 and
  ABSENT | bit 15 L3-PMU = 1, so the METAL HAS PERFORMANCE
  ABSENT | MONITORS and this process is not allowed to read them.
  ABSENT | A first draft of this course said 'this machine has
  ABSENT | no PMU'.  The right sentence is about ACCESS, and it
  ABSENT | is a better one because it is falsifiable: it says
  ABSENT | which permission is missing, and a page that cannot
  ABSENT | name the missing permission is not making a claim.

Three refusals, three different mechanisms, and they have to be read together with the two CPUID bits that follow.

$ ./sysdump | grep -E 'paranoid|core-PMU|EBX 0x'
  /proc/sys/kernel/perf_event_paranoid = 4
  CPUID | 0x80000008 | EBX 0x191ef657 | core-PMU 1 L3-PMU 1

RDPMC in a forked child: killed by signal 11, si_code 128. A #GP with an empty error code, exactly like every other privileged instruction in the boundary table. 128 is the kernel's way of saying the CPU raised a general-protection fault and there is no address to report — which is why a counter that does not exist and a counter you may not read are the same signal.
perf_event_open: EACCES. The kernel's own interface, refused by policy. perf_event_paranoid = 4 is the setting: at 4 a process may not count hardware events at all without privileges, and this one did not have them. This is a number a user process can read and a knob a user can ask to be changed — which is the difference between this refusal and the one above it.
/dev/cpu/0/msr: EACCES, root only. The third door, and the one that would have given the event-select MSRs. A process that could read an MSR could program a counter directly and bypass every perf policy, which is why the file is root-only and why perf_event_paranoid cannot be used to open it.
And yet core-PMU = 1 and L3-PMU = 1. The silicon has performance monitors. The perfctr_core and perfctr_nb flags were in the flags line of /proc/cpuinfo before this course started, and CPUID says so in the same word that says RDPMC is not permitted.

THE CORRECTION, and it is retraction R7. DRAFT "this machine has no performance monitoring hardware" MEASURED the hardware HAS core and L3 performance monitors, and this process is not permitted to read them, by three separate mechanisms The two sentences have different consequences, and that is the whole reason the second is better. The first says the instrument does not exist, so there is nothing to arrange. The second says the instrument exists and names the permission: raise perf_event_paranoid, or grant CAP_PERFMON, or pass a PMU through from the host. One is a dead end and one is a ticket. And the general form is the one this course has now used twice: an ABSENCE OF ACCESS and an ABSENCE OF HARDWARE look identical from inside a process, and they are diagnosed by asking CPUID -- which is unprivileged -- and not by trying harder.

Worked: what the TSC can and cannot stand in for

With no counters, every timing in this course has come from the time-stamp counter, and section 1 measured its rate twice before making a claim about anything:

$ ./sysdump | sed -n '/TSC rate, busy/,/apart/p'
  TSC rate, busy                    2.2957 GHz
  TSC rate, across a 300 ms sleep   2.2957 GHz
  drift                             -0.0006 %
  pinned to cpu3, and the cpu that answered is 3  (VERIFIED -- the rows below are a controlled experiment)
  POINTER CHASE, 64 KiB of nodes    5.725 ticks/op
  ARITHMETIC increment loop         1.198 ticks/op
  the floor used for every band in this file is the POINTER CHASE,
  because the increment loop is printed too and is the CLOCK rather
  than noise: 4.78x apart.

Three things in that block are load-bearing, and the first is the one nobody checks.

The TSC is invariant, and that was measured rather than assumed. Busy and across a 300 ms sleep, the same rate to within 0.0006 %. A TSC tick is therefore TIME, and every figure derived from it is a ratio and not a cycle count. A processor that slept for 300 ms and came back with a different rate would make every constant in this course a function of the machine's load, and the measurement is what licenses the use.
Two floors, and only one of them is noise. A pointer chase waits on a load, so the core clock is not in the measurement; an arithmetic loop is bounded by the core clock, so a frequency ramp reads as a 60 % “noise” that is really the clock. The chase is the floor and the loop is the clock, and saying so is the difference between a noise band and an instrument.
Placement was verified from inside the process. The CPU that answered is read back with sched_getcpu() after pinning, and the artifact prints whether the two agreed. A benchmark that pins and does not check has measured a distribution over the machine and called it a measurement of the code.

And here is the honest accounting of what the TSC can replace. It can bound a count and it cannot measure one. If a change makes a program 2.00× slower, the TSC says so; whether the extra is instructions, cache misses or branch mispredictions is a question only a counter can answer, and this machine has none that this process can read. A duration is evidence of a cost and never of a mechanism, and the four previous courses in this collection that measured mechanisms — the ISA course's branch-versus-cmov result, the vector course's gather cost, the memory course's TLB reach, the SMP course's true-sharing number — all had to mark the mechanism INFERRED for exactly this reason.

WHAT THE INSTRUMENTS ACTUALLY BUY, on a machine with an invariant TSC and no readable counters. TSC MEASURES elapsed time and therefore BOUNDS any count, because a program cannot retire more events than it has ticks multiplied by some rate you do not know. TSC CANNOT separate two programs that take the same time for different reasons, and therefore cannot tell you WHY anything got slower. So a claim of the form "this instruction is N times cheaper" is a claim a TSC can make, and a claim of the form "this instruction retires fewer events" is not, and a course that has only the first kind of instrument should say so on every page that uses it rather than in a footnote at the end.

Apply it

Read the paranoid value and decide what it licenses.cat /proc/sys/kernel/perf_event_paranoid and then read the kernel's documentation for that value. (Expect 4 to mean no hardware counting without privileges, and expect 2 to mean per-process counting with a kernel restriction. The number is a policy and a policy can be changed, which is the whole point of the retraction: an instrument that is switched off is not an instrument that is absent.)
Check whether the metal has a PMU before you write a lesson about its absence.grep perfctr /proc/cpuinfo and CPUID.80000008:EBX bits 6 and 15. (Expect both to say the counters exist. A course that teaches “this CPU cannot be profiled” from a perf failure has taught a property of a configuration file, and the difference shows up the moment somebody profiles the same program on a laptop.)
Measure the TSC rate the way the artifact does, and then deliberately break the measurement. Time a busy loop with clock_gettime and a sleep, compare the two rates, and then repeat with a nanosleep whose duration you got wrong. (Expect the first pair to agree and the second to print a rate that is a function of your own denominator. A denominator you chose is not a clock, and this is the one mistake in the instrument section that a reader can make themselves in five minutes.)
Compare what a TSC can and cannot tell you about one function you know the answer to. Take a loop with a data-dependent branch, one with the same instruction count and no branch, and time both. (Expect them to differ, because the branch mispredicts and the TSC sees the difference. Then ask what it would take to find out how many mispredicts, and answer: a branch-miss counter, which this process cannot read. The observation is free and the explanation is not, and that asymmetry is the concept.)
Find where in your own codebase a claim rests on an event count rather than a duration.(Expect to find comments that say “fewer instructions” rather than “faster”, and expect that every one of them was written by somebody with better access than a student has. The honest move is to rewrite them as a duration, because a duration is the claim you can check, and to say which mechanism the duration is consistent with and which it rules out.)

Connect

Backwards, and this concept is downstream of the fourth: CR4.PCE is bit 8 of the register whose every other bit is unreachable from ring 3, and this is the one place in the course where a bit in that table decides whether an entire category of measurement is available. The assembly course's flags concept is where RDTSC itself is a two-byte instruction with a serialising property; the CPUID bits here are in the same family of small unprivileged instructions that report the machine's shape.

Forwards, and this is the hinge of the course's last third. The next concept is the whole inventory, and its last row is a counterexample: the absence of AVX-512 is a feature bit and the absence of the counters is a permission, and they are told apart by the fact that one is read with an unprivileged instruction and the other is not. Two absences, two kinds, and the distinction is the difference between a hardware fact and a configuration fact.

Outward. The comparison belongs to the other courses, and the one to make is not x86 against ARM: it is a machine with counters against a machine without them as a course-authoring constraint. Four results in this collection had to be marked INFERRED for want of a counter, and every one of those markings is an argument that the absence of an instrument is a limit on what may be claimed and not only on what may be measured. The multiprocessor course's false-sharing result is the cleanest case: the number was real, the mechanism was inferred from the shape of the table, and the page said so.

Keyboard Shortcuts