Six concepts, one artifact, and a habit. The artifact is a decoder: give it bytes and it tells you what instructions they are, and it prints the derivation for every field it consumed. It does not shell out to objdump and it does not use a compiler. It parses the ELF section table to find code, then decodes each instruction from the rules in this course.
$ cd courses/isa/assets/samples $ python3 x86dec.py corpus | tail -12 .plt.got 0x2000 8 B 2 instructions .fini 0x22a0 13 B 4 instructions --- 1885 bytes of code, 472 instructions decoded, mean 3.99 bytes
And the default view, which is the one that matters:
$ python3 x86dec.py 4889e84883ec205bc3 48 89 e8 movq rax,rbp 48 83 ec 20 subq rsp,0x20 5b pop rbx c3 ret
That is a function prologue, epilogue and return, decoded from nine bytes with no toolchain involved. It is the collection’s mission stated as a program — a format, parsed by hand, into a working answer — and it is what the object course’s emit concept did one level down, where the artifact wrote a valid object file rather than reading one.
So why does it need a course? Because a disassembler that prints only answers is a tool you have to trust, and this collection’s standing rule is that you never have to trust anything you can read. So:
$ python3 x86dec.py --why 8b 44 24 20
offset 0, 4 bytes: 8b 44 24 20
mov eax,[rsp+32]
. byte 8b at 0 is a one-byte opcode
. byte 44 at 1 is the ModRM byte: mod=1 reg=0 rm=4
. mod=01: an 8-bit SIGNED displacement follows
. rm=100 means a SIB byte follows: 24 at 2
. ss=0 -> scale 1, index=4, base=4
. index=100 with REX.X=0 means NO INDEX
. 1 byte displacement at 3, value 32 (0x20)
. read little-endian, so the LOW byte is at the LOW
. address -- 11 22 33 44 means 0x44332211
Every line names the byte it came from and the decision it forced. That is the difference between a mnemonic and an argument.
Three design decisions, and each one is a position rather than a default.
And there is a fourth decision that is a limitation stated rather than a shortfall, and it is the one that makes the tool honest:
Which is worth stating as the general principle, because it applies to every artifact in this collection: an artifact that says “I do not know” in a specific place is more useful than one that is confidently wrong everywhere. The object course’s verifier, the image course’s stack walker and the hardening course’s posture reader all have the same property, and all three were built after their authors got something wrong and found that a specific refusal was worth more than a plausible guess.
The claim that matters is not that the decoder works on examples the author chose. It is that it agrees with an independent implementation across real code:
$ ./build_samples.sh 2>&1 | sed -n '/^I7/,/^I8/p' ok corpus .init 27 B 8 instructions chain exact ok corpus .plt 96 B 18 instructions chain exact ok corpus .plt.got 8 B 2 instructions chain exact ok corpus .text 1741 B 440 instructions chain exact ok corpus .fini 13 B 4 instructions chain exact ok mini .init 27 B 8 instructions chain exact ok mini .plt 16 B 3 instructions chain exact ok mini .plt.got 8 B 2 instructions chain exact ok mini .text 288 B 72 instructions chain exact ok mini .fini 13 B 4 instructions chain exact ok probe.o .text 56 B 17 instructions chain exact --- 578/578 instruction boundaries agree (100.00%)
578 of 578, and every chain closes exactly across three specimens including an object file, where every displacement is a zero waiting for a linker. And the crosscheck is not a summary of that run — it re-derives all 118 claims from a fresh build:
$ python3 crosscheck.py 2>&1 | tail -16 A 6 passed 0 failed I1 5 passed 0 failed I2 10 passed 0 failed I3 4 passed 0 failed I4 10 passed 0 failed I5 6 passed 0 failed I6 19 passed 0 failed I7 33 passed 0 failed I8 23 passed 0 failed I9 2 passed 0 failed ------------------------------------------------------------------------------ 118/118 checks passed ALL CLAIMS HOLD
Six of those are in group I9 and they are the two most valuable checks in the file, because I9 is the one that found the bugs:
Every one of those four was invisible in isolation. Each produced a plausible-looking disassembly of a hand-picked example, because the example was the case the author had in mind and the table was right about that case. A table has no internal contradiction to notice, so exhausting it against an independent source is the only way to find out.
And here is the part worth reading twice, because it is the collection’s method applied to its own tooling. Three separate parsers in this course were wrong about the oracle’s output format, and every one of the failures looked like the decoder being wrong.
$ ./build_samples.sh 2>&1 | sed -n '/A SECOND MODE/,/confused/p' 0x62 is the AVX-512 EVEX prefix in 64-bit mode, so it collides with the 32-bit BOUND instruction exactly the way 0x40 collides with INC. and a disassembler WITHOUT AVX-512 support refuses it and loses the instruction stream from that point on -- which is why the corpus below is built -march=x86-64: a reader that gives up is not a reader that disagrees, and the two must not be confused.
Three of them, and each taught the same lesson:
That is why the crosscheck compares boundaries and lengths and never operand text. Not because operand text is uninteresting — it is printed, for a human — but because asking a checker a question one of your tools is known to get wrong is how a correct tool gets “fixed” into a broken one.
The real use is not your own binaries. It is everything else, and it is the same use the other artifacts in this collection have:
$ python3 x86dec.py /usr/bin/objdump | tail -3 $ python3 x86dec.py --section .text /bin/ls | head -20 $ python3 x86dec.py --why 0f 1f 84 00 00 00 00 00 $ python3 crosscheck.py -v 2>& | head -40
Three uses in increasing order of value. Reading unfamiliar code — the --why output teaches the encoding while it answers the question, which is a better way to learn an ISA than a reference. Comparing two builds — decode both and diff the instruction streams, and a compiler-version difference shows up as a shift in the mean length. Failing a build — and this is the one that pays, because the chain property makes it a cheap assertion:
That is the honest division of labour, and it is the same one the posture reader makes. A decoder’s most valuable output on an unknown binary is a refusal, not a summary, because a refusal localises and a summary does not. The decoder was built on x86-64 and would decline most of a modern AVX-512 binary — and declining, with a reason, is more useful than a mean-length statistic computed over the 40% it understood.
One thing the tool deliberately does not do, which is worth naming because every decoder eventually gets asked to. It does not produce a coverage percentage, and it does not claim completeness. The 364-entry table is audited; the naming subset is not claimed to be anything in particular. A tool that printed “97% coverage” would be making a claim about the fraction of instructions in a real binary, which depends on which binary, and the number would be true and useless — it would go up and down with the code rather than with the tool.
$ cd courses/isa/assets/samples $ ./build_samples.sh $ python3 crosscheck.py $ python3 x86dec.py --why 8b 44 24 20 $ python3 x86dec.py corpus
Then extend it, which is the exercise that turns a reader into a decoder:
1. Make the decoder handle the 0F 38 and 0F 3A
maps, and watch it decode a binary built
with AVX or AVX-512:
clang -O2 -mavx2 -o avx corpus.c
python3 x86dec.py avx | tail -3
objdump -d avx | tail -3
(The length arithmetic already handles the
two maps -- they are 3-byte opcodes, and 0F 3A
takes an imm8 after the operand. What is missing
is the NAMES and, more importantly, the VEX
and EVEX PREFIXES, which are 2 and 4 bytes of
their own and sit where a legacy prefix would.
Adding them is a good exercise precisely because
it shows how many places a prefix can appear.)
2. Write a length-only decoder from scratch --
no operand text, no names, no tables beyond
which opcodes take a ModRM. Then run it over
every binary on the system and report the
first offset it cannot size.
./length.py /usr/bin/* 2>&1 | head
(Expect it to be short, correct on the
overwhelming majority, and to fail on AVX-512
binaries. That failure is the right answer: a
length decoder that never refuses is a length
decoder that is guessing, and the refusals are
the only interesting output.)
3. Now break the decoder and let the crosscheck
catch it. Change one table entry, rebuild
nothing, and watch I9 fail:
sed -i "s/'f4': ('pmuludq'/'f4': ('hlt'/" x86dec.py
python3 crosscheck.py 2>&1 | tail -4
Restore it. Then do the same to a LENGTH RULE
rather than a table entry -- add 1 to the
disp32 case -- and watch I7 fail instead,
with a section name and a first-divergence
offset. (The two checks catch different classes
of error: I9 catches a wrong TABLE, I7 catches
a wrong RULE. A checker suite needs both, and
the only way to know which of yours covers
which is to break something on purpose.)
4. Compare against a third reader, and be careful
about what a third reader proves. Install or
locate one -- gdb's disassembler, or a Python
library -- and run all three over one section.
(Then ask the question that matters: do the
three INDEPENDENTLY, or do two of them share an
implementation? Two tools that both consult the
same opcode table are one tool with two
interfaces, and their agreement is worth
nothing. Check the provenance before you count
the agreement -- that lesson arrived in this
course as a real bug, in a reference parser
that silently dropped the oracle's
no-mnemonic entries.)
5. Finally, the exercise that is really the
course: use the decoder to answer a question
about a binary you did not build.
python3 x86dec.py --section .text /bin/ls \
| tail -20
Then find the mean instruction length, the
fraction under 4 bytes, and the most common
opcode. Now go and read /bin/ls's source. Is
the code you found the code you expected from
the source you read?
(It will not match, and the mismatch is the
finding. A compiler rearranges, inlines,
unrolls and vectorises; the source is a
hypothesis about the bytes and the bytes are
the evidence. This is the same relationship the
object
course's oracle loop established between a
reader and its input, applied to a reader and a
program -- and it is the most transferable
habit in the collection: when a claim about
code does not match the code, believe the
code.)
Exercise 3 is the one that makes the harness a habit rather than a script you run. Break the decoder on purpose and confirm the right check fails — a table error caught by I9, a length-rule error caught by I7 with a section name and an offset. A checker that has never failed is not known to work, and the two checks catch different classes, so knowing which covers which is the difference between a suite and a decoration. This is the fifth time in this collection that a harness bug masqueraded as a finding, and the fifth time the fix belonged in the harness.
Exercise 5 is the one that carries the method out of the course, and the mismatch it produces is the point. Read /bin/ls’s source, predict the instruction stream, and then look. You will find a mean length around 4, you will find call and ret and a handful of arithmetic opcodes dominating, and you will find that the source contains loops the binary has unrolled and functions the binary has inlined. The code is the evidence and the source is a hypothesis, and this collection has argued in one direction only.
This concept is the payoff for the chain, and the connection runs the full length of it. The program-header concept taught the segment table. The section-header concept taught the section table and the string-table index. The dynamic-section concept taught DT_* tags and the rule that the array is 16 bytes per entry. Every one of those is a source this artifact reads, and it reads them with struct.unpack_from rather than shelling out. That is the mission stated as a program: a format, parsed by hand, into a working answer, with no dependency on the tool that already understands it.
The connection to the emit concept is the tightest in the collection. That artifact wrote a valid ELF object by hand — section headers, a symbol table, a string table, relocations — and this one reads a valid ELF executable by hand. One course apart, the two ends of the same format, and between them the whole chain: bytes, relocations, linking, loading, hardening, and now the instruction encoding the bytes actually spell. The chain began with a hex dump and has arrived at a decoder, and the arc is the point.
Two connections about what a tool cannot see. The encoding-limits course measured that position independence costs twice the fixups and could not say why; the reason is one ModRM byte, and this decoder can now show you which one in a real binary — find a PC32 relocation, decode the instruction it points into, and read the disp32 at the offset the formula gives you. And the CET concept measured that endbr64 is present in binaries that request no enforcement; this decoder can count them and read their encoding, and it cannot tell you whether the kernel honours them — which is exactly the limit the hardening course set, now visible from the other side.
And the connection forward, which is the next course in the list. Everything this course decoded is a description, and a description has no timing. Now that the bytes are legible, the natural next question is what the hardware does with them: how many bytes it fetches at a time, what a branch predictor is predicting, why an instruction boundary is free and a taken branch is not. That is How a CPU Executes Instructions, and it can assume everything in these seven concepts — which is the only kind of prerequisite worth having, because it means the next course never has to re-derive an encoding.