Everything in the first four modules was reading. This one is writing, and that is not a change of exercise — it is a change of what you know.
A reader can hold a specification open. A writer cannot. To read an object file you need to know which fields exist; to write one you need to know which fields are load-bearing, which are conventions, and which are optional — and the only way to find out which is which is to emit a file and see whether a real linker accepts it.
So here is the mission's “from scratch, not by delegation” rule made concrete. emit_elf.py in this course's assets/samples/ writes a relocatable ELF64 x86-64 object file with nothing but Python's struct module. No library formats a single byte. Every field is computed and packed by hand, and the acceptance test is not “it parses” but “a production linker links it, the resulting program runs, and it prints the right answers.”
$ python3 emit_elf.py hand.o
wrote hand.o (936 bytes)
ELF header 0x0000 .. 0x0040
.text 0x0040 .. 0x0057 (23 bytes)
.data 0x0058 .. 0x0064 (12 bytes)
.rela.text 0x0068 .. 0x0098 (48 bytes)
.rela.data 0x0098 .. 0x00b0 (24 bytes)
.symtab 0x00b0 .. 0x0140 (144 bytes)
.strtab 0x0140 .. 0x0168 (40 bytes)
.shstrtab 0x0168 .. 0x01a5 (61 bytes)
section headers 0x01a8 .. 0x03a8 (8 x 64)
e_phnum = 0 (no program headers: this is a relocatable file)
relocation addends are in the RECORDS, not in the bytes
936 bytes, 8 section headers, 6 symbols, 3 relocations. That is the whole minimum. Not a demonstration — a working object file that GNU ld merges with compiler-produced objects and whose program computes the right answer.
Before the bytes, the plan, because the plan is the transferable part and the bytes are just the plan in a different notation. The object being built is the equivalent of this C:
Function bodies shown brace-free, to fit the page. compute is helper(x) + 1; start is compute(x).
And the requirements that shape the emitter. Notice that each one is a requirement the format imposes, not a choice:
And the two requirements that are conventions rather than rules, which is worth separating because it is the hardest thing to learn from a specification: nothing requires the .strtab to begin with a NUL byte, yet every producer emits one and st_name == 0 depends on it; and nothing requires sections to be 8-byte aligned, yet every producer does it. The emitter in this course does both, with a comment saying why, because a file that is merely legal and a file that behaves like every other file are different targets and only the second one is worth shipping.
start is ten bytes and compute is thirteen, and the relocation offsets are 5 and 15:
.text, 23 bytes, from the file at 0x40
00000040: 8b44 2404 e800 0000 00c3 8b44 2404 e800 .D$........D$...
00000050: 0000 0083 c001 c3 .......
offset 0x00 start:
8b 44 24 04 movl 4(%rsp), %eax
e8 call rel32 - opcode at 0x04
00 00 00 00 - the FIELD, at 0x05, zero
c3 ret
offset 0x0a compute:
8b 44 24 04 movl 4(%rsp), %eax
e8 call rel32 - opcode at 0x0e
00 00 00 00 - the FIELD, at 0x0f, zero
83 c0 01 addl $1, %eax
c3 ret
Note that the two calls are at different instruction offsets but the same offset within the instruction — byte 1 of a five-byte e8 rel32. That is the whole reason the field is at 5 and 15 and not at 4 and 14, and it is the concrete form of the claim A Hole and a Record made about three different instruction lengths sharing one 4-byte field. The encodings concept takes that further with AArch64, where the field is not even contiguous.
And the addend. Both records carry -4:
.rela.text, 48 bytes, two 24-byte records, from 0x68
00000068: 0500 0000 0000 0000 0200 0000 0500 0000 ................
00000078: fcff ffff ffff ffff 0f00 0000 0000 0000 ................
00000088: 0400 0000 0100 0000 fcff ffff ffff ffff ................
record 0 record 1
r_offset = 0x05 r_offset = 0x0f
r_info = 0x0000000500000002 r_info = 0x0000000100000004
type 2 = PC32 type 4 = PLT32
sym 5 = compute sym 1 = helper
r_addend = -4 r_addend = -4
And r_addend is written with struct.pack('<QQq', ...) — a signed 64-bit format. Using Q would produce 0xfffffffffffffffc with no error at all, and the linker would compute a displacement of about 1.8×1019 instead of -4. That is the addend concept's signed-field warning, and in an emitter it is not a reading exercise — it is a one-character bug.
Every one of these produced a file that readelf and llvm-objdump read without a single warning, and that linked and ran and printed wrong answers. They are the reason this concept exists.
BUG 1 e_shoff hardcoded to 64
The header said the section table was at 0x40, where
.text also was. readelf printed garbage section
headers and errored out. FIX: pass the computed
offset. LESSON: layout is data, not a constant.
BUG 2 sh_link and sh_info pointing at the wrong indices
.rela.text had sh_link=8 and sh_info=2; the symtab
is index 5 and .text is index 1. The file PARSED.
readelf warned that the link value was out of range,
and every symbol name came out empty. FIX: name
the indices as constants and use them everywhere.
LESSON: a relocation section's sh_link is the
SYMTAB and its sh_info is the section being patched.
Swapping them resolves every relocation against the
wrong thing and looks fine.
BUG 3 two objects in .data at the same offset
`int answer` and `int *answer_ptr` both had
st_value = 0, in an 8-byte .data. They OVERLAPPED.
The file was completely legal. It linked, ran, and
printed 389845008 -- an ADDRESS -- for `answer`,
because the 8-byte pointer relocation wrote into the
same four bytes the int lived in. FIX: answer at 0,
answer_ptr at 4, .data grown to 12 bytes. LESSON:
nothing checks symbol overlap. Ever.
BUG 4 addl $1 BEFORE the call instead of after
compute(x) was emitted as (x+1) then call helper,
so it computed helper(x+1), not helper(x)+1.
The disassembly was plausible; the source comment
said something else. FIX: put the add after the
call. LESSON: when the bytes and the comment
disagree, the bytes are the program.
Bug 3 is the one worth dwelling on, because it is the only one that no tool reported. Overlapping symbols are not an error in ELF. The section has a size, the symbols have offsets and sizes, and nothing in the file says two symbols may not claim the same bytes. The result was a program that linked, ran, and printed a plausible large number. The only way it was caught was that the expected answer was 41 and the printed value was not 41. That is the honest state of affairs in object-file work, and it is the argument for the next concept.
The acceptance test, and then the proof that the holes were filled. This is the whole concept in one terminal session.
1. does readelf accept it, with no warnings?
$ readelf -SW hand.o
[ 1] .text PROGBITS 000000 000040 000017 00 AX 0 0 16
[ 2] .data PROGBITS 000000 000058 00000c 00 WA 0 0 8
[ 3] .rela.text RELA 000000 000068 000030 18 5 1 8
[ 4] .rela.data RELA 000000 000098 000018 18 5 2 8
[ 5] .symtab SYMTAB 000000 0000b0 000090 18 6 1 8
[ 6] .strtab STRTAB 000000 000140 000028 00 0 0 1
[ 7] .shstrtab STRTAB 000000 000168 00003d 00 0 0 1
$ readelf -sW hand.o
1: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND helper
2: 0000000000000000 4 OBJECT GLOBAL DEFAULT 2 answer
3: 0000000000000004 8 OBJECT GLOBAL DEFAULT 2 answer_ptr
4: 0000000000000000 10 FUNC GLOBAL DEFAULT 1 start
5: 000000000000000a 13 FUNC GLOBAL DEFAULT 1 compute
$ readelf -rW hand.o
0000000000000005 0000000500000002 R_X86_64_PC32 0xa compute - 4
000000000000000f 0000000100000004 R_X86_64_PLT32 0x0 helper - 4
0000000000000004 0000000200000001 R_X86_64_64 0x0 answer + 0
Two independent readers agree, with no diagnostics. Now the real test — a production linker, compiler-produced companions, and execution:
$ gcc hand.o helper.o driver.o -o app $ ./app start(5) = 11 answer = 41 *answer_ptr = 41 (answer_ptr == &answer: yes)
And here is the payoff: the linker filled the holes, and you can see both the before and the after. From the hand-built object's own .text:
BEFORE, in hand.o AFTER, in the linked app
.text+0x04 e8 00 00 00 00 start at 0x1140:
1144: e8 01 00 00 00
.text+0x0e e8 00 00 00 00 1144 + 5 + 1 = 0x114a = compute
compute at 0x114a:
.data+0x04 00 00 00 00 00 00 00 00 114e: e8 0d 00 00 00
114e + 5 + 0x0d = 0x1160 = helper
.data+0x04 all eight bytes zero .data in the executable, vaddr 0x4000:
4000: 00 00 00 00 00 00 00 00
4008: 08 40 00 00 00 00 00 00
4010: 29 00 00 00 - 0x29 = 41
4014: 10 40 00 00 00 00 00 00
^^^^^^^^ 0x4010 = &answer
Every one of those numbers can be checked by hand, and every one of them is a consequence of a decision the emitter made. The 01 in e8 01 00 00 00 is compute - (start + 5). The 0d is helper - (compute + 5). The 10 40 is the address of answer. Three fields, three arithmetic operations, and the emitter contributed nothing to any of the final values — it contributed only the offsets and the addends that made the arithmetic possible.
Which is the last thing worth saying about writing object files, and it is the reason the mission puts this course before the linker course: the emitter's entire job is to describe where the holes are and what goes in them. Every value in the final binary is computed by someone else. A backend that gets the offsets and addends right and the instructions wrong produces a file that links; a backend that gets the offsets wrong produces one that does not. The failure modes are not symmetric, and knowing which is which is most of what this concept is for.
$ cd courses/obj/assets/samples
$ python3 emit_elf.py /tmp/hand.o
$ readelf -a /tmp/hand.o | head -60
$ ld -r /tmp/hand.o -o /dev/null && echo "a real linker accepts it"readelf -a, llvm-readobj-21 -a and llvm-objdump-21 -d -r on the same file. Any field they disagree on is a place where one of them has an opinion, and working out which is the interesting part. Note the one deliberate difference: readelf prints a section symbol's name as the target section's name, while the symbol table stores st_name = 0 for it. The file has no string; the tool is substituting one.readelf's warnings and note that it still produced output — and then check whether llvm-readobj-21 warns at all, because a reader that does not validate is the more dangerous one. For bug 3, the symptom is a plausible wrong number and no diagnostic from anyone. Reproducing a bug you have read about is worth far more than reading about it again..bss section..bss is SHT_NOBITS with sh_type = 8 and, crucially, sh_offset pointing past the end of the file with no bytes in it — a detail worth getting wrong, because a reader that tries to read sh_size bytes from sh_offset will read whatever follows. Add the third function's PLT32 against helper and confirm the linker resolves all three calls correctly.e_machine and the four bytes of .text, which become the ADRP/LDR pair and the RET from the encodings concept. Make the same mistake twice — overlapping symbols — and observe that on this machine nothing will catch it, because there is no AArch64 linker here to fail. That is worth feeling directly: the verification you cannot do is the verification you will eventually miss.sh_info is out of range; an r_info with a symbol index past the end of the symtab; a shstrndx that does not point at a string table. Which of the four does GNU ld accept silently? The answer is the most useful single fact in this concept, and it is different for each.Without looking back: your emitter produces a file that readelf parses without a warning, ld -r accepts, and that links into a program which runs and prints one wrong number. Everything else about the program is correct. The .data section is 12 bytes, answer is declared st_value = 0, st_size = 4, and answer_ptr is st_value = 8, st_size = 8, both in section 2. What is wrong, why did no tool report it, and what general rule about object files does it illustrate?
You are writing a backend for a new architecture. It must emit an object file your linker will accept. You have the specification, a reference producer, and a week. You write the header, the section table, the symbol table and the relocations, and you discover that the first file you emit is rejected with “invalid section header” while the second is accepted but produces a program that crashes on the first call. Given what this concept measured, what is the order in which you should build confidence, and what would you have to be able to check in order to tell a wrong offset from a wrong instruction?
The general habit: an object file is a set of claims, and almost none of the claims are checked by anything. Build the file so that each class of claim can be falsified by a different oracle, in the order that lets the cheaper oracles speak first.
This concept is the mission's destination, and everything in the first four modules is scaffolding for it. The stated goal of the course is to emit an object file a real linker accepts, without using LLVM or any other backend to do it, and emit_elf.py is that goal met in 936 bytes. It is worth being precise about how small a claim that is: two functions, two data objects, three relocations, one undefined symbol. It is not a compiler. But every mechanism a production backend needs is in those 936 bytes — a section table, a symbol table with a weak and an undefined entry, a string table, a relocation set with an internal and an external reference, and a bias that only the record knows about.
The connection to The Fixup Record is the most direct in the course, because the emitter writes the records by hand and gets the packing wrong in instructive ways if it does not. r_info = (sym << 32) | type is not a fact to memorise; it is a fact to type correctly, four times, with the halves the right way round. And the signed r_addend is a one-character difference between 'q' and 'Q' in a format string, with no diagnostic. The next concept is about the class of bug where the tool is silent, and this is the smallest worked example of it in the course.
Bug 3 — the overlapping symbols — connects straight back to COMMON and the ABI Break. COMMON exists precisely because C allows a symbol's storage to be claimed without being placed, and ELF responds with SHN_COMMON, a section index that means “no section, size and alignment instead.” That is a format explicitly acknowledging that symbol placement is a separate problem from section placement. The hand-built object has no COMMON symbols, and that is a choice: it places everything itself, and so has to be right. A backend that emitted tentative definitions as COMMON would hand the placement problem to the linker and would be wrong in a way this file cannot be. The bug is the mirror image of the feature, and seeing both makes the design decision legible.
The bug-4 lesson — that the bytes and the comment can disagree — is a general point about working from specifications, and it is why the COFF course's harness discipline and the JVM course's crosscheck are not bureaucracy. A specification tells you what the bytes mean. It does not tell you what you meant. The comment in emit_elf.py that reads compute(x): ... call helper ; addl $1,%eax is a claim about the code, and when the bytes said otherwise the bytes were right and the comment was a lie. The only defence found in practice is to write the expectation down independently — in a comment, in a test, in a disassembler's output — and then compare.
And there is a connection forward to the linker course, which is where the whole chain leads. The emitter's job ended when it wrote the offsets and addends; every value in the final binary was computed by GNU ld. The next step in the chain is to be the thing that computes them — and the four sections of this object, taken in the order the linker processes them, are a specification for a minimal linker: walk the section table, merge the symbol tables, resolve names, then walk the relocations applying S + A - P with the overflow check that the relocation table concept'sTYPE flag exists to make possible.
Next: the encodings that decide how big a fixup can be, which is the last piece of information you need before you can write the pass that applies them.