Back to course

The Middle of Every Build

18 minintermediateModule 1: Why Object Files Exist
18 min · Module 1: Why Object Files Exist · Foundations

Why This Matters

You have written a great deal of code and almost certainly never looked at what your compiler actually handed to your linker. That artefact is the subject of this course, and it is stranger than the executable you eventually run.

An executable has addresses. A file called a.out knows that main lives at 0x401126, and that number was chosen by a linker that had to satisfy page alignment, a preferred base address, and a section-ordering request. An object file knows none of that. It cannot know, because it is one translation unit out of possibly ten thousand, and the other nine thousand nine hundred and ninety-nine do not exist yet.

So what does it know? It knows where the holes are. Every reference to a symbol it did not define is left as a piece of incorrect bytes, and beside it the file records what belongs there. That is the whole idea, and this course is a detailed study of how three different formats express it — because a linker author has to implement all three, and the differences are not cosmetic.

This is also the first course on the path from a compiler to a shipped binary. The goal for this whole collection is a learner who can go from parsing to a working executable without depending on LLVM or any other backend. That requires understanding this file, because whatever you generate has to be something a real linker will accept.

A Simple Model

The build, and the two files it passes between:

main.c --cc--> main.o --ld--> a.out ^ | the ONLY thing the linker is given about main.c

Two properties define a relocatable object, and they are both consequences of the same fact — it is incomplete:

Everything else in the format is in service of those two. The section table exists so the linker can group bytes. The symbol table exists so the linker can learn the names. And the relocation records exist because of the second property: they are the file's list of its own holes.

A useful reframe. A relocation record is not a patch to be applied. It is a work order: at this offset, this many bytes are wrong, and here is what would make them right. The difference matters the moment you write a linker, because a work order needs a symbol table to be resolvable against, and a patch does not.

The Actual Detail

Everything in this course is measured from one 30-line C file, compiled three ways on this machine. The whole specimen set, reproducible from the source that ships beside it:

  demo.c  --  30 lines, chosen to exercise every mechanism
                    a linker must see: a call to an undefined symbol,
                    a reference to an external variable, a global
                    definition, a static definition, initialised and
                    uninitialised data, a string literal with and without
                    a relocation, and an inline function.

  $ clang -target x86_64-pc-linux-gnu  -O1 -c -o demo_elf.o      demo.c
  $ gcc                                -O1 -c -o demo_elfgcc.o    demo.c
  $ clang -target x86_64-pc-windows-msvc -O1 -c -o demo_coff.o   demo.c
  $ clang -target x86_64-apple-darwin    -O1 -c -o demo_macho.o  demo.c
  $ clang -target aarch64-linux-gnu      -O1 -c -o demo_arm64elf.o demo.c

And the five files, with the format each one actually is:

  demo_elf.o       1952 bytes  7f 45 4c 46   ELF64 x86-64
  demo_elfgcc.o    2136 bytes  7f 45 4c 46   ELF64 x86-64, from gcc
  demo_coff.o      1098 bytes  64 86 07 00   COFF x86-64
  demo_macho.o     1224 bytes  cf fa ed fe   Mach-O 64-bit x86-64
  demo_arm64elf.o  2176 bytes  7f 45 4c 46   ELF64 AArch64

Three things about that table are already worth knowing.

One compiler produced four of the five.clang -target emits ELF, COFF and Mach-O from the same frontend, which is why the comparison later in this module is as fair as it can be — the same source, the same optimisation level, the same version, differing only in the target. gcc is there as an independent second opinion on the ELF half, and it earns its place immediately: it disagrees with clang about which section a relocation should live in.

COFF is the smallest and Mach-O is not far off, while AArch64 ELF is the largest. That is not because AArch64 programs are more complex. It is because a single reference to an address needs two fixups on AArch64 and one on x86-64, and that fact drives an entire module of this course.

And the build is byte-reproducible, which matters more than it sounds. Every claim in this course is "these exact bytes mean this exact thing", so the specimens are committed and a learner can re-derive them:

$ clang -target x86_64-pc-linux-gnu -O1 -c -o re_elf.o demo.c $ cmp demo_elf.o re_elf.o && echo identical identical

What the linker is actually given

Before the concepts start, one look at the whole job, so every later detail has a place to land. A linker takes objects and does, in order:

  1. read and validate each object's headers, sections and symbol tables
  2. pull in archive members, and only those that resolve something still missing
  3. resolve every symbol reference to exactly one definition, or fail
  4. select between competing definitions where the format allows it — weak symbols, COMDAT groups
  5. assign addresses to sections, honouring alignment and any linker script
  6. apply relocations, which means filling the holes this course is about
  7. write out an executable or a shared library, with segment headers, an entry point, and its own symbol table

Steps 1 and 6 are this course. Everything between them is the static linking course, and step 7 onward belongs to loading. The object file is the contract between the compiler and all of that — and it is a contract written in bytes, with no schema and no validation beyond the lengths.

A Real Example

Why the format has to exist at all, told as the problem that a single-file executable does not have. Consider what happens when you split one program into two files:

util.c main.c int helper(int x) extern int helper(int) -- returns x * 2 int main(void) -- returns helper(21) Compile them separately. util.o has: .text 8 bytes a function named "helper" no mention of main Compile main.c separately. main.o has: .text 12 bytes a CALL, and the 4 bytes it needs are currently ZERO -- because the address of helper is not known yet a relocation saying: offset 4, patch 4 bytes, relative to the symbol "helper", minus 4 Neither file can be finished. Neither file is broken.

And the alternatives are worse. Compile both files together and you lose parallelism — a large C++ project has thousands of translation units, and the whole reason cc -c followed by a separate link exists is that a thousand compilations should happen in parallel and only one cheap step should follow.

Or let the compiler invent the addresses. That is the alternative design, and it is what a whole-program compiler does: parse everything, then emit final code. It works, and it is faster at run time because there is no indirection. It also cannot build a library. A library has to be produced before its users are known, so its addresses cannot be final, so it has to ship with holes in it. The moment you want to distribute a compiled library, the object format is forced on you.

The trap that catches people, and it is a real-world one. It is natural to assume an object file is "an executable that is not quite finished". It is not, and the difference is not academic. An ELF or COFF object has every section address equal to zero and no program headers at all — there is nothing to map, because mapping is a decision made at link time. A Mach-O object, by contrast, does assign addresses to its sections, as though it were already a small image. Same idea, opposite convention, and a tool written for one will silently produce garbage for the other. The next concepts measure this; the fourth is entirely about the missing table.

One more practical consequence, because it is the reason a linker exists as a separate program rather than a library call. The object file is the only interface between the two halves of a toolchain, and it is a stable one. A compiler and a linker from different vendors interoperate because they agree on this format and nothing else — no shared headers, no build system, no library version to match. That is why the format survived every change to compilers and linkers for forty years while everything around it was rewritten: it is a file, and files do not need to be recompiled together.

Try It Yourself

$ clang -target x86_64-pc-linux-gnu -O1 -c -o demo.o demo.c
$ llvm-readobj-21 --file-headers --sections demo.o
$ llvm-objdump-21 -r demo.o

Check Your Understanding

Without looking back: a linker is given demo_elf.o and nothing else. It must decide where .text will sit in the output, and it must turn four zero bytes at .text offset 4 into a correct reference to global_counter. Why can the object file not answer the first question, and what is the minimum information the second one needs that the file does supply?

Apply It

You are writing a build tool that reports, for a given object file, which of its symbols are defined here, which are merely referenced, and which are both. On ELF it works. On COFF it reports every function as "referenced only", including compute, which is plainly defined in the file. The file is valid and nm agrees with you. What is different about COFF's symbol table, and what is the general principle for a tool that must read a symbol table it did not write?

The general habit: a table's schema lives in the specification, not in the values. Before trusting a field, know what distinguishes "absent" from "present and zero" — because in most of these formats, and in every one of them today, it does not.

Connect

The "one file, three formats" method used throughout this course is the same method the collection already uses elsewhere, and it is worth knowing why it works. In the ELF and Mach-O courses, a single file is read field by field; here the same file is read in three formats at once. The first approach tells you how a format works. The second tells you which parts of a format are essential and which are accidents of history — and that is a strictly more useful thing to know, because the accidents are exactly what you must not copy when you design the fourth one.

Two examples of that distinction, both measured later in this course. Alignments are an accident: ELF stores a byte count, Mach-O stores a logarithm of it, and both describe the same two alignments. Nothing about an alignment requires either choice, which means a linker author has to convert, and a format designer gets to choose. Section names being unique is an accident of ELF's choice to store names as offsets into a shared table; COFF uses an inline 8-byte field, permits two sections to share a name, and this course's own 1098-byte specimen has two sections both called .rdata. A linker keyed on section name loses one of them silently.

Compare that with the things that turn out to be essential, because the same evidence shows them being reinvented. The relocation record is essential: all three formats have one, with the same four ingredients — where, how wide, what arithmetic, which symbol. COMDAT is essential: ELF calls it SHT_GROUP, COFF calls it IMAGE_SCN_LNK_COMDAT, Mach-O calls it a linkonce section — three names, invented independently, because the problem of two translation units defining the same inline function is unavoidable. Weak symbols are essential for the same reason. A format designer who does not think of these will reinvent them badly, and this course is partly a catalogue of what happens if you do.

The pipeline itself connects to courses you have not met. ELF's sections-versus-segments is the distinction this course's fourth concept is entirely about: one table or two, and which question each table answers. ELF's relocation entries and COFF's relocations are the per-format detail this course assumes you have met; the WebAssembly course's custom sections are the same escape-hatch idea in a different container. And DWARF is what an object file carries for free — a compiler emits debug sections into the object, and a linker passes them through, which is why a stripped binary can be unstripped and a debug binary still runs at full speed.

Next: a hole and a record — the one mechanism the whole format exists for, taken apart byte by byte.