Back to course

One Source, Three Formats

20 minintermediateModule 1: Why Object Files ExistContinues from obj-the-hole
20 min · Module 1: Why Object Files Exist · Intermediate

Why This Matters

The previous concept read one format's relocation record. That is the right way to learn a format and the wrong way to learn what a format is — because every choice you saw in ELF looked like a property of object files rather than a decision ELF's designers made.

So: one 30-line C file, compiled to ELF, COFF and Mach-O, and every field read three times. What comes out is a much sharper picture, because the things that are the same across all three are the ones you cannot do without, and the things that differ are the ones you are free to choose.

This is the most useful single technique in the course, and it is worth being explicit about why it works. Reading one format teaches you how to read ELF. Reading three side by side teaches you what an object file has to contain — and the second is the thing you need in order to design a fourth one, which is where this course is ultimately going.

A Simple Model

Three files from one source, and the first thing to notice is that they are not translations of each other. They are three answers to the same question, by three different designers, in three different decades:

  demo_elf.o       1952 bytes   ELF64      1990s design, still current
  demo_coff.o      1098 bytes   COFF       1980s design, still current
  demo_macho.o     1224 bytes   Mach-O     1980s design, still current

  1952 / 1098 / 1224 bytes for the SAME 30 lines of C.

  A format is a set of answers to five questions:
    1. how do I name a region of bytes?        -> a section
    2. how do I name a thing?                  -> a symbol
    3. how do I store a name?                  -> three different answers
    4. how do I say "these bytes are wrong"?   -> a relocation
    5. where will it go?                       -> NOWHERE. not answered yet.

Four of the five questions get answered. The fifth is the entire point of the format. Every difference you are about to see is an answer to question 3, or a disagreement about how to phrase questions 1 and 2. None of them is a disagreement about the mechanism, because the mechanism is identical in all three.

The Actual Detail

The cleanest comparison available: llvm-nm-21 on all three files, side by side. Same source, same compiler, same optimisation level.

  SYMBOL            ELF           COFF              Mach-O
  ----------------  ------------  ----------------  ------------------
  compute           00000000 T    00000000 T        00000000 T _compute
  call_out          00000020 T    00000020 T        00000020 T _call_out
  use_data          00000030 T    00000030 T        00000030 T _use_data
  global_counter    00000000 D    00000000 D        00000048 D _global_counter
  message           00000008 D    00000008 D        00000050 D _message
  message_bytes     00000000 R    00000000 R        0000005e S _message_bytes
  uninitialised     00000000 B    00000000 B        000000f8 S _uninitialised
  external_fn       (undefined)   (undefined)       (undefined) U _external_fn
  string literal    --            00000000 R        --
                                 ??_C@_05CJBACGMB@hello?$AA@
  clang marker      --            00000000 a @feat.00  --

Five things fall out of that table, and the first is the one that makes the technique worth the trouble.

The code offsets are identical. The data offsets are not.

compute is at 0, call_out at 0x20, use_data at 0x30 — byte-for-byte the same in all three files. That is not a coincidence: .text comes first, nothing precedes it, so every code symbol's offset within its section is the same regardless of format.

Now the data. ELF and COFF both say global_counter is at offset 0 and message at 8. Mach-O says 0x48 and 0x50 — real numbers, not offsets into a section, and they are exactly where those sections were placed in the object's address space.

  Mach-O section addresses in demo_macho.o:

      __text      0x00     __cstring   0x58
      __data      0x48     __const     0x5e
                            __common    0xf8

  and the symbol values are those addresses verbatim:
      _global_counter  0x48  = __data + 0
      _message         0x50  = __data + 8
      _message_bytes   0x5e  = __cstring + 0
      _uninitialised   0xf8  = __common + 0

  ELF and COFF have NO addresses. 0 and 8 are offsets within
  their own section, and "which section" is a separate column.

So the same fact — where a variable lives — is recorded as an offset plus a section index in two formats, and as a single absolute number in the third. A linker written for the first two will compute nonsense from the third, and it will not crash — it will place message 0x48 bytes into the output instead of 8, and the program will read the wrong memory. This is the single most important practical difference in the module, and it is invisible unless you look at both.

The letter vocabulary changed

ELF and COFF both use T for text, D for data, R for read-only, B for uninitialised. Mach-O uses T and D identically and then uses S for both message_bytes and uninitialised — two different kinds of storage collapsed into one letter.

That is a real information loss in the display, and it is worth understanding why it is not a loss in the file. message_bytes is in __cstring with Offset: 792 — real bytes in the file. uninitialised is in __common with Offset: 0 — no bytes in the file at all. The distinction is not in the symbol's letter, it is in the section's offset. Mach-O trusts the section table to carry what the letter cannot, and nm is the thing that loses information.

COFF has two symbols nobody asked for

ELF and Mach-O list eight symbols. COFF lists ten, and the extras are both instructive:

  ??_C@_05CJBACGMB@hello?$AA@   the string literal "hello"

      MSVC mangles string literals into the symbol name: ?_C@ then
      a length byte, then the characters with '?' for '.', then
      ?$AA@ to mark the end. The literal is a first-class symbol.

  @feat.00                          a clang-internal marker

      clang emits this so its C output can be consumed by a C++
      linker that might otherwise apply C++ mangling. It is a
      compatibility hack between two languages, stored in the
      object file.

  ELF has 12 entries in .symtab rather than 8: the extra four are
  the STT_FILE entry naming demo.c, and STT_SECTION entries for
  .text and .rodata.str1.1 -- and only those two, because only
  those two are named by a relocation.

That last fact is the subtlest thing in the table. A section symbol is a convenience entry meaning "this section, as a whole". ELF emits one for .text and for .rodata.str1.1 and for nothing else — not for .data, even though .data exists and even though .data has a relocation against it. The reason is demand: the relocation against .rodata.str1.1 targets the section because that is where a string literal pool lives, while the relocation against .data targets the variable message by name and needs no section symbol.

For a linker author this is a trap with a specific failure. A natural implementation resolves a relocation's symbol by always looking for a section symbol when the referenced thing is a section, and falls back to a name lookup otherwise. That works. The bug is a tool that pre-allocates a section symbol for every section because assuming one always exists is easier — and then a tool that compares symbol tables against a reference reader finds two extra entries in one and not the other, for a file that is completely valid. A section symbol is a fact about what something referenced, not a fact about what sections exist. ELF's own llvm-readobj output shows exactly this: STT_SECTION entries for two sections out of sixteen.

Sections, side by side

The same comparison for the region table, which is where the storage strategies diverge most:

  ELF   16 entries   .text .data .bss .rodata .rodata.str1.1
                              .rela.text .rela.data .rela.data.rel.local
                              .eh_frame .rela.eh_frame .note.GNU-stack
                              .comment .symtab .strtab .shstrtab
                              .llvm_addrsig

  COFF   7 entries    .text .data .bss .rdata .rdata
                              .debug$S .llvm_addrsig
                              -- note: .rdata appears TWICE

  Mach-O 6 entries   __text __data __cstring __const
                              __common __eh_frame

Three observations that each cost a linker author something.

A Real Example

Sorting the differences into essential and accident, because that distinction is the transferable skill and it is the reason to do this exercise at all.

ESSENTIAL -- three independent teams all invented these, because the problems are unavoidable: the section table group bytes by purpose the symbol table name a thing, or want one named the relocation record say "these bytes are wrong" a way to store a name ... and every format needs one 64-bit and 32-bit targets ... and they must coexist ACCIDENT -- one team's choice, another did differently, and both are correct: inline name vs offset ELF offsets, COFF 8 bytes, Mach-O 16 bytes one string table or two ELF needs two, others one sections carry addresses Mach-O yes, ELF and COFF no alignment units ELF a byte count, Mach-O a log a reloc type enum ELF and COFF enum, Mach-O bits how to spell read-only R in ELF/COFF, S in Mach-O what to call a call PLT32 / REL32 / BRANCH the practical consequence: a linker needs an ARCHITECTURE plugin (essential, per-arch) and a FORMAT plugin (accident, per-format), and those are two different pieces of code with two different reasons to exist.

The compiler difference is worth separating out, because it is a fourth axis and it is the one that bites in practice. Same source, same flags, two ELF producers:

              clang 21            gcc 15.2
  size          1952 bytes         2136 bytes
  .text holes   0x04 0x0a          0x06 0x10
                0x21 0x33          0x1e 0x2e
  data reloc    in .data           in .data.rel.local
  first func    compute at 0       compute at 0
  last func     use_data at 0x30   use_data at 0x2e

gcc emits a section ELF's specification does not require anyone to emit, and puts a relocation in it. Both files are valid. A tool that hard-codes "the sections a relocatable object has" is wrong for one of the two most common producers on Earth — and the failure is a missing relocation on a data pointer, which links cleanly and produces a binary that reads the wrong address at run time.

Why gcc does this, and why it is defensible..data.rel.local exists so the linker can discard that section once the relocations are applied, for data that is only referenced within the object and becomes dead once linked. Putting such relocations in .data would make them indistinguishable from relocations that must be kept. The section name is carrying a policy the linker can act on, which is a good reminder that a section name is not always just a label — and that an unfamiliar one may encode an optimisation you have not implemented.

And the deepest lesson, which is the one a compiler author needs. Look again at the relocation counts: 4 on x86-64 ELF, 8 on AArch64 ELF, 4 on COFF, 4 on Mach-O. The AArch64 doubling is not a format difference at all — it is an instruction encoding difference, and it arrives in the object file unchanged. The object format did not choose to record two fixups; it had no choice, because the code contains two fields that need filling.

So the dependency runs instruction set → object format → linker, and the object format is downstream. That is why this course has a module on encodings, and why an object-file course that skipped the architecture would be teaching a mechanism without knowing why its shape is what it is. A learner who wants to write a code generator needs the encodings first; a learner who only wants to read object files can skip them, and the course marks which is which.

Try It Yourself

$ llvm-nm-21 courses/obj/assets/samples/demo_elf.o   # and coff, and macho
$ llvm-readobj-21 --sections courses/obj/assets/samples/demo_coff.o

Check Your Understanding

Without looking back: llvm-nm reports global_counter at 00000000 in demo_elf.o and at 00000048 in demo_macho.o. Meanwhile compute is at 00000000 in both. Why do the code symbols agree and the data symbols not, and what specifically breaks in a linker that reads both formats with one code path?

Apply It

You are writing an object-file inspector for a build tool that must handle output from any toolchain. It reads a symbol's section and value, adds them, and reports a file-absolute address. It is correct on every ELF object in the project and on every COFF object. On Mach-O objects it reports addresses that are far too large, and downstream code that opens files by computed offset fails. The tool has a single read_symbol function and a format string it switches on in three other places. What is the actual defect, and why is adding a fourth if format == "macho" the wrong fix?

The general habit: a value read from a file is not a value until you have decided what it means. Encode that decision in the type your reader returns, not in a branch at each place you use it — then the assumption has nowhere to hide.

Connect

This concept is the course's method made explicit, and the method is the same one the collection uses in the JVM class file course, where one 409-byte Hello.class was read field by field and then again as the source of a pool that turned out to be 69% of the file. In both cases the claim being tested was a number, and in both cases the number came from a file rather than from memory. The discipline is identical: measure, then teach, and never assert a field's value because it is what the format "should" put there.

The essential-versus-accident distinction is the same one the JVM attribute mechanism turns on, and it is worth stating as a general reading skill. In ELF a section name is an offset into .shstrtab; in COFF it is 8 bytes inline; in Mach-O it is 16 bytes inline. All three are accidents, and all three are specified. But SHT_GROUP, COFF's IMAGE_SCN_LNK_COMDAT and Mach-O's linkonce section are three names for the same essential idea, invented by three teams who all hit the same problem. When you see one format reinvent something another already has, that is the signal it is essential — and the signal that your own design will need it too.

The compiler-divergence finding connects to something with real consequences. ELF's common sections teaches .data.rel.local and friends as ELF features, and this concept shows why they exist: they are an optimisation hint the linker acts on, and gcc uses one that clang does not. A tool that reads section names as opaque labels throws away a policy the producer encoded on purpose, and the ELF course's warning about e_shstrtab being a place a producer can legally lie to you applies here too.

The instruction-encoding dependency is the hinge to the architecture section, and it is worth being blunt about the direction. Object-file formats are downstream of instruction sets. The reason ELF has R_AARCH64_ADR_PREL_PG_HI21 as a distinct type is that adrp has a 21-bit field; the reason x86-64 can get by with R_X86_64_PC32 is that mov with a RIP-relative operand has a 32-bit displacement. Neither format chose this. So a code generator cannot be written before the encodings are understood, and a linker cannot be written before the encodings are understood either — the linker writes into those fields and must know they are wide enough. That is the argument for the Encoding concept, and it is a structural one rather than a pedagogical preference.

Two more connections, both about tools rather than formats. Mach-O's symbol table teaches the leading underscore and the S letter, and this concept shows where they come from: Mach-O prefixes C symbols because its runtime does, and it collapses read-only and uninitialised into one letter because its section table already distinguishes them. Almost every oddity in a format is a decision made to solve a problem the neighbouring format did not have. And COFF is the format whose object file is the subject of this course — its first module is "The Relocatable Object" — so if you have read it, this course's job is the part COFF alone cannot teach: that the same seven questions have seven different answers, and which of those answers you are free to change.

Next: the table that is not in any of these files, and the reason it cannot be.