Back to course

Why a Binary Format

15 minbeginnerModule 1: The Container
15 min · Module 1: The Container · Foundations

Why This Matters

Every binary format in this collection exists because somebody needed to put one thing in a file where two parties could not both be the source of truth. The ELF header exists so a kernel and a compiler agree on where the program starts. The DWARF sections exist so a compiler and a debugger agree on what a variable means. WebAssembly's binary format exists so that a compiler and a browser agree on what a program does, across language and vendor boundaries, with the agreement settled by a written specification rather than by a conversation.

That framing matters more for WebAssembly than for the others, because here the two parties are unusually far apart. A compiler written in one language emits code for a runtime written in another, on a machine neither author controls, and the only thing they share is this file format. There is no header to be added by a vendor, no field that means "vendor private", no room to negotiate. Eight bytes of fixed prefix, then a strictly ordered sequence of length-delimited sections, and that is the entire negotiation surface.

The rigidity is the feature. It is why a WebAssembly module can be validated by a fifty-line checker before it is allowed to run, and why the same file can be handed to a JIT, an interpreter, or a validator and all three agree on what it contains. Every design decision in this module is downstream of that goal: a format that can be checked without executing it, by code that does not trust it.

A Simple Model

What is actually in the file. The smallest legal module, in full:

00 61 73 6d 01 00 00 00
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  +-- version 1, FOUR fixed bytes
|  |  |  |  |  |  |  +----- little-endian, so 01 comes first
|  |  |  |  |  |  +-------- the 0 byte here is the high half
|  |  |  |  |  +----------- an ASCII "m"
|  |  |  |  +-------------- an ASCII "s"
|  |  +------------------ an ASCII "a"
|  +--------------------- 0x00, the NUL that makes it not a text file
+------------------------ the magic: 0x00 followed by the letters "asm"

Eight bytes, and that is a complete, valid, empty WebAssembly module. Not a stub and not a placeholder — a module that does nothing, correctly encoded, that a runtime will load and instantiate without complaint.

Build it yourself in two commands, which is the fastest way to make the format concrete:

$ printf '(module)' > min.wat
$ wat2wasm min.wat -o min.wasm
$ xxd min.wasm
00000000: 0061 736d 0100 0000                      .asm....
$ wc -c < min.wasm
8

Two interesting things in that output. The magic is 0x00 followed by the ASCII letters asm — the format is literally named by its own first four bytes, with a NUL in front so that the file is not mistaken for text. And xxd's ASCII gutter prints the same eight bytes as .asm...., which is the format telling you what it is in a display you can read at a glance.

The three tools, and why there are three

This course checks every claim against three independent implementations, and the reason is not ceremony. A binary format has almost no redundancy in it: if a decoder misreads a size by one byte, it still produces a table that looks entirely reasonable. The only defence is disagreement.

ReaderWhere it comes fromWhat it is good for
wasm_decode.pywritten from the specification, shipped with this courseBeing wrong in the way the specification is ambiguous, rather than in the way an implementation is sloppy
wasm-objdumpwabt, the reference toolchainThe most complete section-aware reader; the disassembler
llvm-objdumpLLVM, which also compiles to WebAssemblyA second implementation with entirely different internal assumptions

And a fourth participant, which is a reader of a different kind: clang --target=wasm32. It is a producer rather than a reader, and it matters because it gives this course something the ELF and COFF courses do not have in the same form: a real compiler that emits real modules from real source, so every claim about the format can be checked against a file that something intended rather than something hand-assembled.

$ clang --target=wasm32 -O2 -c w2.c -o w2.o
$ file w2.o
w2.o: WebAssembly (wasm) binary module version 0x1 (MVP)

Note what that object is: a WebAssembly module. Not a container wrapping one, not a distinct format. The compiled output of one C file is already a module, with the same header as the eight-byte one above, and it can be handed straight to a runtime. That is a genuinely unusual property — ELF objects and COFF objects both need a linker, and what the linker produces is a different file — and the module format doubles as the object format because there is no separate "unlinked" representation.

The Actual Detail

The crosscheck, and what it is protecting against

Run it on the samples shipped with this course:

$ cd courses/wasm/assets/samples
$ python3 crosscheck.py
ok    externs.o                10 sections, three readers agree
ok    handwritten.wasm          9 sections, three readers agree
ok    long_custom.wasm         10 sections, three readers agree
ok    minimal.wasm              0 sections, three readers agree
ok    simple.o                  7 sections, three readers agree

5 file(s) checked, 0 with disagreements
CROSS-CHECK PASSED -- all three readers agree on every section

A green result is worth nothing on its own, so the harness is itself tested by breaking files on purpose. Three corruptions, three detections:

Injected faultResult
one byte of a section's size changedFAIL — section count disagreement
two section ids swappedFAIL — section count disagreement
one byte of the magic changedFAIL — section count disagreement

And notice how the tools disagree about broken files, because it is instructive. On a file with a duplicated Type section:

$ wasm-validate dup.wasm
  0000063: error: multiple Type sections

$ llvm-objdump -h dup.wasm
llvm-objdump: error: 'dup.wasm': out of order section type: 1

$ wasm-objdump -h dup.wasm
   ...prints a section table quite happily...

Three behaviours, and the difference is a design decision each tool made. wasm-validate is a validator, so it reports the specific rule it broke. LLVM expresses the same rule as an ordering violation — "out of order section type" — because in its model a repeated section is a special case of appearing out of sequence. wasm-objdump is a printer, so it prints; it will happily show you the structure of a file that could never run, and that is a reasonable thing for a disassembler to do.

Which produces a habit worth having. A tool that prints something is not a tool that vouched for it. Throughout this course, "the disassembler shows" and "the validator accepts" are different claims and are never used interchangeably. The 1.0.36 wabt and LLVM versions quoted here are what produced every number in this module.

What is in a real module

The samples are not toys. simple.o is what clang emits for two C functions, and externs.o is what it emits when those functions touch a static array and an external global:

$ wasm-objdump -h simple.o
  Type start=0x0000000e end=0x0000001a (size=0x0000000c) count: 2
Import start=0x00000020 end=0x00000038 (size=0x00000018) count: 1
Function start=0x0000003e end=0x00000041 (size=0x00000003) count: 2
Code start=0x00000047 end=0x00000058 (size=0x00000011) count: 2
Custom start=0x0000005e end=0x0000007e (size=0x00000020) "linking"
Custom start=0x00000084 end=0x000000bc (size=0x00000038) "producers"
Custom start=0x000000c2 end=0x00000156 (size=0x00000094) "target_features"

Four standard sections and three custom ones, and the custom ones are where a compiler puts everything the module format does not have a place for: which toolchain produced it, which target features it needs, and — in linking — the information a linker needs. That last one is why there is a Module 5 in this course at all. For now the point is narrower: the standard sections are a closed, specified set, and everything a real compiler needs to say beyond them goes in custom sections that the format promises not to interpret.

A Real Example

Why a format with this little redundancy has to be read by more than one tool. Take the section table of simple.o and remove any one of the three readers, and ask what class of bug it would now fail to catch.

a decoder is wrong about a size by one byte -> it starts the next section one byte early -> it reads a plausible section id from the middle of a payload -> the id is usually a small number, so it is usually VALID -> the following "size" is also a plausible small number -> the section table it prints has a plausible number of rows -> NOTHING looks wrong the same bug, caught by disagreement my decoder says the next section starts at 0x16 wabt says the next section starts at 0x16 llvm says the next section starts at 0x16 two of them agree -> the byte is probably 0x16 all three agree -> the byte IS 0x16, because a single shared misunderstanding of the format is far less likely than a single implementation's slip

That is the whole argument, and it is the reason this course's harness exists rather than a prose promise. Note the shape of the failure: a decoder that is off by one does not look broken, it looks like a decoder for a slightly different format. And there are many slightly different formats to be wrong about — the size convention for custom sections differs between tools, as the next concept shows, and a decoder that has silently picked the wrong one produces a table that is wrong in a way nobody can see.

The same argument applies to reading the specification. Where a spec is ambiguous, two implementations will differ, and the difference is a fact about the format that a third reader can often settle. That happened twice in the previous course in this collection: a readelf and an llvm-dwarfdump disagreed about DWARF 5 location lists, and the byte offsets settled it. It will happen again here. The method is the same and it is the reason this course ships a decoder rather than only prose: an independent implementation written from the specification is the only way to find out whether the specification said what you thought it said.

What you should take from this concept is a working habit rather than a fact. When you next read a binary format, the first question worth asking is not "what does this field mean" but "what would it take to check that claim, and who else has already tried". A format that cannot be checked cannot be learned reliably, and a great many can.

Try It Yourself

$ printf '(module)' > min.wat && wat2wasm min.wat -o min.wasm && xxd min.wasm
$ clang --target=wasm32 -O2 -c mycode.c -o mycode.o && wasm-objdump -h mycode.o
$ cd courses/wasm/assets/samples && python3 crosscheck.py

Check Your Understanding

Without looking back: you have a file that starts with the eight-byte header, then a Type section of size 10, then something that does not parse. You have three readers and they all report a section table. What is the strongest evidence that the Type section's size is really 10, and why is a single reader's agreement with itself not that evidence?

Apply It

You are writing a WebAssembly module builder — a tool that emits .wasm files, rather than one that reads them. Your first output loads in a browser and runs correctly. Your second, which adds one data segment, loads and then silently produces an empty string where the data should be. No error, no warning, from any tool you try. Walk through what you would check, in order, and say what result would send you to which part of the file.

The general habit: on a silent wrong answer, check the length fields before the value fields. In a format with no redundancy, a wrong length does not look wrong — it looks like the next thing in the file.

Connect

WebAssembly's binary format is the fourth container this collection has decoded, and the comparison is the fastest way to place it. ELF and COFF both begin with a fixed header describing the whole file, and both have a section table that can appear in any order. DWARF has no header at all, just a bag of sections the consumer looks up by name.

WebAssembly shares ELF's strict ordering requirement and rejects the rest: its section ids must appear in increasing order, with custom sections as the only exception. That is a stronger constraint than ELF's and it buys something specific — a reader can walk the file once, in one direction, and know when it is finished, without tracking which sections it has already seen. ELF cannot make that promise because a section may legitimately be absent, and its reader must therefore be written to tolerate holes. WebAssembly trades flexibility for a reader that is a loop.

It is also the only one of the four whose container doubles as its object format, which the next two concepts make concrete: there is a header and then sections, and what a compiler emits and what a runtime consumes is the same kind of file. ELF needed a separate .rel section to say how to get from object to executable; WebAssembly will carry that information in custom sections instead, and Module 5 decodes them.

Next: The Eight-Byte Header.