Back to course

Line Numbers

16 minintermediateModule 3: Containers and VariantsContinues from coff-archives
16 min · Module 3: Containers and Variants · Structures

Why This Matters

Every section header you decoded in Module 1 had ten fields, and eight of them had a non-zero value. Two did not: PointerToLinenumbers and NumberOfLinenumbers, both zero in every section of every object in this course.

That is not an oversight and it is not dead weight. COFF has native support for mapping code addresses to source lines — it was one of the format's original features, decades before DWARF existed — and this course is built on a toolchain that does not use it. Knowing which fields a compiler leaves alone, and why, is the difference between reading a format and guessing at it.

A Simple Model

COFF's line-number support is the simplest possible design, and it is worth contrasting with what came later.

A section header points at a table of 6-byte records, each a pair:

offset size field 0 4 SymbolTableIndex which symbol in the section's symbol table 4 2 LineNumber the line, in the units defined below

The table ends with a record whose two fields are both zero. That is the entire format: a sorted list of (symbol, line) pairs, and LineNumber is monotonically increasing within a symbol's run. There is no address in the table at all — the address comes from the symbol, and the symbol's Value is an offset within the section, exactly as in the symbol table concept.

Three properties follow, and they are why the design is so small:

Why that last point is a problem, not a simplification. A plain (symbol, line) pair assumes code is laid out in source order, which is true at -O0 and false the moment a compiler reorders, interleaves or inlines. An optimiser that hoists a loop, merges two functions or lays out a cold block separately cannot express that with these records, because the format has nowhere to say “this line moved”. COFF line numbers work for a debug build and have no way to work for an optimised one — which is precisely the gap DWARF was designed to close, with a state machine, a location list per variable, and a line program that can describe code in any order.

The Actual Detail

What the compiler actually does

Every section of the reference object, from llvm-readobj:

$ llvm-readobj --sections sample_msvc.obj | grep -E 'Name:|LineNumberCount'
    Name: .text            LineNumberCount: 0
    Name: .data            LineNumberCount: 0
    Name: .bss             LineNumberCount: 0
    Name: .xdata           LineNumberCount: 0
    Name: .rdata           LineNumberCount: 0
    Name: .rdata           LineNumberCount: 0
    Name: .debug$S         LineNumberCount: 0
    Name: .pdata           LineNumberCount: 0
    Name: .llvm_addrsig    LineNumberCount: 0

Nine sections, nine zeros. And it is not that clang lacks the feature: the same objects carry a .debug$S section holding CodeView debug data, and llvm-objdump can disassemble .text with line annotations. clang simply does not write COFF line numbers — it writes CodeView records into a section that the COFF section header has no field for, which is a different mechanism entirely.

The same is true of the GNU dialect. The i386 build, same source, same answer.

Verifying the field layout anyway

A field that is always zero cannot be verified by reading it, so the layout was verified by construction. A line-number table was appended to a real object and the section header patched to point at it:

$ python3 -c "
import struct, shutil
shutil.copy('sample_msvc.obj','lineno.obj')
d = bytearray(open('lineno.obj','rb').read())
entries = [(0,0x1000), (1,3), (0,0x2000), (1,4), (0,0)]
tbl = b''.join(struct.pack('<IH', s, l) for s, l in entries)
linptr = len(d)
d += tbl
struct.pack_into('<I', d, 0x14 + 28, linptr)   # PointerToLinenumbers
struct.pack_into('<H', d, 0x14 + 34, len(entries))  # NumberOfLinenumbers
open('lineno.obj','wb').write(d)
print('appended at 0x%x, %d entries, %d bytes' % (linptr, len(entries), len(tbl)))
"
$ llvm-readobj --sections lineno.obj | head -12
    Number: 1
    Name: .text
    PointerToRelocations: 0x1E8
    PointerToLinenumbers: 0x5AF     <- read back exactly what was written
    RelocationCount: 3
    LineNumberCount: 5

The reader reports PointerToLinenumbers: 0x5AF and LineNumberCount: 5, which is precisely what was written. That proves the two fields are where the specification says they are, and that the count is a record count rather than a byte count. It is shipped as lineno.obj alongside the other samples.

What that does and does not prove. It proves the field offsets, because an independent reader found the values at the positions the specification predicts. It proves nothing about what the line numbers mean, and the temptation to teach a semantics here is exactly the mistake this course keeps documenting elsewhere — the relocation layout that the specification documents and the compiler does not emit, the DWARF high_pc form that turns an address into a length. So: the layout is verified and taught; the semantics are not, and are not taught.

Why the values would not mean much anyway

This is the part worth understanding, because it explains a design that looks like an oversight.

In a COFF object, LineNumber cannot be a source line number in any useful sense. Consider what a debugger would have to do with the table above: symbol 0, line 0x1000 — 4096. Line 4096 of a 30-line file? The value is an ordinal into a line-number program that lives in the debug information, and that program is produced during the link, when the real source line for each address is finally known. In a standalone object the referent does not exist.

You can watch this go wrong. Disassemble the constructed file with line annotations enabled:

$ llvm-objdump -l --section=.text lineno.obj
0000000000000000 <add>:
; add():
       0: 50        pushq %rax
; g_ptr():
       8: 8b 04 24  movl (%rsp), %eax
; g_addr():
      10: c3        retq

The annotations name add, g_ptr and g_addr — functions that have nothing to do with the invented numbers, attached to instructions that are all from one function. The disassembler is resolving the ordinals against whatever symbol data it can find, and the result is confident, plausible, and meaningless. That output is the best possible argument for the discipline this course follows: a field whose meaning you cannot verify should not be taught, and a value that decodes without complaint is not thereby a value you understand.

A Real Example

What the table looks like when it is meaningful, so the shape is clear. Suppose an unoptimised build emitted this for .text:

SymbolTableIndex  LineNumber        what it means
------------------  ---------------   ----------------------------
       20             3                symbol 20 = add(),  line 3
       20             4                ...still add(),  line 4
       20             5
       20             6
       21            11                symbol 21 = call_through(), line 11
       21            12
      ...
        0             0                the terminator: both fields zero

And how a reader turns an address back into a line with it:

  1. Find the symbol. Given address A within section S, scan the section's symbols for the one whose Value is the greatest value not exceeding A - S.base. That is symbol 20.
  2. Scan the table for that symbol's run. The rows with SymbolTableIndex == 20.
  3. Take the last row whose line value is not greater than the one you want. Because the values increase within a run, the table is effectively sorted per symbol and can be scanned or binary-searched within the run.
  4. Report that line. There is no address to check, no file to name and no column to give, because the table has nowhere to put any of them.

Compare that with the DWARF line number program, which does the same job and additionally stores the file, the column, a discriminator for macro expansions, a basic-block flag, and an explicit end-of-sequence marker per function. DWARF also handles the case COFF cannot express at all: code that is not in source order. The COFF table has three fields; the DWARF program is a state machine with a dozen opcodes, and the difference in size is the difference between a format that can only describe a debug build and one that can describe any build.

Try It Yourself

$ llvm-readobj --sections sample_msvc.obj | grep -c 'LineNumberCount: 0'
$ llvm-readobj --sections sample_gnu.obj  | grep -c 'LineNumberCount: 0'
$ llvm-readobj --sections sample_m32.obj  | grep -c 'LineNumberCount: 0'

All three should report every section, in all three dialects. Then confirm the fields are real rather than vestigial:

$ python3 -c "
import struct, shutil
shutil.copy('sample_msvc.obj','lineno.obj')
d = bytearray(open('lineno.obj','rb').read())
entries = [(0,0x1000), (1,3), (0,0x2000), (1,4), (0,0)]
tbl = b''.join(struct.pack('<IH', s, l) for s, l in entries)
linptr = len(d); d += tbl
struct.pack_into('<I', d, 0x14 + 28, linptr)
struct.pack_into('<H', d, 0x14 + 34, len(entries))
open('lineno.obj','wb').write(d)
"
$ llvm-readobj --sections lineno.obj | grep -E 'PointerToLinenumbers|LineNumberCount'
$ llvm-objdump -l --section=.text lineno.obj | head

Three checks, in order of value:

And the negative experiment worth doing: look for a compiler that does emit COFF line numbers, if you have MSVC's cl.exe to hand. cl /Zi writes CodeView data; the COFF-native tables are what you get from other toolchains. Confirming which of the two a given compiler writes is a more useful habit than memorising either layout.

Check Your Understanding

Without looking back: you are writing a tool that lists the source lines covered by a COFF object. Every section reports NumberOfLinenumbers: 0. What is the correct conclusion, and what must you not conclude?

Apply It

Your build tool wants to map a crash address to a source line for a COFF object, and gets plausible-looking but wrong answers. You have established that the objects report NumberOfLinenumbers: 0 and that the only line information present is CodeView in .debug$S. What is actually going on, and what is the correct fix?

The general habit: before debugging a value, establish that the thing it refers to exists. A number that indexes into a program, a table, or a pool is not a value until the pool is in front of you — and a decoder that never checks will hand you plausible answers indefinitely.

Connect

This is the direct contrast the whole module has been building toward. The DWARF line number program solves the same problem with a state machine, explicit file and column, a discriminator for macros, and per-function end-of-sequence markers — and it works on optimised builds, because a program can describe code in any order. COFF’s three-field table can only describe code that is laid out in source order, which is why it works at -O0 and has no way to work anywhere else.

The same shape of trade-off runs through the COFF course: the relocation table is ten bytes per record and cannot express an arbitrary computation, while DWARF’s location expression is a small program that can. Both formats put a bound on how much a reader can describe, and in each case the answer to “what happens when the compiler does something the format cannot express” is the same: a new format, or a new mechanism. CodeView in .debug$S is the Windows answer to that question, and it is why a COFF section header has no field for it.

That closes Module 3. The two working halves of COFF are complete — the container that says what the pieces are, and the tables that let a linker place them — along with the containers and variants that surround them.

What is still specified in courses/coff/research.md and deliberately unwritten: linker map files, and the incremental-linking and LTCG machinery, neither of which can be produced or observed on this machine because there is no COFF linker installed. The landing page lists them without links rather than as 404s.