bigobj

20 minadvancedModule 3: Containers and VariantsContinues from coff-comdat

20 min · Module 3: Containers and Variants · Structures

Why This Matters

Two fields in the 20-byte header are 16 bits wide: NumberOfSections and NumberOfSymbols. 65,535 of each sounds like plenty until you meet a machine-generated translation unit, where a single C++ header with templates can emit a hundred thousand COMDAT sections. At that point a compiler has a choice: emit a file it cannot describe, or switch formats.

bigobj is that second format, and its central trick is the reason it is worth a whole concept. It does not widen the fields. It moves them. Every offset after NumberOfSections changes, which means a parser written for the classic layout does not fail on a bigobj — it silently describes a different file.

A Simple Model

Two headers, side by side. This is the whole concept:

Classic (20 bytes)	bigobj (56 bytes)
`Machine` (2)	`Sig1` = 0x0000 (2)
`NumberOfSections` (2)	`Sig2` = 0xFFFF (2)
`TimeDateStamp` (4) `PointerToSymbolTable` (4) `NumberOfSymbols` (4) `SizeOfOptionalHeader` (2) `Characteristics` (2)	`Version` = 2 (2)
	`Machine` (2) — moved here
	`TimeDateStamp` (4)
	`ClassID` (16) — a GUID, new
	`SizeOfData` (4), `Flags` (4), `MetaDataSize` (4), `MetaDataOffset` (4) — all new
	`NumberOfSections` (4) — widened to 32 bits
`PointerToSymbolTable` (4), `NumberOfSymbols` (4)

Note what happened to Machine. In a classic object it is the first field, at offset 0. In a bigobj it is the fourth, at offset 6 — and offsets 0 and 2 now hold the signature. SizeOfOptionalHeader and Characteristics simply do not exist: there is no optional header in any object, so a variant that goes to the trouble of adding 36 bytes does not spend them on a field that is always zero.

The section table moves too, and that is the visible consequence. In a classic object the section table starts at 20. In a bigobj it starts at 56, because it follows the bigger header. And every file offset in the file — PointerToRawData, PointerToRelocations, PointerToSymbolTable — has to be adjusted by the 36 extra bytes, because all of that data was shifted up when the header grew. A bigobj is not the same file with wider integers; it is the same file with everything moved.

The Actual Detail

First: does a compiler ever produce one?

This is the question worth answering before anything else, and the answer is measurable. Here is a translation unit with seventy thousand functions in it:

$ python3 -c "
lines = ['int f%d(int x){return x+%d;}' % (i, i) for i in range(70000)]
open('huge.c','w').write('\n'.join(lines))
"
$ clang --target=x86_64-pc-windows-msvc -c huge.c -o huge.obj
$ file huge.obj
huge.obj: x86-64 COFF object file, not stripped, 7 sections,
          symbol offset=0x468072, 70017 symbols, ...
$ llvm-readobj --file-headers huge.obj
  Machine: IMAGE_FILE_MACHINE_AMD64 (0x8664)
  SectionCount: 7
  SymbolCount: 70017          <- over the 65535 limit
  OptionalHeaderSize: 0

70,017 symbols, comfortably past the 16-bit ceiling, and clang emitted a perfectly ordinary classic object. It did not switch. That is a fact about this toolchain rather than about the format, and it is worth stating plainly: the bigobj path is a capability of the format that this compiler does not exercise.

So: can a bigobj be verified at all?

Constructed one by hand, from the field list above. The naive approach — patch the first word to the signature, the second to the version — produces this:

0000: 00 00 ff ff 02 00 64 86 63 ae b6 6a 00 00 00 00
      |—— Sig1——| |— Sig2 —| |—v2—| |—Machine—| |—TimeDateStamp—|
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0020: 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00
      |—— 16 bytes of ClassID, all zero——| |—NumberOfSections—|
0030: 96 80 46 00 81 11 01 00 2e 74 65 78 74 00 00 00
      |—PtrToSymTab| |—NumSyms| |— ".text" — the section table

This is a bigobj header laid out correctly, built from a real 70,017-symbol object by shifting all its data up by 36 bytes and fixing up the symbol table pointer. Handing it to the reference tools produces something more interesting than success or failure:

$ llvm-readobj --file-headers huge_bigobj.obj
File: huge_bigobj.obj
Format: COFF-import-file-x86-64
Type: code
Name type: ordinal
Symbol: __imp_

$ file huge_bigobj.obj
huge_bigobj.obj: Common Data Format (Version 2.5 or earlier) data

LLVM recognises the file and misidentifies it as a short-import library. And the reason is a signature collision:

The short-import format also begins 00 00 FF FF, followed by a version and a machine. Those first four bytes are the whole of its identity as far as LLVM is concerned.
A bigobj begins 00 00 FF FF, followed by its version and machine. The only difference is the version field: 0 for short-import, 2 for bigobj.
LLVM dispatches on the signature alone. Set the version to 0 and it is still reported as COFF-import-file-x86-64; set it to 2 and it is reported the same way.

And the decisive check: the string bigobj does not appear anywhere in the llvm-readobj binary.

$ strings llvm-readobj | grep -c bigobj
0

There is no bigobj reader in this LLVM. It cannot be used to confirm a bigobj, and the collision means the file it is instead read as something structurally unrelated.

What this means for the course, stated honestly. The field list above is from the specification, and the layout is confirmed by the fact that a correctly built file parses into a coherent structure by hand. But no independent reader on this machine accepts a bigobj, so unlike every other claim in this course it has not been cross-validated against a second implementation. It is taught as a documented layout plus two verified observations — that clang does not emit one, and that the signature collides with short-import and defeats LLVM — and not as a decoded example. The DWARF course handles a comparable situation the same way.

A Real Example

Take the reference object from Module 1 and ask what a classic-layout parser would do with a bigobj. This is the practical content of the concept, and it needs no bigobj to demonstrate.

sample_msvc.obj begins:

0000: 64 86 09 00 2a a5 b6 6a 48 03 00 00 1f 00 00 00
0010: 00 00 00 00 2e 74 65 78 74 00 00 00 00 00 00 00 00
      |—Machine—| |—Secs—| |— TimeDateStamp —| |—PtrToSymTab—|
      |— NumberOfSymbols —| |opt| |chars|  |— ".text" —

Now feed a bigobj to a parser that does exactly this. It reads:

Machine = 0x0000 (Sig1) -> "unknown machine" NumberOfSections = 0xFFFF (Sig2) -> 65535 sections TimeDateStamp = 0x0002 (Version) PointerToSymbolTable = 0x8664 (Machine) NumberOfSymbols = <the real TimeDateStamp> SizeOfOptionalHeader = <low half of PtrToSymTab> Characteristics = <high half of PtrToSymTab>

Every field is wrong, and the parser has no way to know. A machine value of 0 is at least suspicious; 65,535 sections is entirely plausible-looking; the section table would then be read starting at offset 20, which is 36 bytes before the real one, and would interpret the middle of the ClassID GUID as a section name.

Compare that with how a malformed classic object fails. Corrupt the Machine field and file reports data and llvm-readobj refuses it. Corrupt NumberOfSections to something absurd and the section table runs off the end of the file. Both produce errors. The bigobj failure mode is the absence of errors, which is what makes the format's design worth knowing about even if you never meet one.

And the defence is cheap: the signature is at offset 0 and costs one comparison.

w = struct.unpack_from('<I', data, 0)[0]
if w == 0xFFFF:            # Sig1=0, Sig2=0xFFFF -> bigobj
    parse_bigobj(data)
else:
    parse_classic(data)   # check the Machine field for plausibility

Try It Yourself

$ clang --target=x86_64-pc-windows-msvc -c huge.c -o huge.obj
$ llvm-readobj --file-headers huge.obj | grep SymbolCount
$ file huge.obj | grep -o 'COFF object file'

Then build the variant by hand and watch it fail to be recognised:

$ python3 -c "
import struct
d = bytearray(open('huge.obj','rb').read())
mach, nsec, ts, symptr, symcnt = struct.unpack_from('<HHIII', d, 0)
hdr  = struct.pack('<HHHH', 0x0000, 0xFFFF, 2, mach)   # Sig1 Sig2 Version Machine
hdr += struct.pack('<I', ts)                            # TimeDateStamp
hdr += bytes(16)                                         # ClassID
hdr += struct.pack('<IIII', 0, 0, 0, 0)                # SizeOfData Flags Meta*
hdr += struct.pack('<III', nsec, 0, symcnt)            # counts + symbol table
out = bytearray(hdr) + d[0x14:]                          # shift everything up
struct.pack_into('<I', out, 48, symptr + len(hdr) - 20) # fix the pointer
open('huge_bigobj.obj','wb').write(out)
"
$ llvm-readobj --file-headers huge_bigobj.obj
$ file huge_bigobj.obj
$ strings $(command -v llvm-readobj) | grep -c bigobj

Three things to confirm, in order:

clang stayed classic at 70,017 symbols. Raise the count to 200,000 if you like; the header is still 20 bytes. The switch is not automatic in this toolchain.
The constructed file is reported as a short-import library, with Symbol: __imp_ — a symbol that has nothing to do with anything in the file. That is the signature collision, visible.
Zero occurrences of bigobj in the reader. This is the honest boundary of the concept: the layout is documented and hand-verified, but not cross-validated, because no tool here implements it.

And a last check worth doing on a real classic object: verify that the two-word signature does not appear at offset 0 of anything clang emits normally. If it does, you have a short-import file and not an object, and your file-header decoder is about to misparse it exactly as described above.

Check Your Understanding

Without looking back: a bigobj file is handed to a parser that assumes the classic 20-byte header. It reads Machine as 0 and NumberOfSections as 65535, and no error is raised. Why is that more dangerous than a corrupt classic object, and what single check would have caught it?

Apply It

Your build system starts failing on one machine only. The error is “unrecognised object file” from a third-party tool, and only for that machine’s outputs. Every other machine builds the same source with the same compiler version. What are the plausible causes, and which one does the fact that the failure is machine-specific eliminate?

The general habit: when a failure is specific to one machine, suspect a difference in tools rather than in inputs. And when a tool says “I do not recognise this format”, check the magic number first — several formats share one, and the version that follows is the only thing that tells them apart.

Connect

The signature collision is the same lesson the ELF identification concept teaches from the other side. ELF opens with four unambiguous magic bytes and then a class byte and a data-encoding byte, so a reader knows the version before it reads anything else. COFF objects have no magic at all, and the one format that does have one shares it. Both are consequences of the same history: COFF grew out of a linker input format and was never given the identification discipline that ELF got deliberately.

The widened NumberOfSections is the same motivation as the DWARF 64-bit unit length: a 32-bit field eventually is not enough, and the least disruptive fix is a second format rather than a new version of the first. DWARF signals it with a 0xFFFFFFFF escape in the length field; bigobj signals it by taking a signature that means something else entirely.

Module 3 continues with the container that actually holds these objects — and which has a magic number that does work.

Next: The .lib Archive.