Back to course

The .lib Archive

19 minintermediateModule 3: Containers and VariantsContinues from coff-bigobj
19 min · Module 3: Containers and Variants · Structures

Why This Matters

Static libraries are the one part of a Windows build that is not COFF. A .lib you pass to the linker is a Unix ar archive containing COFF objects, byte for byte the objects Module 1 decoded. Nothing inside it is Windows-specific except the choice of extension.

That is worth understanding for two reasons. First, the container has a magic number that works, which makes a nice contrast with bigobj and with the object format itself. Second, the archive carries a symbol index that exists so a linker can answer “which of these hundred members defines printf?” without opening all hundred — and that index turns out to be the tidiest offset-based lookup structure in the whole toolchain.

A Simple Model

Three pieces, in this order:

  1. An 8-byte magic:!<arch>\n — ASCII, plus a newline.
  2. One or more index members, named /. There is no requirement that there be only one, and in practice there are two with different contents.
  3. The object members, each preceded by a fixed 60-byte ASCII header.

The 60-byte header is the part worth memorising, because every field in it is text and the field boundaries are defined by their widths, not by delimiters:

offset width field 0 16 name (ASCII, space-padded; "/" or "//" for the indexes) 16 12 mtime (decimal, usually 0) 28 6 uid (decimal, usually 0) 34 6 gid (decimal, usually 0) 40 8 mode (octal, e.g. "644 ") 48 10 size (decimal BYTES of the member body) 58 2 terminator the two bytes 0x60 0x0A, i.e. "`" and newline

Two consequences follow immediately. The size field is decimal, not hexadecimal, so a member of 823 bytes is stored as 823 and one of 16 bytes as 16 — a reader that assumes hex will be wrong by a factor of sixteen on most members. And the terminator is a fixed two-byte sequence, so a parser can detect a truncated or misaligned archive immediately rather than discovering it as garbage further in.

Members are padded to an even length. After a member body, if the size was odd, one pad byte follows before the next header. The 60-byte header keeps the next header even-aligned, so the padding is always exactly one byte at most — but a reader that computes offsets as 60 + size without the parity fix will land one byte early and read a corrupt header on every odd-sized member. In the sample below, the 40-byte and 46-byte index members are even, and the objects are 823 and 802 bytes, so the parity rule happens not to bite — which is exactly why it is worth stating.

The Actual Detail

Built here from two real objects, both produced by the same clang invocation as Module 1:

$ cat lib1.c
int lib_add(int a, int b) { return a + b; }
int lib_val = 11;
$ cat lib2.c
int lib_mul(int a, int b) { return a * b; }

$ clang --target=x86_64-pc-windows-msvc -c lib1.c -o lib1.obj
$ clang --target=x86_64-pc-windows-msvc -c lib2.c -o lib2.obj
$ llvm-ar rcs demo.lib lib1.obj lib2.obj
$ file demo.lib
demo.lib: current ar archive

file calls it a “current ar archive”, which is a Unix description of a file that Windows tools call a static library. Here is the first sixteen bytes:

0000: 213c 6172 6368 3e0a 2f20 2020 2020 2020  !<arch>./
0010: 2020 2020 2020 2020 3020 2020 2020 2020          0
      |— magic: "!<arch>" + \n —| |— name "/" —| |— mtime "0" —

And the whole 1960-byte file, member by member, decoded by a parser written from the archive format rather than from llvm-ar:

MemberHeader atBody atSizeName fieldWhat it is
00x80x4440/symbol index, plain form
10x6c0xa846/symbol index, extended form
20xd60x112823lib1.obj/a COFF object
30x44a0x486802lib2.obj/a COFF object

The object members are the files from Module 1, unchanged. The decoder reads lib1.obj’s member and finds machine 0x8664, 9 sections, a symbol table and 11 named symbols — the same values the standalone object has, which is the check that the archive did not rewrite anything.

The two indexes, which are not the same structure

Both are named /. The first four bytes of the body tell you which one you have, and they are not the same size, layout, or byte order.

Member 0, the plain form. 40 bytes: a big-endian count, then one big-endian value per symbol.

00 00 00 03 count = 3 00 00 00 d6 symbol 0 -> file offset 0xd6 00 00 00 d6 symbol 1 -> file offset 0xd6 00 00 04 4a symbol 2 -> file offset 0x44a 6c 69 62 5f 61 64 64 00 "lib_add\0" 6c 69 62 5f 76 61 6c 00 "lib_val\0" 6c 69 62 5f 6d 75 6c 00 "lib_mul\0"

Two things to notice. The values are offsets of member headers in the file, not offsets into this index: 0xd6 is where lib1.obj’s header starts, and 0x44a is lib2.obj’s. So lib_add and lib_val both point at the same member, which is correct — they are both defined there. And the three names that follow the offset array are stored in link order, not sorted: lib_add, lib_val, lib_mul.

Member 1, the extended form. 46 bytes, starting with the marker 02 00 00 00:

02 00 00 00 marker = 2 (extended form) 00 00 00 d6 member header offset 00 00 04 4a member header offset 03 00 00 00 count = 3 <-- LITTLE-endian, unlike the index above 01 00 02 00 01 00 two bytes of binding flags per symbol 6c 69 62 5f 61 64 64 00 "lib_add\0" 6c 69 62 5f 6d 75 6c 00 "lib_mul\0" 6c 69 62 5f 76 61 6c 00 "lib_val\0"

The same three symbols, sorted, in a different byte order.lib_add, lib_mul, lib_val is alphabetical; the plain index had them in link order. And the count is little-endian where the plain index's count was big-endian — a difference that is easy to miss because on this machine both encodings of the member offsets happen to agree, since each value's low byte comes first and the rest are zeros.

The practical consequence: a reader must not assume either list is sorted, and must read each index with the byte order its own marker implies. Binary-searching the plain index would work here by luck and fail the moment a symbol is added out of order.

Names longer than sixteen characters

The name field is sixteen bytes, so a_very_long_member_name_here.obj does not fit. The escape is a third special member name:

  member 2  header at 0xd6  name '//'   size 34
      LONG NAME TABLE: b'a_very_long_member_name_here.obj\x00\n'
  member 3  header at 0x134 name '/0'   size 823
      -> a COFF object: machine=0x8664, 11 named symbols

The // member holds the long names, NUL-terminated, each followed by /. The object’s name field is then /0: a slash and the decimal offset into that table. This is the same idea as a COFF section name stored as /17 in the section header concept, and as the mangled symbol name stored as a string-table offset in the string table concept. Three structures in one format, one idea: when eight or sixteen bytes is not enough, store a number.

A Real Example

The question the index exists to answer: the linker needs lib_add. Which member is it in?

  1. Check the magic. Bytes 0 to 7 are !<arch>\n. Not an archive → stop. Two bytes would do, and unlike a COFF object this format is identifiable.
  2. Find the index. The first member’s name field is / and its body does not begin with the 0x02 marker, so it is the plain form. Read the count big-endian: 3.
  3. Walk the three values.0xd6, 0xd6, 0x44a. These are file offsets of member headers.
  4. Match the name. The names follow the offset array, in the same order, NUL-terminated. Scanning them, lib_add is first, so its defining member is the one whose header is at 0xd6.
  5. Open that member. Its header is 60 bytes, so its body starts at 0x112 and runs 823 bytes. Seek past the header, and you are looking at lib1.obj — the same COFF object you would have had on disk, with its own symbol table at its own offsets.
  6. Read the symbol from the object, not from the index. The index told you which file; the file’s symbol table is the authority on the offset. This is the same index-versus-source-of-truth split as .debug_pubnames, and for the same reason: the index is derived and can be wrong without the object being wrong.

Six steps, and step 5 is the only one that touches real object data. That is the entire value of the index: without it, answering “where is lib_add” means opening both members and reading both symbol tables.

Why an archive holds a whole object rather than a piece of one. A static library is not a linker input in the way an object is; it is a search path. The linker pulls in a member only if it is needed to resolve a currently-undefined symbol, and pulling one in can create new undefined symbols, which pulls in more. That is why the index must map a symbol to a whole member: the unit of extraction is the object file, and the index exists to find the right one. It is also why an archive of two objects, one of which defines nothing anybody wants, costs a header and an index entry and nothing else.

Try It Yourself

$ cd courses/coff/assets/samples
$ python3 ar_decode.py demo.lib
$ llvm-ar t demo.lib
$ llvm-nm demo.lib

Four experiments, each of which teaches something the tools do not tell you:

And the test of whether you understand the whole thing: extract lib1.obj from demo.lib by hand — skip 8 bytes of magic, skip two 60-byte headers plus their index bodies, land on 0x112, take 823 bytes — and confirm it is byte-identical to the lib1.obj on disk. If it is, you have understood that the archive is a container and not a re-encoding.

Check Your Understanding

Without looking back: a member header reports size = 16. How many bytes of member body follow, and what does the next member header start at relative to this one?

Apply It

You are writing a tool that lists the members of a .lib. It works on every archive until it hits one built by a different toolchain, and then it reports members with empty names and nonsensical sizes — but the magic checked out and the file is a genuine archive. What are the two most likely causes, and what distinguishes them?

The general habit: when a format uses a small number of reserved names to mean special things, dispatch on those names before you dispatch on anything else. A tool that assumes “the first record is the interesting one” will work on most inputs and fail on exactly the ones that were produced by a different implementation.

Connect

The archive’s symbol index is the same structure as the ELF dynamic symbol table in purpose and the opposite in mechanism: both answer “which file provides this symbol”, and both exist so the answer can be found without opening everything. The ELF course covers the .gnu.hash acceleration on top of that table, which is the next step past what an archive does.

The offset-for-name trick — /17 in a section header, a zero dword plus an offset in a symbol name, /0 into a long-name table — is the format’s single most repeated idea, and the DWARF file tables concept shows it in a fourth format. Learning to recognise “this small field is an index, not a value” is probably the single most transferable thing in either course.

One field remains, and it is the module’s smallest concept: a section header slot that is always zero.

Next: Line Numbers.