Debug information is bigger than the program. On a modest C++ project it routinely exceeds the executable several times over, and almost none of it is used during a normal run: you need one function’s DIEs to step through it, and you need the rest only when a breakpoint lands somewhere you did not predict.
Split DWARF is the answer, and it is not a compression scheme — nothing is packed. It is a file split. The information a debugger needs before it can even start, plus the addresses, stays in the object file where the program can be shipped. The bulk goes into a separate file that the developer keeps and the user never sees.
Two files, two jobs.
| File | Contains | Shipped? |
|---|---|---|
the .o / executable | one skeleton unit per compilation, a dwo_id, the .debug_addr table, the line program | yes — the skeleton is tiny |
the .dwo | the real DIE tree, its abbreviations, its strings, the string-offset table | no — kept next to the build |
The skeleton is not a summary. It is a stub whose entire content is "the real thing is over there", plus the few things that must be known before you can go and get it.
dwo_id is a hash the compiler invents. It exists so a debugger holding an executable can be handed the .dwo for a specific compilation unit without trusting a filename — and so it can verify it has the right file. Notice what the skeleton does not contain: not one DW_TAG from the real tree. If you dump the skeleton expecting to find function names there, you will conclude the build produced no debug information at all.Two source files, compiled gcc -gdwarf-5 -O0 -gsplit-dwarf. The result is one executable and one .dwo holding both compilations.
Here is the skeleton unit, decoded:
--- unit at 0x0: unit_length=0x31 version=5
unit_type=0x04 (DW_UT_skeleton) dwo_id=0x12e9c545cc3ee5a5
address_size=8 abbrev_offset=0x0
header ends at 0x14, so the first DIE is at <0><14>
1 abbreviation: 1 = DW_TAG_skeleton_unit
<0><14> DW_TAG_skeleton_unit
<15> DW_AT_low_pc 0x1149
<1d> DW_AT_high_pc 0x80
<25> DW_AT_stmt_list 0x0
<29> DW_AT_dwo_name types_split-types.dwo
<2d> DW_AT_comp_dir /tmp/.../dwarf3
<31> DW_AT_GNU_pubnames 1 (flag_present: no bytes)
<31> DW_AT_str_offsets_base 0x8
And the second unit, at 0x35, with a different dwo_id and a different abbreviation table:
--- unit at 0x35: unit_length=0x31 version=5
unit_type=0x04 (DW_UT_skeleton) dwo_id=0x57891407d925f05c
abbrev_offset=0x15
first DIE at <0><49>
Points worth noticing, all confirmed against readelf and llvm-dwarfdump:
address_size a skeleton carries 8 bytes of dwo_id, where a DW_UT_type unit would carry a 16-byte type signature and offset. A decoder that reads the header by unit type gets this right; one that assumes a fixed 12-byte DWARF 5 header reads the first DIE 8 bytes early..debug_info, and the second points at abbreviation offset 0x15 rather than 0. Exactly the situation that makes the DIE tree concept’s multi-table trap dangerous — and here it is load-bearing, because the abbreviations in the executable are for skeletons and nothing else.DW_TAG_skeleton_unit DIE per unit, two in total. All 39 DIEs of the real tree are in the .dwo.Five sections, and every one has a .dwo suffix so they can coexist with the skeleton's own:
The .dwo’s unit is a different unit type again — DW_UT_split_compile (5) rather than DW_UT_skeleton (4) — and it repeats the same dwo_id:
--- .debug_info.dwo: unit_length=0x181 version=5
unit_type=0x05 (DW_UT_split_compile) dwo_id=0x12e9c545cc3ee5a5
first DIE at <0><14>: DW_TAG_compile_unitThat repetition is the pairing. A debugger that has the executable walks to a skeleton, reads the dwo_id, opens each candidate .dwo, and looks for a unit with a matching id.
Inside the .dwo, names are not string offsets. They are indices:
<0><14> DW_TAG_compile_unit
DW_AT_producer : GNU C23 15.2.0 ... -gsplit-dwarf ...
via str_offsets[16]
DW_AT_name : types.c
via str_offsets[26]
DW_AT_comp_dir : /tmp/.../dwarf3
via str_offsets[29]str_offsets[16] is not an offset into .debug_str.dwo — it is an index into .debug_str_offsets.dwo, and that table is itself addressed from a base. The base is DW_AT_str_offsets_base, which is 0x8, and it is in the skeleton, not in the .dwo.
Ignore it, use base 0, and every string shifts by two entries. This is what that looks like:
It is the worst class of bug: no error, a well-formed string, and the wrong one. A reader that gets this wrong will happily print unsigned int as the compiler that built the program.
The same problem exists for addresses. .debug_addr in the skeleton file is a table that DW_FORM_addrx indexes, and its base is DW_AT_addr_base, also on the skeleton. The reference build's .debug_addr is 0x38 bytes — 56 bytes, which is 7 addresses on a 64-bit target.
What a debugger does with a split binary, step by step, using the offsets above:
.debug_info, hit a DW_UT_skeleton unit, and note three things: dwo_id = 0x12e9c545cc3ee5a5, DW_AT_dwo_name = types_split-types.dwo, and DW_AT_str_offsets_base = 8.DW_AT_comp_dir plus DW_AT_dwo_name gives an absolute path. This is the one place a skeleton still refers to things by name, which is why the file is found by name and then verified by id..dwo and find the matching unit. Scan its units for dwo_id == 0x12e9c545cc3ee5a5. It is there, as DW_UT_split_compile. If the id does not match, the .dwo is stale — rebuilt from different source — and its contents cannot be trusted even though the path resolved..debug_abbrev.dwo for shapes, and resolve every strx through .debug_str_offsets.dwostarting at 8..dwo. DW_AT_stmt_list = 0 points into the skeleton’s own .debug_line. The .dwo also has a .debug_line.dwo, and the line program is the one thing that stays in both places.low_pc/high_pc and .debug_aranges) and where is the rest (from dwo_name and dwo_id). It also keeps the line program, because stepping one line must not require opening a second file. Everything expensive — the 39-DIE tree, its types, its strings — is in the .dwo and is only ever read for the unit you are actually stopped in.$ gcc -gdwarf-5 -O0 -gsplit-dwarf -o shape types.c main.c
$ ls -la shape types_split.dwo
$ readelf --debug-dump=info shape # skeletons only
$ readelf --debug-dump=info types_split.dwo # the real tree
$ readelf -S -W shape | grep debugFive things to try, in order of how much they teach:
readelf --debug-dump=info on the executable prints 56 DIEs even though the file contains only 2, because readelf follows DW_AT_dwo_name and merges the .dwo in. On the .dwo alone it prints 39. Three numbers, three files, and it is worth being able to explain each.readelf --debug-dump=info types_split.dwo resolves names without complaining, because readelf read the base from the skeleton. Read the .dwo with no skeleton available and a naive index of 0 gives you unsigned int as the producer..dwo (or recompile the other file), and compare the ids. A stale .dwo has a different id from the executable that points at it, and that mismatch — not the filename — is what tells you the debug info does not match the binary.ls -la on both. The skeleton is a few hundred bytes; the .dwo holds the tree. Then build without -gsplit-dwarf and see the difference in the executable, which is where the bulk would otherwise live..debug_addr, .debug_line and the aranges and pubnames indexes all remain in the executable. Split DWARF moves the tree, not the indexes — which is exactly what the next concept is about.And the check for your own reader: after resolving a strx, confirm the string looks like a string for that attribute. A DW_AT_producer that comes back as a C type name is the signature of a missing str_offsets_base, and it is worth asserting on, because nothing else will tell you.
Without looking back: you are decoding .debug_info.dwo on its own, with no skeleton file to hand, and DW_AT_producer comes back as unsigned int instead of the compiler version. What is wrong, and where does the value you need actually live?
A build system starts shipping -gsplit-dwarf builds. Stepping works; hovering for types does not — the debugger reports an unknown type for every variable, and the log says it could not find the .dwo for one compilation unit. Everything else is fine. What is the most likely cause, and what is the check that distinguishes it from a genuinely stale .dwo?
The general habit: when a system is split into a stub and a payload, the stub has to record enough to find the payload, and that means it records a location. Location is the part that breaks — not the content, not the identifier, and not the format.
Split DWARF is a file-level version of a pattern the COFF course already covers: COMDAT keeps one authoritative copy of something that would otherwise be emitted many times, and the linker discards the duplicates. Here the duplication is at whole-file granularity and the identifier is a hash instead of a name — because the two compilations must agree on it without ever meeting.
The COFF archive container is the other half of this idea from the other direction: an archive is a file of objects with a symbol index, so a linker can pull in only the members it needs. Split DWARF splits one object into two files along a different seam — what a program needs to run versus what a developer needs to debug it — and the seam is invisible in the file format.
What the skeleton deliberately keeps is the subject of the last concept: the indexes that let a debugger answer “which unit covers this address” and “what is the name of that DIE” without reading the whole tree.