Split debugging solved a storage problem: the debug information was a large fraction of the binary, and shipping it to every machine was expensive. The fix was to move it out into .dwo files and leave a skeleton behind. But it created a lookup problem, and the problem is worse than it first appears.
Consider what a debugger actually has when it opens a program. It has one executable, whose skeleton units each name a .dwo file. It has a directory full of .dwo files, possibly hundreds, from possibly dozens of libraries. And the user just pressed a breakpoint in a function whose name they can see in a backtrace.
To answer "which unit describes this function", the debugger must find the right .dwo file and then the right unit inside it. Scanning every .dwo file, opening each, and walking every unit to compare identifiers works, and it is exactly as slow as it sounds — on a large C++ program it is seconds of latency before the first breakpoint resolves. The fix is an index, and the index is a file format in its own right.
Two identifiers do the work, and it is worth being precise about each:
| Identifier | Size | What it is |
|---|---|---|
The DWO_id | 64 bits | A number in the unit's own header that identifies the unit. Unique per unit, and assigned by the producer |
The signature | implementation-chosen | A hash of the .dwo file's contents, so a reader can notice a stale or mismatched file |
These are different things solving different problems, and conflating them is the usual first mistake. The DWO_id answers which unit is this one. The signature answers is this the file I think it is. A debugger doing a fast lookup needs the first; a build system checking that a .dwo was not rebuilt under it needs the second.
Now the package. A .dwp file is not a new format — it is an ordinary ELF object file holding the .dwo sections of many units, concatenated, plus one section that did not exist before:
$ llvm-dwp -e split_exe -o split.dwp $ readelf -S -W split.dwp | grep -oE '\.debug[a-zA-Z_.0-9]*' | sort -u .debug_abbrev.dwo .debug_cu_index .debug_info.dwo .debug_line.dwo .debug_str.dwo .debug_str_offsets.dwo
Every one of those except the last is a section you have already decoded, and the names keep the .dwocontribution suffix — the marker that tells a linker these are the parts of a split unit, not ordinary debug sections. The content is the same DW_UT_split_compile units that live in the individual .dwo files. readelf and llvm-dwarfdump both open the package and both read its units:
$ readelf --debug-dump=info split.dwp
Compilation Unit @ offset 0:
Length: 0xb3 (32-bit)
Version: 5
Unit Type: DW_UT_split_compile (5)So a package is a concatenation with an index. The interesting file is the index.
DWO_id is a 64-bit number a producer chose; there is nothing in the format tying it to a unit's position in a file. The same DWO_id could legitimately appear in two different builds with different units in different orders. So "scan until the DWO_id matches" is correct, and it is the slow path the index exists to avoid.Build a split-DWARF executable, package it, and look at the index:
$ gcc -g -gsplit-dwarf types.c -c -o types.o $ gcc -g -gsplit-dwarf main.c -c -o main.o $ gcc -g types.o main.o -o split_exe $ ls *.dwo main.dwo types.dwo $ llvm-dwp -e split_exe -o split.dwp
The tool is told which file to work from, not which .dwo files to read. It gets the list by walking the skeleton units and reading their DW_AT_dwo_name attributes:
$ readelf --debug-dump=info split_exe | grep dwo_name
DW_AT_dwo_name : (indirect string, offset: 0): types.dwo
DW_AT_dwo_name : (indirect string, offset: 0x1c): main.dwoThat is the whole discovery mechanism, and it explains why the tool errors out when given the wrong input:
$ llvm-dwp -e types.o -o x.dwp
warning: executable file does not contain any references to dwo filesIt is not scanning for a file named *.dwo on disk. It reads the executable, follows the references, and packages what it finds. Now the index itself, 144 bytes:
0000: 05 00 00 00 04 00 00 00 02 00 00 00 04 00 00 00 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0020: 3a c9 20 1a 41 2d e4 5c 0f b3 b9 69 1f 53 96 c2 0030: 00 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00 0040: 01 00 00 00 03 00 00 00 04 00 00 00 06 00 00 00 0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0060: b7 00 00 00 b4 00 00 00 4c 00 00 00 34 00 00 00 0070: b7 00 00 00 b4 00 00 00 4c 00 00 00 34 00 00 00 0080: 95 00 00 00 b3 00 00 00 4a 00 00 00 24 00 00 00
All four header fields are four bytes wide, and the values are the useful part:
| Bytes | Field | Value | Means |
|---|---|---|---|
05 00 00 00 | version | 5 | The index has its own version field, independent of the units it indexes |
04 00 00 00 | section_count | 4 | How many sections are indexed — the five unit sections minus .debug_cu_index itself |
02 00 00 00 | unit_count | 2 | Two units packaged: types.dwo and main.dwo |
04 00 00 00 | slot_count | 4 | The size of a hash table that would let a lookup find a unit by hashing a signature |
Two of those deserve a moment. section_count is 4 and the package has 5 unit-bearing sections, because the index does not index itself. And slot_count is 4 while unit_count is 2: the hash table is a fixed-size structure chosen at package time, independent of how many units went in, and it is a trade of table size for lookup speed. Four slots for two units is nearly half empty. The producer could have made a hundred-unit package with the same four slots, and then most lookups would fail and fall back to a scan.
DWO_id values are in the index, in fullBytes 0x20 to 0x2f hold two 8-byte little-endian numbers. Compare them with what the units themselves declare:
index 0020: 3a c9 20 1a 41 2d e4 5c = 0x5ce42d411a20c93a index 0028: 0f b3 b9 69 1f 53 96 c2 = 0xc296531f69b9b30f $ llvm-dwarfdump --debug-info split.dwp | grep DWO_id unit 0: DWO_id = 0x5ce42d411a20c93a unit 1: DWO_id = 0xc296531f69b9b30f
They are the same numbers, byte for byte. That is the index doing its one job, and it is the fact worth remembering: a debugger holding a DWO_id — obtained from the skeleton unit in the executable — can look it up in the index and jump straight to the right unit, without opening any other .dwo file and without walking a single unit header.
Contrast that with the naive path. To find the same unit, a reader would have to read both .dwo files from disk, read each unit's header far enough to find its DWO_id field, and compare. The index reduces that to arithmetic on a table that is already in memory. On a large program with hundreds of units, the difference between a scan and a lookup is the difference between a debugger that feels instant and one that does not.
DWO_id values, the section list and the fact that both reference readers open the package are all established above. The trailing offset table is not. The header plus the hash table plus the signatures plus four sections of two offsets accounts for 80 of the 144 bytes, and the remaining 64 could not be attributed to a single reading of the layout — a sixteen-byte block is visibly duplicated at offsets 0x60 and 0x70, which is not a shape any single-table reading explains. 0xb7 does appear in that region and does match the offset of the second unit, which is suggestive, but suggestive is not the same as accounted for.Three things it is easy to assume and worth ruling out explicitly, because the file gives no hint either way:
DWO_id claims. A DWO_id collision, or a package built from a different set of files than the executable names, produces a lookup that succeeds and returns the wrong unit. This is a real hazard and it is why the separate signature mechanism exists.Answering "give me the DW_TAG_subprogram named total" against a split, packaged build. Every step is one of the joins from the concepts before it:
Step 5 is the one that produces the most confusing bugs, so it is worth being explicit about the division of labour. In a split build, the skeleton unit and the split unit are not the same unit. They are two units that share a DWO_id, and they carry different attributes. The skeleton in the executable holds the attributes a linker needs: DW_AT_low_pc, DW_AT_ranges, DW_AT_stmt_list — the ones that say where. The split unit in the .dwo or the .dwp holds the ones a type system needs: DW_AT_name, DW_AT_type, DW_AT_byte_size — the ones that say what.
So the lookup you sketched in step 4 is not a simplification. A debugger genuinely does read two different units and join them on the DWO_id, and the join is the reason the identifier exists. A reader that reads the whole unit out of the package and the whole unit out of the executable and concatenates them gets duplicates; a reader that reads only the package has no addresses; a reader that reads only the executable has no types. Each of those three mistakes produces a debugger that works until the moment it needs the half it is missing.
And the cost of not packaging at all is worth stating as a number rather than an adjective. Without a .dwp, step 3 is a scan: open every .dwo, read every unit header, compare DWO_id. With one, step 3 is a table lookup. The number of files read goes from all of them to one, and the amount of unit data parsed goes from all of it to one header. Everything after step 3 is identical work either way, which is why packaging is worth a file format.
$ llvm-dwp -e split_exe -o split.dwp
$ readelf -S -W split.dwp
$ llvm-dwarfdump --debug-info split.dwp | head -8
$ readelf --debug-dump=info split.dwp | head -8DWO_id match from both ends. Read the DWO_id out of the packaged unit with llvm-dwarfdump, read the same field out of the skeleton in the executable, and read the bytes at 0x20 in the index by hand. Three sources, one value. If any of them disagree, stop and find out why before trusting a lookup.slot_count trade. Package the same project with a larger index unit size and compare the resulting slot_count and section size. Then reason about what happens to lookups that miss — and about which is better for a debugger holding one unit in memory..dwo files so that it gets a new DWO_id, and rebuild the executable too but leave the package stale. Now ask what a debugger does. This is the failure mode the signature mechanism exists to catch, and seeing the gap is more instructive than reading about it..debug_cu_index is the one thing in this concept that is not accounted for. You have the section, the header, and two known-good offsets from the tool. Work out the geometry, and if you cannot, notice what evidence would settle it — that is the same discipline that produced the loclists adjudication two concepts ago, applied to your own file.Without looking back: a debugger has a skeleton unit from the executable. That unit carries a DW_AT_dwo_name and a DWO_id. A .dwp package exists and is loaded. What does the debugger do with the DWO_id, and what would it have to do differently if the package did not exist?
A debugger on a large C++ project shows correct types, correct line numbers, and correct variable values. Symbol lookup, however, takes eight to ten seconds after a restart, and only on a cold cache — the second time it runs it is instant. The project is built with -gsplit-dwarf and the .dwp is present. Given what you have just decoded, what is the most likely explanation, and what is the cheapest experiment that would confirm it without changing the build?
The general habit: separate "the answer is wrong" from "the answer takes too long". Correctness bugs live in the decode path and performance bugs live in the search path, and they rarely share a cause. When values are right and latency is wrong, stop reading the data and go look at how it is being found.
This concept is the payoff for split debugging, which introduced the DWO_id and the skeleton idea without being able to explain what the identifier was for. It is also the payoff for putting a lookup together: that walk assumed one unit in one file, and this is the case where the unit and the file are separate and the identifier is the join.
The shape of the solution is one this course has now seen three times, and recognising it is the transferable part. A table of identifiers mapping to positions in a file, so a reader can go straight to what it needs: the .debug_pubnames and .debug_gnu_pubnames sections you used in Module 3 map a name to a DIE; .debug_aranges maps an address range to a unit; .debug_cu_index maps a DWO_id to a unit in a package. Each replaces a scan of everything with a lookup in a small table, and each is optional in the sense that a debugger works without it — just more slowly.
That closes Module 4, and with it the parts of DWARF that can be checked against files on this machine. The next module goes after the thing DWARF is read alongside but is a different format entirely: the unwind information, read straight from the binary, and what a debugger does when there is no debug info at all.