Back to course

Thread Local Storage

19 minintermediateModule 5: Other Targets and Other SectionsContinues from coff-arm64
19 min · Module 5: Other Targets and Other Sections · Algorithms

Why This Matters

Every variable you have met in this course has had exactly one address. A global has an address, a local has an address computed from the stack pointer, and both are the same for every thread. A thread-local variable does not have that, and the reason is worth stating precisely: its address depends on which thread is asking.

That breaks the assumption the whole relocation machinery rests on. A DW_AT_low_pc or a DW_OP_addr is an address, and an address is a single number. But a thread-local's address is an expression involving a base that the thread itself supplies — on Windows, a value called _tls_index that the loader assigns per module and the runtime turns into a per-thread slot on first use.

So the section is interesting for a reason beyond TLS: it is the one place in a COFF object where a variable's address is not a constant, and the format has to encode that as a section rather than as a relocation. A section whose name means something is unusual, and this one is named .tls$ — with a dollar sign, which is a hint rather than a convention.

A Simple Model

Compile a thread-local variable and look at the sections:

Two thread-local variables, and one function that reads both:

DeclarationWhat it is
__declspec(thread) int counter = 5an initialised thread-local scalar
__declspec(thread) int shared_buf[4]an uninitialised thread-local array, so it goes to the bss-like part
$ clang --target=i686-pc-windows-msvc -c tls.c -o tls.obj
$ llvm-readobj --sections tls.obj | grep Name

Four ordinary sections and one that is not:

sec1  .text       char=0x60500020
sec2  .data       char=0xc0300040
sec3  .bss        char=0xc0300080
sec4  .tls$       char=0xc0300040     <-- the one that matters
sec5  .debug$S    char=0x42300040
sec6  /4          char=0x00100800

Read the characteristics of section 4 with the parser shipped with this course:

IMAGE_SCN_CNT_INITIALIZED_DATA | IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_WRITE

That is identical to .data — same value 0xc0300040, same three flags. And that is the point. A thread-local initialised variable is genuinely initialised data, genuinely readable, genuinely writable, and the format says so with the ordinary flags. Nothing in the characteristics marks it as thread-local. The only thing that says so is the section name.

Which is a genuine fragility, and worth naming as one. Everything else in a COFF section header is either a fact about the data or an instruction to the linker. The name, for most sections, is a label a human chose. Here it is load-bearing: .tls$ is not .data with a different letter, it is a request for per-thread allocation semantics, and a tool that reports a section by its characteristics alone cannot tell the two apart. This is the same lesson as the long section name /4 in section 6, arrived at from the other direction — sometimes the name is the payload.

Why the dollar sign

The $ is not decoration. In the COFF convention, .text$mn and .text$zz are suffixes that let a linker order code fragments within .text — the mn and zz pairs are alphabetical orderings chosen by convention so that multi-threaded and single-threaded startup paths land in a known order. .tls$ uses the same idea: the dollar sign marks a section whose position within its group is meaningful, and bare .tls$ sorts before any .tls$zz-style variant.

So the section name here is doing two jobs at once: it selects TLS semantics, and its suffix participates in an ordering. Neither job is visible in the section header's numeric fields, and both are visible in eight bytes of name.

The Actual Detail

What is in the section

The two variables in the source, and the raw contents of section 4:

raw bytes:  05 00 00 00  00 00 00 00  00 00 00 00
             |     |     |      |      |      |      |      |
             |     |     |      |      |      |      |      +-- shared_buf[0..3] = 0
             |     |     |      |      |      |      +--------- shared_buf[1..3] = 0
             |     |     |      |      |      +---------------- counter's address is
             |     |     |      |      |                       irrelevant here: it is
             |     |     |      |      |                       a *template*
             |     |     |      |      +-------------------------- stored at offset 4
             |     |     |      +--------------------------------- 4 bytes of data
             |     |     +---------------------------------------- initial value, 5
             |     +---------------------------------------------- 4 bytes of padding
             +---------------------------------------------------- 4 bytes for the
                                                                   initialised value

Read that carefully, because it is the concept. The bytes are the template — the initial values, laid out exactly as they would be in .data. And yet the addresses in this section are not the addresses of the variables. Every thread gets its own copy, allocated at a different address, and this section is the template those copies are made from.

That is the whole design in one sentence: the section holds initial values; it does not hold the variables. The variables live in per-thread storage the loader creates, and the template is consumed at thread-creation time.

How the code refers to it

Disassemble the one function:

$ llvm-objdump -d --section=.text tls.obj
  0:  a1 04 24 24 24        mov   0x2424,%eax
  5:  03 05 24 28 24 24     add   0x2428,%eax
  b:  c3                   ret

Two absolute-looking addresses, 0x2424 and 0x2428 — four bytes apart, which is exactly the gap between counter and shared_buf in the template. In a relocatable object those are IMAGE_REL_I386_DIR32 fixups with zeros, and after an ordinary link they would become real addresses into .data.

For TLS they must not. They have to become something like the address of this thread's copy, which is base + _tls_index-derived offset. So the fixups in a TLS object are not plain DIR32 against .tls$ — and that is where the module runs out of road on this machine, which is the honest place to stop.

What happens when you try to link it

Attempt the link with the linker this course has been using throughout:

$ ld --oformat pei-i386 -m i386pe tls.obj -o tls.exe --entry read_it
tls.obj:tls.c:(.text+0x5): undefined reference to `_tls_index'
tls.obj:tls.c:(.text+0xb): undefined reference to `_tls_array'

Two undefined symbols, and they are the two things a Windows TLS program needs from the runtime. _tls_index is the per-module index the loader assigns; _tls_array is the per-thread pointer array the runtime indexes with it. A program that uses thread-locals computes its variable's address as something involving these two, and nothing in the object file can supply them.

What is established and what is not, precisely.

Established from the object file: a .tls$ section exists; its characteristics are byte-identical to .data and say nothing about TLS; its contents are an initial-value template, not addresses; and the section name is the only thing marking it.

Established from the link attempt: the linker reads the section and plans to place it, and then fails on _tls_index and _tls_array. The map file it produced before failing contains a row for .tls — note the name change, the output section is .tls where the input was .tls$ — which is direct evidence that the linker renames the section on the way out.

Not established, and therefore not taught: the exact IMAGE_REL_I386 type used for a TLS address on this target, the bytes of the linked .tls section, and what the loader writes into _tls_array. Those need a linker that implements PE TLS, and this one does not. A reader who wants them needs lld-link.

So this concept ends where Module 4's archive discussion ended, and for the same reason: the container and the flags are fully verifiable here; the runtime behaviour is not. That is still worth a concept, because the container is the part a tool author has to get right and the part a reader will meet in every Windows binary that uses a library with thread-locals — which is most of them.

A Real Example

What a tool has to do with a .tls$ section, and where each of those jobs is done. The value of walking this is that it separates the things an offline tool can get right from the things only a running process can.

OFFLINE, from the object file: .tls$ contents are the initial-value template the layout is the layout: offset of each symbol within the section is its offset within every thread's copy so a symbol's offset is knowable, and the section size is the per-thread footprint of all of them together AT LINK TIME, by the linker: collect every .tls$ input section into one .tls output section, applying each object's own alignment assign each contributing module a _tls_index -- this needs PE TLS support, which this linker lacks AT LOAD TIME, by the loader and the runtime: allocate the per-thread blocks publish _tls_array[ _tls_index ] the first access in a thread initialises that thread's block from the template AT RUN TIME, in the generated code: the variable's address is _tls_array[ _tls_index ] + the symbol's offset which is why the fixup cannot be a plain DIR32

Four stages, and they are worth keeping apart because tools operate at different ones. The crucial observation for anyone writing a tool is in the first block: the offset of a thread-local within its section is a static fact, knowable offline, and it is the same offset in every thread's copy. That is the property that makes TLS tractable — the block is copied, so the internal layout is invariant, and only the base moves.

It also explains why the .tls$ section is named the way it is, and why the dollar sign matters even though the section is a single group here. A toolchain that needs several TLS groups — one per initialisation priority, which is how C++ thread-local objects with destructors are ordered — uses names like .tls$zz and .tls$mn to give the linker an explicit order. The name carries the ordering because there is nowhere else to put it: the characteristics say "initialised data", the data says "these are the values", and neither can express "initialise this group before that one".

And the reason a plain DIR32 cannot work is the last block. The generated code needs the address of the current thread's copy, which is a runtime value reached through two symbols neither of which exists until the program loads. So the relocation the compiler emits is not a reference to the .tls$ section at all — it is a reference to _tls_index or _tls_array, with the symbol's offset added. The section never appears in the fixup; it appears only in the loader's copy step. Which means a COFF object with thread-locals carries two references to each variable, in two different worlds, and a tool that resolves only the one it can see offline will report a section-relative offset where the program means a per-thread one.

Try It Yourself

$ llvm-readobj --sections --relocations tls.obj
$ llvm-objdump -d --section=.text tls.obj
$ python3 courses/coff/assets/samples/coff_parse.py tls.obj

Check Your Understanding

Without looking back: an object's section 4 is named .tls$ and has characteristics 0xc0300040. Section 2 is named .data and has characteristics 0xc0300040. A tool lists a symbol's address as image_base + section.VirtualAddress + symbol.Value, which is correct for every other symbol in the object. What does it report for a thread-local, and where does the information it needs actually live?

Apply It

You write a static analysis tool that reports every global's address in a Windows binary, by reading symbols and section headers. It works on most binaries. On one — a large C++ program that links a threading library — it reports a handful of symbols at addresses that fall inside a section called .tls$, and those symbols are the ones you would expect to be per-thread caches and counters. The tool reports no error. What is it doing, and what is the smallest change that makes its output correct rather than merely non-crashing?

The general habit: when two sections in a format are allowed to be identical in every numeric field, the name is the only discriminator — so treat names as data, and check the name before trusting any derived value.

Connect

Two threads of this course meet here. The first is the section characteristics: those flags are not a description, they are instructions, and this concept found a case where the instructions say something ordinary and the name says something else. The link concept then showed what happens when the instructions are obeyed — and the .tls$ to .tls rename is a third transformation it did not cover, visible only in a map file from a link that failed for an unrelated reason.

The second thread is the one about addresses not being constants. Relocations exist because a symbol's address is not known when the code is written. TLS pushes that one step further: the address is not even a constant once the program is running, because it differs per thread. Every relocation mechanism in this course has assumed a program has one set of addresses; this is the case where that assumption is false, and the format's response — a section that holds values rather than variables — is a genuine change of level rather than another relocation type.

Which is a good note to end the module's format coverage on before the last two concepts, because both of them are the same idea seen from opposite ends. This one carries information that only the loader can use, at run time, per thread. The next one carries information that only the linker can use, at link time, and is thrown away immediately after.

Next: Weak Externals.