One variable, one source file, two threads. The addresses are tens of megabytes apart.
$ cat tt.c
#include <stdio.h>
#include <pthread.h>
__thread int t[4];
void *show(void *a){ printf(" thread %p sees t at %p\n",
(void*)pthread_self(), (void*)t); return 0; }
int main(void){ pthread_t b;
printf(" main sees t at %p\n",(void*)t);
pthread_create(&b,0,show,0); pthread_join(b,0); return 0; }
$ ./tt
main sees t at 0x7091c023f770
thread 0x7091bffff6c0 sees t at 0x7091bffff6b0
Two different addresses for one variable. The Relocations course established why — a thread-local has no address until a thread exists — and showed the four models and the one that has to call a function. What it could not show is where the storage comes from, and the answer turns out to be a decision the kernel makes before main runs, using a number the linker computed and the loader copied.
Three parties, and the order they act in is the whole point.
The offset being constant is what makes the fast models fast. The kernel guarantees the block is at a page-aligned address, so the low 12 bits of any thread-local’s address are the same in every thread — which is exactly what the four relocation models are built on. TPOFF32 is “my offset in the block”, and the instruction computing the address is %fs:0-relative plus that constant.
And the reason the general-dynamic model exists is now visible: step 2 is a kernel limit, and it is not unbounded.
The layout, and the fact that it is consecutive:
$ cat tls.c
__thread int big_tls[64] = {1}; /* 256 bytes */
__thread int small_tls = 2;
int *where_big(void){ return big_tls; }
int *where_small(void){ return &small_tls; }
$ ./tlsmain
big=0x7ae26674a770 small=0x7ae26674a870
Exactly 0x100 apart, which is sizeof(big_tls). Two variables declared in one file, one after the other, and their addresses differ by precisely the size of the first. The block is a contiguous allocation and the offsets are the distances between them — which is the observation that makes the TPOFF relocations in the Relocations course obviously correct rather than merely stated.
And the template the kernel copies from:
$ readelf -lW tlsmain | grep '^ TLS'
TLS 0x002cc0 0x0000000000003cc0 0x0000000000003cc0 0x000104 0x000104 R 0x10
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
filesz = memsz = 0x104
$ readelf -SW tlsmain | grep -E '\.tdata|\.tbss'
[20] .tdata PROGBITS 0000000000003cc0 002cc0 000104 00 WAT 0 0 16
^^^^^ 0x104
p_filesz is the template and p_memsz is the reservation, and they are equal here — because both variables are initialised, so everything is in .tdata and there is no .tbss to zero. The WAT flags on the section are worth reading: Writable, Allocatable, Thread-local Storage, all three of which the kernel acts on.
Uninitialised thread-locals would show the difference, and that is the case worth understanding rather than measuring:
And the demonstration that makes the whole thing concrete — the same variable, two threads, and the offsets are identical even though the addresses are not:
$ ./tt main sees t at 0x7091c023f770 thread 0x7091bffff6c0 sees t at 0x7091bffff6b0
Look at the low bits.770 and 6b0 — not equal, and not obviously related, because the thread blocks are page-aligned but the offset within a page differs by however the two blocks were placed. The high bits are tens of megabytes apart; the page offset is a per-thread allocation detail. What is identical in both threads is the offset within the block, and that is the value the compiler baked in and the TPOFF32 relocation carries.
And one thing this course does not claim. The static-surplus threshold — the point at which the kernel’s reservation runs out and __tls_get_addr starts appearing — is real and is the mechanism the Relocations course measured on the code side. It is not measured here. The attempt to measure it on this machine, by building thirty-two dlopen-able libraries each carrying 256 bytes of thread-local storage, failed to link at all: --as-needed dropped every one of them because nothing referenced a symbol from them. Measuring it needs a program that actually uses them, and that was not built. The mechanism is taught; the number is not claimed.
Why a thread that is created, used and destroyed leaves its block behind, and what that costs.
the kernel does not hand a thread's TLS block back when the thread exits -- the STACK is cached for reuse, and the TLS block goes with the thread structure. a program that creates and joins a million threads in a loop therefore allocates a million TLS blocks, and the peak is one block per LIVE thread. the fix is not in the linker and not in the loader. it is: do not create a million threads, or make the per-thread state small, or use a pool. which is worth knowing because the cost is invisible until it is enormous: a thread costs a stack (8 MB of address space by default) AND a TLS block, and the second is proportional to how much __thread state the THREAD'S libraries declare -- including libraries the thread's own code never touches.
The per-thread cost of a library is paid by every thread, whether or not that thread uses the library. That is the practical consequence of the model, and it is the reason __thread in a widely-loaded library is a more expensive decision than it looks.
And the one genuinely counter-intuitive property, which is what makes the model worth a concept:
PT_TLS describes the block for the WHOLE PROCESS, not per object. the linker concatenates .tdata and .tbss from every object and every library into ONE segment with ONE size. so: a library cannot ask for its own TLS block. it contributes a fragment, the linker concatenates, and the kernel reserves the sum for every thread. and the offsets handed out are global. a library compiled standalone does not know its offset until it is linked into something, which is exactly why the general-dynamic model exists to cope with being loaded when the answer is no longer knowable.
That is the last piece of the thread-local story, and it closes a loop three courses long. A __thread variable cannot have a link-time address, because the block is per-thread; it cannot have a small per-object offset either, because the block is per-process; and it cannot be given one by a dlopened library, because the reservation has already happened. General-dynamic TLS is not an optimisation that got out of hand — it is what is left when a value has no constant address for three independent reasons.
$ cd courses/dyn/assets/samples $ ./build_samples.sh 2>&2 | sed -n '/D13/,$p' $ python3 crosscheck.py 2>&2 | sed -n '/\\[13\\]/,/\\[14\\]/p'
Then finish the experiment this page declined to do, which is the honest next step:
1. Build N libraries, each with 256 bytes of
__thread state, and a program that REFERENCES a
symbol from every one -- otherwise --as-needed
drops them, which is exactly what went wrong when
this page was written. Then:
for N in 1 2 4 8 16 32 64:
readelf -dW progN | grep -c tls_get_addr
Find the N where the answer turns from 0 to 1.
(That number is the surplus on this machine, and it
is a property of the KERNEL -- not of the linker,
not of the compiler, and not portable.)
2. Read PT_TLS p_memsz for progN. Does it grow with
N? It should grow by exactly 256 per library, and
that is the number the kernel multiplies by the
thread count.
3. Add a __thread variable with an initialiser to one
library and an uninitialised one to another.
Compare .tdata and .tbss, and PT_TLS p_filesz
against p_memsz. (This is the part this page
describes by mechanism and does not measure.)
4. Write a program that creates and joins 100,000
threads, each touching one __thread variable.
Watch RSS. Then remove the __thread and watch
again. (The stack dominates -- 8 MB of ADDRESS
space, not necessarily resident -- and the TLS
block is the part that scales with the LIBRARIES.)
5. In a real program, find the thread with the
largest TLS block. It is not the one you expect,
and finding it is a genuinely useful thing to know.
Exercise 1 is the one this page explicitly could not complete, and leaving it as an exercise rather than a result is the right call. The failure was instructive: --as-needed silently dropped thirty-two libraries that had been built, linked and reported no problem, because nothing referenced a symbol from them. A measurement that requires the thing under test to actually be present has a failure mode that looks exactly like success, and that is worth internalising independently of thread-local storage.
Exercise 3 is the one that makes the .tbss description real rather than remembered. p_filesz is what the kernel copies and p_memsz is what it reserves, and the gap between them is exactly the zero-filled tail. Every other segment in the file has filesz ≤ memsz for the same reason — .bss occupies address space with no file content — and thread-local storage is simply the case where the kernel has to do the zeroing itself, per thread, at run time.
This closes module 3, and it is the third and last piece of a story the Relocations course started and could not finish. That course established that a thread-local has no address until a thread exists, measured the four models, and found that general-dynamic is the only one that has to call a function. This concept answers the two questions that left over: where the storage comes from, and why the fast models can be fast at all.
The answer turns out to be a link-time constant, which is the pleasing part. The offset of a thread-local within its block is decided by the linker, concatenated across every object, and is the same in every thread — because the kernel guarantees each block starts at a page boundary and copies the same template into each one. That is why TPOFF32 is a four-byte immediate rather than a pointer chase, and it retroactively explains the Relocations course’s measurement that -fPIE needed no __tls_get_addr call: a PIE’s block is arranged so the offset is knowable, and the general-dynamic path only appears when it is not.
The second connection is to the ELF course’s entry-point concept, because the kernel’s TLS reservation happens before the program’s first instruction. PT_TLS is read by the same kernel code that reads PT_INTERP, before it decides whether to load a dynamic linker at all. A static binary with thread-locals still gets its blocks, which is a small and pleasant fact: the TLS machinery is the loader’s job, but the reservation is the kernel’s, and it happens in both cases. That is why thread-local storage is one of the few features that works identically in a static binary and a dynamic one.
And the connection to the module it sits beside, which is about a different kind of reservation. Lazy, Eager is about when a binding decision happens. This concept is about where storage comes from. Both are the loader doing work no linker could, and the pair covers the two things that make dynamic linking more than a name-resolution scheme: it needs a program to be running, and it needs memory that did not exist when the file was written.
Forward to the last concept, which is the build. Rebuilding the Scope Yourself does not handle thread-locals — and saying so is part of it. The artifact reads DT_NEEDED, walks it breadth-first, and resolves plain names. Versioned symbols, RTLD_NEXT, and thread-local offsets are all extensions that need more tables, and naming them precisely is how you know what the two-hundred-line version does and does not model. A resolver that quietly ignored a version suffix would be worse than one that refused the file, and the refusal is a feature.
One connection outward, for the platform framing. The Static Linking course claimed that dlopen is one of the things -static gives up, and cited thread-local storage as part of the reason. This concept is the mechanism behind that claim, and it is a satisfying one: dlopen of a library with thread-locals either needs the surplus or needs a call to __tls_get_addr, and a static binary has no dynamic loader to provide either. The static-linking course stated the consequence; this one gives the cause, three courses apart.