Back to course

The One Relocation That Calls the Loader

25 minadvancedModule 3: How It FailsContinues from pic-violation
25 min · Module 3: How It Fails · Advanced

Why This Matters

Every relocation so far had the same shape: read three numbers, do arithmetic, write four bytes. This one does not, and the reason is not an oddity of the encoding. It is because a thread-local variable has no address until a thread exists, and threads are created at run time, long after every program that could resolve it has exited.

Here is the smallest specimen that shows it, and the relocations the compiler emits:

$ cat tls2.c
extern __thread int ext_tls;      /* another module */
static __thread int my_tls;       /* THIS module     */
int rd_ext(void){ return ext_tls; }
int rd_mine(void){ return my_tls; }

$ clang -O0 -fPIC -c tls2.c -o tls2_pic.o
$ llvm-objdump-21 -r tls2_pic.o | sed -n '/.text/,/^$/p'
0000000000000008 R_X86_64_TLSGD        ext_tls-0x4
0000000000000010 R_X86_64_PLT32        __tls_get_addr-0x4
0000000000000027 R_X86_64_TLSLD        my_tls-0x4
000000000000002c R_X86_64_PLT32        __tls_get_addr-0x4
0000000000000033 R_X86_64_DTPOFF32     my_tls

Two __tls_get_addr calls, in one function, in one object file. Not one relocation each that the loader patches — two function calls, to a function the object file imports from somewhere else. Nothing in the previous two modules prepares you for a relocation list that contains a call.

A Simple Model

Why the address is unknowable, stated so the impossibility is obvious rather than technical.

a normal global: one address, decided by the LINKER and true for the life of the process. a thread-local: one address PER THREAD. so "the address of my_tls" is not a value. It is a FUNCTION of which thread is asking. and the offset of a thread's TLS block depends on: - which modules are loaded - in what ORDER they were loaded - how big each module's TLS block is - WHICH THREAD (each thread gets its own copy) none of which exist when the linker runs, and the last of which does not exist even when the loader runs.

So the address is not a constant to be written into a field; it is a computation to be performed later, on a specific machine, about a specific thread. That is why this is the one relocation class where the loader must execute code rather than store a number, and it is the reason the type name says “get addr”.

Which raises the obvious question: if the answer is expensive and dynamic, why does any program use thread-locals at all? The honest answer is that thread-locals are cheap at run time and expensive to set up, and the four models are four different answers to “how much setup does this program want”.

THE FOUR MODELS, in order of increasing speed and decreasing generality 1. GENERAL-DYNAMIC the loader is asked about the SYMBOL, one call per reference relocs: TLSGD + PLT32 __tls_get_addr works for: a symbol in ANY module, loaded or not yet loaded, at ANY time cost: a function call per access 2. LOCAL-DYNAMIC the loader is asked about the MODULE, once, then a module-relative offset is added relocs: TLSLD + PLT32 __tls_get_addr + DTPOFF32 works for: a symbol in THIS module only cost: one call for N symbols, so it amortises 3. INITIAL-EXEC the loader assigns an offset at load time; the code reads the GOT relocs: GOTTPOFF works for: a module that is loaded AT STARTUP cost: no call at run time 4. LOCAL-EXEC the offset is known AT LINK TIME; code reads it straight from the thread pointer relocs: TPOFF32, and nothing else works for: the executable's OWN thread-locals cost: no call, no GOT, no indirection at all

Read that as one sentence: the more you assume about the load, the faster the access. General-dynamic assumes nothing. Local-exec assumes the module is the executable and is the first thing mapped — which is why it is only usable for a program’s own thread-locals and never for a library’s.

Does It Actually Work?

The whole ladder, measured. -O0 is deliberate and the reason matters, so read the note under the table:

$ for fl in "-fPIC" \
           "-fPIC -ftls-model=local-dynamic" \
           "-fPIC -ftls-model=initial-exec" \
           "-fPIC -ftls-model=local-exec" \
           "-fPIE" \
           "-ftls-model=local-dynamic"; do
    clang -O0 $fl -c tls2.c -o y.o
    printf "%-38s " "$fl"
    llvm-objdump-21 -r y.o | grep -oE "R_X86_64_[A-Z0-9_]+" | sort -u | tr "\n" " "
    echo
  done

-fPIC                              TLSGD TLSLD DTPOFF32 PLT32
-fPIC -ftls-model=local-dynamic   TLSLD DTPOFF32 PLT32
-fPIC -ftls-model=initial-exec    GOTTPOFF
-fPIC -ftls-model=local-exec      TPOFF32
-fPIE                              GOTTPOFF TPOFF32
-ftls-model=local-dynamic         GOTTPOFF TPOFF32   <-- OVERRIDDEN

Three things to take from that table. First, the ladder is monotone: each step down removes a loader call, and the last step removes the GOT as well. local-exec is a single TPOFF32 — one relocation, no call, no slot, nothing to initialise.

Second, and this is the part that surprises people: -fPIC chooses per symbol, not per file. Row one has bothTLSGD and TLSLD, because the extern symbol needs general-dynamic and the module-local one can use local-dynamic. The compiler is not picking a model for the file; it is picking, per access, the cheapest model that is valid.

Third, look at the last row. -ftls-model=local-dynamic without -fPIC is silently overridden — no TLSLD, no call. You asked for local-dynamic and did not get it. The reason is worth stating as a general rule about compiler flags:

  THE FLAG IS A REQUEST.  PIC-NESS IS A CONSTRAINT.

  without -fPIC, the compiler knows the output is an
  ET_EXEC, which is loaded FIRST and at a KNOWN
  position in the TLS layout. Under that constraint
  local-dynamic is not merely unnecessary -- it is
  WRONG, because the module id of the executable is
  known at link time and asking the loader for it is
  pure overhead.

  so the compiler picks for itself and ignores you.

  and the models only become selectable at all when
  -fPIC is present, because only then is the answer
  genuinely unknown.

This is the same shape as the finding in the PIE module that a codegen flag and a link flag can disagree, and it is the same lesson: the compiler answers a question about the output, and it has to guess the output from the flags it was given. Here the guess is wrong in a way that costs performance rather than correctness, which makes it easy to miss.

Now the note on -O0, because it is a measurement lesson and not a formality:

  the first version of this test used:
      static __thread int my_tls = 7;
  compiled at -O1, and concluded that local-dynamic
  and initial-exec "emit the same relocations".

  the truth: -O1 folded `return my_tls;` to the
  constant 7. the memory access disappeared, and the
  TLSLD relocation disappeared with it. there was
  nothing left to distinguish the two models.

  and the extern-only specimen could not have shown
  the difference anyway, because local-dynamic
  REQUIRES the symbol to be in the same module.

  two mistakes, in the same test, pointing the same
  way. the crosscheck now builds both specimens at
  -O0 precisely so this cannot recur.

The general form of that mistake is worth carrying: an optimiser that removes the code removes the evidence. When a measurement is “these two things look the same”, the first hypothesis should always be my optimiser deleted the difference, not they really are the same.

A Real Example

What the two loader calls actually do, and why general-dynamic takes two arguments.

  __tls_get_addr is a real exported function.
  find it:

  $ nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep tls_get_addr
  0000000000a1b30 T __tls_get_addr@@GLIBC_2.34
                ^ a global, in libc, versioned

  GENERAL-DYNAMIC asks a QUESTION ABOUT A SYMBOL:
      __tls_get_addr(&modid, &offset, &ext_tls)
      "this is the module id I think it is; this is
       the offset I think it is; here is the symbol.
       correct me."

  LOCAL-DYNAMIC asks A QUESTION ABOUT A MODULE:
      __tls_get_addr(&modid, &offset, 0)
      "same question, but the symbol is in THIS module,
       so I do not need to look it up -- just tell me
       where my module's block is, once."

  the loader compares what you claimed against its own
  tables and fixes you up if you guessed wrong. it
  caches the answer, which is why the SECOND access
  in a thread is cheap even in general-dynamic.

That “claim and correct” protocol is the actual mechanism, and it explains a property that surprises everyone the first time they see it. The compiler emits the module id and offset it believes, the loader checks, and the answer is usually right. That is what makes a PLT32 against __tls_get_addr safe even though the offset is genuinely unknowable at link time: the value in the file is a guess, and there is a protocol for being wrong.

And the reason general-dynamic cannot be optimised away even when the symbol is in the same module is exactly the claim in the table above — the compiler cannot know whether something will later interpose a definition. -fPIC means “assume it might”, and that assumption is what forces the general model.

What the linker cannot do is catch a misapplied model. Forcing -ftls-model=local-dynamic on a symbol defined in another module links without a diagnostic:

$ clang -O0 -fPIC -ftls-model=local-dynamic -c tls3.c -o t3ld.o
$ clang -o t3ld t3ld.o t3def.o        # t3def.c DEFINES ext_tls
$ echo $?
0
$ ./t3ld; echo $?
42

It linked, and it ran, and it was right. And that last part needs a caveat, because it is easy to over-read: this test contains exactly one thread-local module, so the module id the code guessed and the offset the linker assigned happen to agree. I did not construct a case where it gives the wrong answer. The honest claim is that the misuse is latent here, not observable — the linker has no mechanism to detect it, which is the part that matters. A tool that cannot detect a class of misuse is a tool where the misuse is a debugging session rather than a message.

And the practical reason to care about all four models is that choosing badly is silent. A library compiled with general-dynamic because it might be dlopened pays a function call per thread-local access forever; one compiled initial-exec because the author was in a hurry crashes at load time on a dlopen, with a message that points at the caller rather than the model.

Try It Yourself

$ cd courses/reloc/assets/samples
$ ./build_samples.sh 2>&1 | sed -n '/F6/,/F7/p'
$ python3 crosscheck.py 2>&1 | sed -n '/\\[6\\]/,/\\[7\\]/p'

Then derive the model rather than memorise it:

  1. Add a third thread-local, module-local, and
     compile -fPIC. How many __tls_get_addr calls now?
     (Answer: still TWO, not three. Why is that the
     whole point of the local-dynamic model?)

  2. Compile tls2.c at -O1 and at -O0 and diff the
     relocations. Whatever differs, explain the
     optimiser step that removed it.

  3. Make a THREAD-LOCAL MODULE. Two .so files, each
     with its own __thread int, linked into one program.
     Then force local-dynamic on a cross-module symbol
     in a third file, and see whether you can now make
     it give a wrong answer. (This is the case the
     concept says was not constructed here. Try it.)

  4. Add a static __thread int with an initialiser,
     compiled at each -O level. At which level does the
     access vanish, and what is the smallest initialiser
     that survives? (The answer is not zero, and finding
     out why is the exercise.)

  5. Find the __tls_get_addr PLT entry in a real
     binary. Does it exist even in a program that never
     calls it? What does that tell you about when the
     decision to call the loader was made?

Question 3 is the one that closes the loop, and it is deliberately the measurement this course declined to make. The claim on this page is that the misuse is latent under a one-module test and that the linker cannot detect it; whether it is actually wrong under two modules is a separate experiment with a separate answer. A course that quietly turns “I did not test this” into “it is broken” is teaching you to trust a number nobody checked.

Connect

This is the second failure concept and it is the deliberate contrast to The Relocation That Cannot Be Fixed. Both are relocations the linker cannot finish, and they fail in opposite ways. A PIC violation is a request with no answer, so the link fails and the message names the relocation type. A general-dynamic TLS reference is a request with a deferred answer, so the link succeeds and the answer is computed by calling a function while the program runs. One is a compile-time constraint enforced at link time; the other is a run-time protocol. Holding both in your head is what makes the phrase “the loader resolves dynamic relocations” stop sounding like a catch-all.

It also amends the group model from the vocabulary module, and the amendment is worth isolating. That concept put TLSGD, GOTTPOFF and TPOFF32 in group 4 — the loader’s relocations — alongside GLOB_DAT and JUMP_SLOT. Group 4 has to be split.GLOB_DAT is a number to store; TLSGD is a function to call. The unifying idea is not “the loader does it” but “the answer is not available until run time”, and that criterion admits both the arithmetic and the non-arithmetic cases.

The connection to the AArch64 contrast in the first concept is structural and quite pleasing. AArch64 needs two relocations per reference because one instruction cannot reach far enough. TLS general-dynamic needs two relocations — a TLSGD and a PLT32 — and here it is not about range at all. Same shape, different reason: when one relocation record cannot carry the whole answer, the format allows a sequence. A tool author reading a relocation list has to be prepared for entries that are only meaningful in pairs, and the two architectures fail in opposite directions — AArch64 pairs by proximity, TLS pairs by role.

Forward to the build exercise, this concept is the one case the applier cannot handle, and knowing why is the point. Applying Them Yourself builds a patcher that does arithmetic on three numbers. R_X86_64_TLSGD has no arithmetic to perform — the value is a runtime function result, and the file contains a guess plus a protocol for correcting it. So the applier has to recognise the class and decline, and declining correctly is part of applying relocations correctly. A patcher that wrote a number into a TLSGD field would produce a binary that links, runs, and returns garbage for a thread-local read — the most expensive kind of bug, because nothing reports it.

And one connection out to the platform layer, for the security framing. Does the Executable Actually Move? listed the attacks PIE removes, and attack (c) was a return address on the stack, “not address-dependent, not defeated by PIE.” Thread-local storage is a second, subtler instance of that: an offset from a thread pointer, where the pointer changes per thread and the offset is a claim the loader may correct. An attacker who can read or write a thread pointer is in the same position as one who can read a stack pointer, and PIE does nothing for either.