Back to course

COMMON and the ABI Break

20 minintermediateModule 2: Anatomy, ComparedContinues from obj-strings
20 min · Module 2: Anatomy, Compared · Intermediate

Why This Matters

One line of C is about to cost you an afternoon, and it is the shortest declaration in the language:

int tentative;

No initialiser. No extern. No storage class. The C standard says this is a tentative definition, and it also says what happens when two translation units both write it: one definition wins, every reference resolves to it, and all of them are zero-initialised. The language answers the question. The question is how a file format expresses an answer that says "somebody, somewhere, not necessarily me, is holding four bytes of zeroes, and here is how big they are and how aligned they must be".

Three formats answer it three ways, and the history of one compiler flag changing its default in 2019 broke essentially every prebuilt Linux package in the world. That is not a metaphor; it is a well-documented incident, and the mechanism behind it is a single bit in a symbol table. This concept is about that bit, and about a field that does not mean what it looks like it means.

A Simple Model

First, the ordinary case, because the interesting case only makes sense against it. There are two different things a declaration can mean:

int initialised = 5; a DEFINITION. 4 bytes, value 5, known at compile time, goes in .data int tentative; a TENTATIVE DEFINITION. 4 bytes, value 0, not known to be unique -- another file may also define it

Both end up as four bytes somewhere. The difference is not the bytes, it is the promise. A definition says "this is the one and only copy". A tentative definition says "if nobody else provides one, I will be it, and we will merge if they do too."

A format can express that in three ways, and all three do:

The second is the one with the trap, and it is worth stating the trap before the measurement because it is a field that does not mean what it means.

The Actual Detail

One source file, three words, compiled five ways on this machine. The source is:

  int tentative;             /* tentative definition */
  int initialised = 5;       /* real definition       */
  int main(void)      -- returns tentative + initialised

And here is every difference between them, measured with readelf -s and llvm-nm:

                    tentative              initialised
  build             Ndx  Val  Size  Type       Ndx  Val  Type
  ----------------  ---  ---  ----  --------  ---  ---  --------
  clang -fno-common   5    0     4  B (.bss)      4    0  D (.data)
  clang -fcommon    COM    4     4  C (COMMON)   4    0  D (.data)
  gcc (default)      5    0     4  B (.bss)      4    0  D (.data)
  COFF               ?    0     4  B            4    0  D
  Mach-O           0x60    -     4  S (__common) 0x1c  -  D

Four findings, and the first is the one that will cost you an afternoon.

st_value is 4, and it is not an address

Look at the COMMON row. Value = 4. That is not an offset and not an address — it is the required alignment. A COMMON symbol has no section and no position, so ELF repurposes the st_value field to hold the alignment the linker must give the allocation, and puts the size in st_size.

This is the only symbol in the format whose Value field does not contain a value. A reader that adds a section base to it, or treats it as an offset into a COMMON section, gets a number that is meaningless. And note what the two rows above it show: for .bss the value is 0, for COMMON it is 4, and both mean "position 0 of a 4-byte object" to a human reading the file. The difference is invisible unless you know that one of them is an alignment. This is the same species of bug as the Mach-O provisional-address finding, one level deeper: a field whose meaning depends on a neighbouring field, which depends on a section index whose meaning is a special case.

With -fcommon there is no .bss section at all

The section lists are the proof, and they are the whole story of what COMMON buys:

  clang -fno-common:  .strtab .text .rela.text .data .bss
                      .comment .note.GNU-stack .eh_frame
                      .rela.eh_frame .llvm_addrsig .symtab
                                                 ^^^^^ an 11-section file

  clang -fcommon:     .strtab .text .rela.text .data
                      .comment .note.GNU-stack .eh_frame
                      .rela.eh_frame .llvm_addrsig .symtab
                                                .bss is GONE
                      1240 bytes vs 1312. 72 bytes smaller.

The file is 72 bytes smaller because the section header that would have described .bss is not emitted at all. A COMMON symbol needs no section to live in, so no section is created; the linker invents a .bss-shaped region in the output once it has merged every COMMON symbol it collected.

And that merge is the entire point. The linker's rule is: collect every COMMON symbol of a given name, allocate the largest requested size at the strictest requested alignment, and point every reference at that one allocation. A .bss-based tentative definition cannot do that, because it is already inside a section with a fixed size — so two files each declaring int tentative; produce two symbols in the same .bss and the link fails with a duplicate-symbol error.

Mach-O solved it with a real section

Mach-O's __common is the third approach and it is the most literal: a real section, of type S_ZEROFILL (flags = 0x1), with size 4 and offset 0, at address 0x60. The symbol is a normal symbol in a normal section — nm shows it at 0x60 — and the section exists so the symbol has somewhere to point.

  Mach-O tentative definition:

    __common  addr=0x60  size=4  offset=0  align=2  flags=0x00000001

    offset = 0 AND size = 4: four bytes of memory that
    occupy no bytes of file. Same idea as .bss, but reached
    by making a section out of the concept rather than
    putting the concept in a section.

So three mechanisms for one C rule, and the honest ranking by how much machinery each needs. ELF's SHN_COMMON is the cheapest in the file (no section at all, and the field reuse is a genuine trick) and the most expensive in the reader, because it turns st_value into a second thing depending on st_shndx. Mach-O's __common is the most expensive in the file and the cheapest in the reader, because a symbol in a section means what it always means. COFF sits in between, using a storage class rather than a special section index, and inheriting the same "one field, two jobs" problem as the previous concept found.

What -fcommon actually changed in 2019

Here is the incident, and it is worth telling precisely because it is the strongest available argument for the mission's "verify, do not assume" rule.

GCC 9 and earlier: -fcommon was the DEFAULT GCC 10 and later: -fno-common became the default a library built with GCC 9: lib.a <- symbols marked COMMON an application built with GCC 10: o.o <- symbols in .bss linking them: o.o's .bss tentative definition vs lib.a's COMMON -> "duplicate symbol" error -> the application does not build

Every prebuilt C library on a Linux distribution, and every application linking against one, broke. Distributions fixed it by rebuilding everything with matching flags, and the lasting change was that -fcommon became something you opt into rather than something you inherit.

Now the interesting part for a format designer. Neither behaviour is a bug and neither is the standard. C permits a translation unit to treat a tentative definition as a definition, so -fno-common is conforming; it permits merging, so -fcommon is conforming too. The two disagree about whether two tentative definitions in different files are the same object or a conflict — a question the C standard arguably leaves open, and which the two flags answer differently. So the ABI break was not a compiler bug. It was two toolschains answering an underspecified question differently, and the file format had no way to make them interoperate because the format faithfully recorded a decision each side made locally.

The general lesson, which is about formats rather than about C. When two producers of the same format disagree about a semantic question, the format cannot fix it — it can only record the answer faithfully, and then a consumer has no way to tell that two records mean incompatible things. ELF's answer to that is the .gnu.linkonce and section-group mechanisms, which exist to let a producer say "these definitions may be merged" without adopting the whole COMMON convention. A format that wants to survive disagreement has to give producers a way to record the disagreement, not just a way to record agreement. That is a design requirement, and it is the same reason the class file's attribute mechanism exists.

A Real Example

What a linker actually does with each of the three, because the reason COMMON exists is a scheduling problem rather than a semantic one:

the problem: file A says int buf[1000]; file B says int buf[8]; both tentative. Who allocates what? ELF COMMON: the linker collects every COMMON symbol by name, takes the MAX size and the MAX alignment, allocates once, and rewrites all references. The output gets a .bss-like region sized to the largest request. ELF .bss: each file allocated its own, in its own section. A second tentative definition is a DUPLICATE and the link fails. Correct, and incompatible. Mach-O: a real __common section exists, so the merge can happen the ordinary way -- but the linker still has to recognise these as mergeable, so the compatibility problem is the same one, hidden behind a section that looks like ordinary data.

So all three converge on the same linker algorithm, and the difference is purely how much the file has to say about it. That is the useful way to hold the concept: COMMON is not a different kind of storage, it is deferred storage, and the format's job is to represent "I have not been allocated yet" in a way that survives round-tripping through a file.

The related trap is worth naming because it catches people building tools rather than linkers. A .bss section has a nonzero size and occupies zero file bytes — measured in the previous concept on both COFF's .bss and Mach-O's __common — and a reader that copies size bytes from offset writes zeroes over whatever follows. For .bss the offset is 0 and the section is not in the file at all, so the copy reads the ELF header. The rule is simple and worth stating as a check: a section with a nonzero size and a zero offset contributes nothing to the output file, and that is a valid, common, load-bearing case rather than a malformed one.

And the reason this matters to a code generator, which is where this course is going. Emitting a tentative definition is a decision with an interoperability consequence, and the compiler made it for you by default. A code generator that always emits .bss produces objects that cannot link against libraries built by a -fcommon toolchain. A generator that always emits COMMON produces objects of a kind most modern toolchains do not produce. Neither is safe by default, and the flag exists because the choice has to be made somewhere — so it is a decision your backend has to be able to make, and document.

Try It Yourself

$ clang -target x86_64-pc-linux-gnu -O0 -c -o a.o tent.c
$ clang -target x86_64-pc-linux-gnu -O0 -fcommon -c -o b.o tent.c
$ readelf -sW a.o | grep tentative
$ readelf -sW b.o | grep tentative

Check Your Understanding

Without looking back: an ELF symbol has st_shndx = COM, st_value = 4 and st_size = 8. Another has st_shndx = 5, st_value = 0 and st_size = 8. What does st_value = 4 mean in the first, what does it mean in the second, and what does a reader that adds a section base to the first one produce?

Apply It

You are writing a code generator, and you have a linker bug report: a program links cleanly but reads the wrong value from a global that its own translation unit declared with no initialiser. Your object puts the symbol in .bss with st_value = 0. A library object in the same link has the same symbol with st_shndx = COM and st_value = 8. The linker's merge output gives your symbol 8 bytes of storage. What did your generator do differently from the library's, and what is the one-line change that makes your objects link against both kinds?

The general habit: a code generator inherits whatever its compiler's default was, and defaults are compatibility decisions rather than correctness ones. If your objects must interoperate with objects you did not produce, that decision is now yours to make explicitly — and to document.

Connect

COMMON is the clearest case in the whole course of a semantic decision that has to travel through a file, and the ELF course teaches .bss and the common sections as features. What this concept adds is the why they are optional — that the format has a representation for "not allocated yet" and that using it or not is a choice with observable consequences. That is the difference between learning a format and understanding what it is for.

The 2019 break connects to something structural about the platform rather than about C. A distribution ships prebuilt libraries and expects applications to link against them, which means the libraries' object-file conventions become a published interface whether or not anyone meant them to. The same mechanism appears in the PE world with C runtime libraries and in the Mach-O world with the C++ standard library's ABI, and it is the reason symbol visibility and weak symbols exist as separate concepts. A tentative definition is a weak definition in all but name, and both mechanisms exist so that two files can agree to share storage without either having to know the other exists.

That last point makes COMMON the natural on-ramp to the rest of Module 4, and specifically to weak and undefined-weak symbols. The rule is the same in both: a symbol may be defined in several places, and exactly one definition wins, and everybody gets it. The mechanisms differ — COMMON carries a size and an alignment and is merged by a sum-like rule, while a weak definition carries no size and is simply chosen — but the intent is identical, and a reader who has internalised the COMMON case has most of what it takes to implement the weak case.

On the byte-count side, this concept is the third appearance of "nonzero size, zero offset means no file bytes", after COFF's .bss and Mach-O's __common in the sections concept. Three sightings of one pattern, in three formats, and the rule a reader needs is a single line: a section with a nonzero size and a zero offset contributes nothing to the output. It keeps coming back because it is the one case where size and offset describe different things — memory and file — and a reader that assumes they are the same quantity writes zeroes over the ELF header. That assumption is the kind that looks reasonable in code review and is invisible in testing, which is why it is worth stating as a check rather than a caveat.

And the meta-lesson, which is the one this course is built to teach. A format can only record a decision faithfully; it cannot make two producers make the same one. ELF gave producers two sound ways to say "these definitions merge" — SHN_COMMON and the .gnu.linkonce section-group mechanism — and the 2019 break still happened, because nothing in the format says that .bss and COMMON mean incompatible things. When you design a format, ask what happens when two correct producers disagree, and make sure the answer is not "silently, forever". That is the same question the JVM course's attribute mechanism answers by letting a new attribute be skipped rather than misread, and it is why skipping unknown attributes is a safety property and not merely a convenience.