Back to course

Security & Subsystem Flags

15 minintermediateModule 6: Relocations & HardeningContinues from pe-base-relocations
15 min · Module 6: Relocations & Hardening · Structures

Why This Matters

Before mapping a single section, Windows reads a few tiny fields and decides: is this a console tool or a GUI app, is a 1 MiB stack enough, may this image move for ASLR, may its pages be non-executable, does it claim Control Flow Guard, is there a signature at the end of the file? None of these fields hold code or data — they are the binary's self-declared posture, and both the OS and the security ecosystem treat them as promises.

This is the first lesson in the course where a wrong value has consequences beyond "your program crashes". Misreading DYNAMIC_BASE tells you ASLR is on when it is not; misreading the certificate directory's first field as an RVA sends your parser into the wrong part of the file entirely. And because these are plain bitfields, they are also the fastest thing in the whole format to verify by hand — which is exactly what incident responders and malware analysts do first.

A Simple Model

Four independent knobs, in four places:

Sitting next to Subsystem are four size fields the loader also honors: stack reserve, stack commit, heap reserve, heap commit. The model's missing piece: reserve versus commit. Reserve is address space set aside; only the commit size is backed by physical memory up front, and the rest is granted one page at a time as it is touched.

The Actual Detail

COFF Characteristics bits, from the spec's Characteristics table (the subset that matters for loading and hardening):

ValueConstantMeaning
0x0001IMAGE_FILE_RELOCS_STRIPPEDNo base relocations; must load at ImageBase or the loader reports an error
0x0002IMAGE_FILE_EXECUTABLE_IMAGEThe image is valid and can run; if clear, it is a linker error
0x0004IMAGE_FILE_LINE_NUMS_STRIPPEDCOFF line numbers removed (deprecated, should be zero)
0x0008IMAGE_FILE_LOCAL_SYMS_STRIPPEDLocal COFF symbols removed (deprecated, should be zero)
0x0020IMAGE_FILE_LARGE_ADDRESS_AWAREApplication can handle addresses above 2 GB
0x0100IMAGE_FILE_32BIT_MACHINEMachine is a 32-bit-word architecture
0x0200IMAGE_FILE_DEBUG_STRIPPEDDebugging information removed
0x1000IMAGE_FILE_SYSTEMSystem file, not a user program
0x2000IMAGE_FILE_DLLThe file is a DLL

Subsystem values (spec's "Windows Subsystem" table): 1 NATIVE (drivers and native processes), 2 WINDOWS_GUI, 3 WINDOWS_CUI (character/console). More exist for EFI, Xbox, and boot applications.

DllCharacteristics bits (spec's "DLL Characteristics" table) — bits 0x0001-0x0008 are reserved and must be zero:

ValueConstantMeaning
0x0020HIGH_ENTROPY_VACan handle a high-entropy 64-bit address space
0x0040DYNAMIC_BASECan be relocated at load time (needs a .reloc table to be true)
0x0080FORCE_INTEGRITYCode integrity checks are enforced
0x0100NX_COMPATImage is NX (DEP) compatible
0x0200NO_ISOLATIONIsolation aware, but do not isolate
0x0400NO_SEHNo structured exception handling; no SEH handler may be called in this image
0x0800NO_BINDDo not bind the image
0x1000APPCONTAINERMust execute in an AppContainer
0x2000WDM_DRIVERA WDM driver
0x4000GUARD_CFSupports Control Flow Guard
0x8000TERMINAL_SERVER_AWARETerminal Server aware

Decoded real values: cli-64.exe has DllCharacteristics 0x8160 = HIGH_ENTROPY_VA | DYNAMIC_BASE | NX_COMPAT | TERMINAL_SERVER_AWARE; conpty.dll has 0x4160 = the same first three plus GUARD_CF instead of the terminal-server bit. Characteristics are 0x0022 (exe) and 0x2022 (DLL | executable | large-address-aware). Subsystems are 3 (console) and 2 (GUI). In PE32+, the four size fields sit at optional-header offsets 72 (SizeOfStackReserve), 80 (SizeOfStackCommit), 88 (SizeOfHeapReserve), 96 (SizeOfHeapCommit) — all four 8-byte fields, all four reading 0x100000 reserve and 0x1000 commit in both samples. The spec notes CheckSum (offset 64) is validated at load time for drivers, boot-time DLLs, and DLLs loaded into critical Windows processes.

The certificate side: data directory index 4 stores file offsets, not RVAs — the spec is explicit that the certificate table "is not loaded into memory as part of the image", so its VirtualAddress field is a file pointer. Each entry is a WIN_CERTIFICATE: 4-byte dwLength, 2-byte wRevision (0x0100 v1, 0x0200 v2/current), 2-byte wCertificateType (0x0001 X509 unsupported, 0x0002 PKCS#7 SignedData = Authenticode), then the certificate bytes, padded so each entry ends on an 8-byte boundary. The Authenticode image hash deliberately excludes the CheckSum field and the Certificate Table directory entry, because adding a signature changes both.

Common mistake. Reading GUARD_CF here and concluding CFG protection is configured. This bit only advertises support — the actual CFG machinery (the check/dispatch pointers, the valid-target table, GuardFlags) lives in the Load Configuration structure, the next lesson. Also: CheckSum is not a signature. A non-zero checksum proves nothing about who wrote the file; only directory index 4 does that, and only if the signature chain itself verifies.

A Real Example

conpty.dll is signed. Its Security directory entry reads VirtualAddress = 0x18600, Size = 0x3D20 — and because this directory holds file offsets, the WIN_CERTIFICATE is exactly there, at the very end of the file (0x18600 + 0x3D20 = 0x1C320 = the file's 115488 bytes):

00018600: 203d 0000 0002 0200 3082 3d10 0609 2a86   =......0.=...*.
00018610: 4886 f70d 0107 02a0 823d 0130 823c fd02  H........=.0.<..
  1. 203d 0000 = dwLength 0x00003D20 — the entry's full length, equal to the directory Size because this file has one certificate.
  2. 0002 = wRevision 0x0200 — WIN_CERT_REVISION_2_0, the current structure version.
  3. 0200 = wCertificateType 0x0002 — WIN_CERT_TYPE_PKCS_SIGNED_DATA.
  4. Then 30 82 3d 10: a DER SEQUENCE of length 0x3D10, followed by 06 09 2a 86 48 86 f7 0d 01 07 02 — OID 1.2.840.113549.1.7.2, the PKCS#7 signedData content type. That is a genuine Authenticode blob, not padding.

Decode the flag fields of any PE in one pass — this is the same script that produced 0x4160 HIGH_ENTROPY_VA | DYNAMIC_BASE | NX_COMPAT | GUARD_CF above:

e_lfanew=0x100  COFF Characteristics @0x116 = 0x0022
Subsystem        @0x15C = 3   (WINDOWS_CUI)
DllCharacteristics @0x15E = 0x8160
  0x0020 | 0x0040 | 0x0100 | 0x8000

Try It

objdump already knows every bit name in the spec tables:

$ objdump -x cli-64.exe | grep -A6 "Characteristics"
Characteristics 0x22
	executable
	large address aware
...
Subsystem		00000003	(Windows CUI)
DllCharacteristics	00008160
					HIGH_ENTROPY_VA
					DYNAMIC_BASE
					NX_COMPAT
					TERMINAL_SERVICE_AWARE

Decode DllCharacteristics yourself, bit by bit:

$ python3 -c "
import struct
d=open('conpty.dll','rb').read()
e=struct.unpack_from('<I',d,0x3c)[0]
dll=struct.unpack_from('<H',d,e+24+70)[0]
t=[(0x20,'HIGH_ENTROPY_VA'),(0x40,'DYNAMIC_BASE'),(0x80,'FORCE_INTEGRITY'),
   (0x100,'NX_COMPAT'),(0x400,'NO_SEH'),(0x4000,'GUARD_CF'),(0x8000,'TSAWARE')]
print(hex(dll),[n for b,n in t if dll&b])"
0x4160 ['HIGH_ENTROPY_VA', 'DYNAMIC_BASE', 'NX_COMPAT', 'GUARD_CF']

Then confirm the signature offset: objdump -p conpty.dll | grep Security prints Entry 4 0000000000018600 00003d20 Security Directory — and remember, unlike every other entry around it, that 0x18600 is a file offset.

Check Your Understanding

Without looking back: decode cli-64.exe's DllCharacteristics value 0x8160.

Apply It

conpty.dll's Security directory entry reads VirtualAddress = 0x18600, Size = 0x3D20. What exactly is 0x18600?

The pattern: every data directory is an RVA/size pair except index 4. That single exception is why signature parsers that treat all sixteen directories alike walk straight off the end of the mapped image.

Connect

These flags are promises; the Load Configuration is where the hard parts of those promises are actually implemented — the /GS security cookie, the whitelist of valid SEH handlers, and the Control Flow Guard tables that the GUARD_CF bit only announces. It is also the first structure in the course whose size, not its position, tells you which version you are reading.

Next: The Load Configuration — cookies, handler lists, and CFG targets.