From: opencoti
Subject: [PATCH 0007] build infra — compiler-driven header dependency tracking

Editing any llama.cpp/src (or common/, tools/server/) header never marked its
dependent objects out of date, so every header change forced a full
from-scratch rebuild — the single biggest drag on iterating the F4/F5 vendored
patches.

Root cause: the build tracks header deps via cosmopolitan's `mkdeps`
(build/deps.mk → o/$(MODE)/depend). mkdeps only resolves cosmopolitan's
root-relative include style; llama.cpp's sources use `#include "foo.h"`
resolved through `-iquote` dirs, which mkdeps cannot follow, so it emitted
ZERO edges for llama.cpp/src headers. Source->object tracking worked (the
pattern rule), but header->object did not.

Fix (compiler-driven deps, the modern standard): add `-MMD` to the COMPILE.c /
COMPILE.cc commands so cosmocc emits an accurate per-translation-unit
`<obj>.o.d` (project headers only — system headers excluded), then `-include`
those `.o.d` files in build/deps.mk. clang resolves the real `-iquote`
includes, so the header set is exact. cosmocc supports -MMD natively
(get_dependency_outputs in bin/cosmocc): the x86 `.o.d` lands next to the
object and an `.aarch64/*.o.d` twin under the arch subdir; only the x86 copy is
included (header sets are arch-independent).

Effect: a header edit now recompiles exactly the affected objects (both fat
arches, via the existing rule), and nothing else — no more scratch rebuilds.
The first build after this patch lands populates the .o.d set (a one-time full
compile); every build thereafter is correctly incremental.

Interaction with ccache: ccache wraps cosmocc's FAT driver, so a cache hit
restores only the x86 .o (never the .aarch64 twin). With accurate header
tracking, objects are no longer manually deleted to force recompiles, so the
.aarch64 twin always persists on disk in a warm tree and hits stay safe. The
remaining hazard — a warm cache restoring x86-only objects into a freshly
reset (o/-wiped) tree — is handled in the opencoti build pipeline, which runs
the full `build` with CCACHE_RECACHE=1.

Bench: docs/features/llamafile_build.md (incremental-rebuild verification:
touch an included header -> only its dependents recompile; touch a
non-dependency header -> no recompile)
Milestone: build infrastructure (supports F4/F5 — see docs/features/advanced_kv.md)
Upstreaming: build-only; no behavior change to the produced binary. Not
intended for upstream (cosmopolitan/llamafile use mkdeps by design); opencoti
carries it to make iterating the vendored patch series tractable.

diff --git a/build/deps.mk b/build/deps.mk
--- a/build/deps.mk
+++ b/build/deps.mk
@@ -23,3 +23,15 @@ $(INCS):
 	rm -f o/$(MODE)/depend
 
 -include o/$(MODE)/depend
+
+# opencoti build-header-deps — see docs/features/llamafile_build.md
+# Pull in the per-object dependency files cosmocc emits under -MMD (added to
+# build/rules.mk COMPILE.*). mkdeps' o/$(MODE)/depend above cannot resolve
+# llama.cpp/src's -iquote includes, so it tracked zero of those headers and a
+# header edit never recompiled its dependents — every change forced a
+# from-scratch rebuild. The compiler-emitted `<obj>.o.d` files carry the exact
+# header set per translation unit. Only the x86 `.o.d` is pulled in; the
+# `.aarch64/*.o.d` twins list identical headers (arch-independent). The shell
+# find runs at parse time over prior-build outputs (none on a cold build → a
+# harmless empty include, everything compiles anyway).
+-include $(shell [ -d o/$(MODE) ] && find o/$(MODE) -name '*.o.d' -not -path '*/.aarch64/*' 2>/dev/null)
diff --git a/build/rules.mk b/build/rules.mk
--- a/build/rules.mk
+++ b/build/rules.mk
@@ -6,8 +6,22 @@
 # ==============================================================================
 
 LINK.o = $(CXX) $(CCFLAGS) $(LDFLAGS)
-COMPILE.c = $(CC) $(CCFLAGS) $(CFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -c
-COMPILE.cc = $(CXX) $(CCFLAGS) $(CXXFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -c
+# opencoti build-header-deps — see docs/features/llamafile_build.md
+# -MMD makes cosmocc emit a per-TU dependency file listing the real (project,
+# not system) header set; build/deps.mk -includes them so a header edit
+# recompiles exactly the affected objects. Without this, mkdeps' depend tracked
+# zero llama.cpp/src headers (it can't resolve -iquote includes), so any header
+# change forced a from-scratch rebuild.
+#
+# `-MF $@.d` is REQUIRED, not cosmetic: cosmocc defaults the dep file to
+# `<obj>.d` (it appends `.d` → e.g. foo.cpp.o.d), but ccache derives the
+# expected name from `-o foo.cpp.o` by stripping `.o` → foo.cpp.d. That
+# mismatch made ccache stat a file cosmocc never wrote ("failed to stat
+# …foo.cpp.d") and abort every compile with "ccache internal error" → nothing
+# was ever cached. Pinning -MF to cosmocc's actual name (`$@.d` = foo.cpp.o.d)
+# makes the two agree, so ccache caches and restores normally.
+COMPILE.c = $(CC) $(CCFLAGS) $(CFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -MMD -MF $@.d -c
+COMPILE.cc = $(CXX) $(CCFLAGS) $(CXXFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -MMD -MF $@.d -c
 
 # ==============================================================================
 # Standard Compilation Rules