| From: opencoti |
| Subject: [PATCH 0007] build infra — compiler-driven header dependency tracking |
|
|
| Editing any llama.cpp/src (or common/, tools/server/) header never marked its |
| dependent objects out of date, so every header change forced a full |
| from-scratch rebuild — the single biggest drag on iterating the F4/F5 vendored |
| patches. |
|
|
| Root cause: the build tracks header deps via cosmopolitan's `mkdeps` |
| (build/deps.mk → o/$(MODE)/depend). mkdeps only resolves cosmopolitan's |
| root-relative include style; llama.cpp's sources use `#include "foo.h"` |
| resolved through `-iquote` dirs, which mkdeps cannot follow, so it emitted |
| ZERO edges for llama.cpp/src headers. Source->object tracking worked (the |
| pattern rule), but header->object did not. |
|
|
| Fix (compiler-driven deps, the modern standard): add `-MMD` to the COMPILE.c / |
| COMPILE.cc commands so cosmocc emits an accurate per-translation-unit |
| `<obj>.o.d` (project headers only — system headers excluded), then `-include` |
| those `.o.d` files in build/deps.mk. clang resolves the real `-iquote` |
| includes, so the header set is exact. cosmocc supports -MMD natively |
| (get_dependency_outputs in bin/cosmocc): the x86 `.o.d` lands next to the |
| object and an `.aarch64/*.o.d` twin under the arch subdir; only the x86 copy is |
| included (header sets are arch-independent). |
|
|
| Effect: a header edit now recompiles exactly the affected objects (both fat |
| arches, via the existing rule), and nothing else — no more scratch rebuilds. |
| The first build after this patch lands populates the .o.d set (a one-time full |
| compile); every build thereafter is correctly incremental. |
|
|
| Interaction with ccache: ccache wraps cosmocc's FAT driver, so a cache hit |
| restores only the x86 .o (never the .aarch64 twin). With accurate header |
| tracking, objects are no longer manually deleted to force recompiles, so the |
| .aarch64 twin always persists on disk in a warm tree and hits stay safe. The |
| remaining hazard — a warm cache restoring x86-only objects into a freshly |
| reset (o/-wiped) tree — is handled in the opencoti build pipeline, which runs |
| the full `build` with CCACHE_RECACHE=1. |
|
|
| Bench: docs/features/llamafile_build.md (incremental-rebuild verification: |
| touch an included header -> only its dependents recompile; touch a |
| non-dependency header -> no recompile) |
| Milestone: build infrastructure (supports F4/F5 — see docs/features/advanced_kv.md) |
| Upstreaming: build-only; no behavior change to the produced binary. Not |
| intended for upstream (cosmopolitan/llamafile use mkdeps by design); opencoti |
| carries it to make iterating the vendored patch series tractable. |
|
|
| |
| |
| |
| @@ -23,3 +23,15 @@ $(INCS): |
| rm -f o/$(MODE)/depend |
| |
| -include o/$(MODE)/depend |
| + |
| +# opencoti build-header-deps — see docs/features/llamafile_build.md |
| +# Pull in the per-object dependency files cosmocc emits under -MMD (added to |
| +# build/rules.mk COMPILE.*). mkdeps' o/$(MODE)/depend above cannot resolve |
| +# llama.cpp/src's -iquote includes, so it tracked zero of those headers and a |
| +# header edit never recompiled its dependents — every change forced a |
| +# from-scratch rebuild. The compiler-emitted `<obj>.o.d` files carry the exact |
| +# header set per translation unit. Only the x86 `.o.d` is pulled in; the |
| +# `.aarch64/*.o.d` twins list identical headers (arch-independent). The shell |
| +# find runs at parse time over prior-build outputs (none on a cold build → a |
| +# harmless empty include, everything compiles anyway). |
| +-include $(shell [ -d o/$(MODE) ] && find o/$(MODE) -name '*.o.d' -not -path '*/.aarch64/*' 2>/dev/null) |
| |
| |
| |
| @@ -6,8 +6,22 @@ |
| # |
| |
| LINK.o = $(CXX) $(CCFLAGS) $(LDFLAGS) |
| -COMPILE.c = $(CC) $(CCFLAGS) $(CFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -c |
| -COMPILE.cc = $(CXX) $(CCFLAGS) $(CXXFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -c |
| +# opencoti build-header-deps — see docs/features/llamafile_build.md |
| +# -MMD makes cosmocc emit a per-TU dependency file listing the real (project, |
| +# not system) header set; build/deps.mk -includes them so a header edit |
| +# recompiles exactly the affected objects. Without this, mkdeps' depend tracked |
| +# zero llama.cpp/src headers (it can't resolve -iquote includes), so any header |
| +# change forced a from-scratch rebuild. |
| +# |
| +# `-MF $@.d` is REQUIRED, not cosmetic: cosmocc defaults the dep file to |
| +# `<obj>.d` (it appends `.d` → e.g. foo.cpp.o.d), but ccache derives the |
| +# expected name from `-o foo.cpp.o` by stripping `.o` → foo.cpp.d. That |
| +# mismatch made ccache stat a file cosmocc never wrote ("failed to stat |
| +# …foo.cpp.d") and abort every compile with "ccache internal error" → nothing |
| +# was ever cached. Pinning -MF to cosmocc's actual name (`$@.d` = foo.cpp.o.d) |
| +# makes the two agree, so ccache caches and restores normally. |
| +COMPILE.c = $(CC) $(CCFLAGS) $(CFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -MMD -MF $@.d -c |
| +COMPILE.cc = $(CXX) $(CCFLAGS) $(CXXFLAGS) $(CPPFLAGS_) $(CPPFLAGS) $(TARGET_ARCH) -MMD -MF $@.d -c |
| |
| # |
| # Standard Compilation Rules |
|
|
