diff --git a/CHANGELOG.md b/CHANGELOG.md index f3183b4..175ed30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +- Optional GGUF load-time hook: when `sidecarEntangle` is on and a known safe Python loader sits beside the model, stripping the `bla.sidecar.v1` metadata note or a sibling sidecar fails closed at import. Tensor bytes stay bit-exact. Default remains OFF. +- REA catalog hygiene on tip `c84fde05e` (honesty only; no new 1:1 densify counters): `FDO_ALLOWANCE`, `BUNDLE_INFO_PLIST_INPUT_BYTES`, `INFO_PLIST_DECODE_MIB`, `MAX_WASM_SECTIONS`. + ## [3.5.1] — 2026-11-18 Optional `.pyc` import-time hook: when `sidecarEntangle` is on and a known safe Python loader sits beside the module, stripping the marshal note or a sibling sidecar fails closed at import. Default remains OFF. diff --git a/NIGHTLY.md b/NIGHTLY.md index 6788fb8..6377951 100644 --- a/NIGHTLY.md +++ b/NIGHTLY.md @@ -1,5 +1,65 @@ # BLA nightly development log +## 2026-11-19 ~America/Los_Angeles / ~2026-11-20 02:00 Asia/Shanghai + +### REA delta +- Pulled `morluto/rea`: `31aaec6b4` → `c84fde05e` (**+345** commits). Dirty + `third_party/*` submodule checkouts were left untouched. +- No new 1:1 trip-able densify constants. The delta is mostly fix / refactor / + test / perf (230 fix / 41 refactor / 22 test / 19 perf / 14 docs / 9 ci / + 10 feat). Feat commits add inspection capability, not plantable densify + targets: FDO packaging/dlopen notes (#7237), Ghidra prototype conflicts + (#8161), MIPS32 PIC shared images, RISC-V initializers / PIC, cargo-auditable + dependencies from WASM (#5069), DOS COM import, Ghidra 12.2, EM_PPC64 + classification. New or newly advertised ceilings are admission / decode / + format-validation caps, not plantable without breaking the artifact: + - `FDO_ALLOWANCE` = 64 KiB (`ElfFdoNote.ts`, #7237) — shared per-report JSON + decode cap for ELF FDO packaging / dlopen notes; overflow is left + undecoded, and planting dense 64 KiB notes would bloat the ELF. + - `BUNDLE_INFO_PLIST_INPUT_BYTES` = 64 MiB and `INFO_PLIST_DECODE_MIB` = 16 + (`AppBundleExecutable.ts`) — Info.plist admission / decode caps; a 64 MiB + property list would bloat or break the app bundle. + - `MAX_WASM_SECTIONS` = 65,536 (`RustWasmImage.ts`, #5069) — cargo-auditable + WASM walk cap; planting 65,536 sections would bloat or break the module. + - Also noted, not catalogued as new trip targets: `MIPS_GNU_ATTRIBUTES_MAX_BYTES` + (4096, format validation), PE resource `METADATA_BYTES` (8 MiB), + `ASN1_DEPTH_BUDGET` (64, Authenticode nesting), `GUARDIAN_READINESS_BYTES` + (4096, JADX workspace). `MAX_VENDOR_HEADER_SIZE` (1 MiB) was already + catalogued as `ANDROID_BOOT_MAX_HEADER_SIZE`. +- Catalogued the new headline maxima in `REA_INTERNAL_LIMITS` (honesty only). + +### Done tonight +- Landed the GGUF load-time hook (`generateSidecarGgufLoaderSnippet` / + `prependGgufSidecarLoaderHook` / `entangleGgufLoaderWithSidecars` / + `computeGgufSidecarCodeIdentity`; marker `_installGgufSidecarLoaderHook`; + `protectTarget` + `hardenProject` post-pass). Auto-wired only into a + same-directory Python loader named `loader.py`, `__init__.py`, or + `__main__.py` that already mentions the `.gguf` basename + (`selectPythonGgufLoader` / `discoverPythonGgufLoader` / + `isSafePythonGgufLoader` / `isGgufSidecarLoaderHookWired` / + `sourceMentionsGgufFile`). The hook does not rewrite the `.gguf` (tensor + infos, relative tensor offsets, and tensor bytes stay bit-exact; the + existing alignment-safe `bla.sidecar.v1` STRING metadata note remains the + only GGUF mutation). Test / already-entangled / self-seal / unknown helper + files are never hooked. Vector `gguf-sidecar-entangle` is recorded and + `gguf-sidecar-host-verify-only` is cleared only when that hook was actually + written. Default OFF via existing `sidecarEntangle`. +- Live BLA tree was pulled `4bb09a6` / `3.5.1` → `1d4d490` (still `3.5.1`, + unreleased daytime work already on main: WASM Code-section opaque flower + bytecode, decoy-tell scrub, rare-CJK BPE amplification). +- Gates: build OK, tests **1296 passed / 5 skipped**, benchmark **14/14** + runtime-equivalent. + +### Next step (following night) +1. Safetensors / DEX / SQLite load-time hooks (same honesty bar: only record + the vector / clear `*-host-verify-only` when a real load-time hook was + written). Safetensors Python loaders are the closest remaining analogue. +2. Browser WASM loader auto-wire (the hook still needs Node `fs` / + `crypto`; a browser snippet would have to verify without those). +3. ESM hook that runs before hoisted *user* imports; MCP one-click + polish; empirical JADX / Android red-team; ESP #8228; Dependabot majors. + + ## 2026-11-18 ~America/Los_Angeles / ~2026-11-19 02:00 Asia/Shanghai ### REA delta diff --git a/docs/design/sidecar-entry-entanglement.md b/docs/design/sidecar-entry-entanglement.md index 7146d47..189b6d6 100644 --- a/docs/design/sidecar-entry-entanglement.md +++ b/docs/design/sidecar-entry-entanglement.md @@ -1,6 +1,6 @@ # DESIGN: Sidecar / entry entanglement (issue #40 / STATE) -Status: **JS + Python + WASM + Bun companion + ASAR archive-note + ASAR main JS preamble strip⇒fail; .pyc marshal-note + import-time hook (when auto-wired); GGUF STRING metadata note is host-verify only** — binding digest + JS/Python preambles + WASM `bla.sidecar.v1` custom-section note + Bun sibling `.bla-sidecar.v1.json` companion note (byte-exact binary) + ASAR in-archive `.bla/sidecar.v1.json` note + optional Electron main JS preamble (`generateSidecarAsarMainJsPreamble`) + CPython `.pyc` TYPE_ASCII marshal trailer (`generateSidecarPycMarshalNote` / `verifyPycSidecarMarshalNote`, importlib-ignored) + GGUF `bla.sidecar.v1` STRING metadata KV (`generateSidecarGgufMetadataNote` / `verifyGgufSidecarMetadataNote`, alignment-safe rewrite keeps tensor bytes bit-exact; loaders ignore unknown STRING keys) + protect/harden opt-in landed; defenseScore gap when `sidecarEntangle` is off landed; GGUF / Safetensors / DEX / SQLite host-only notes annotate `gguf-sidecar-host-verify-only` / `safetensors-sidecar-host-verify-only` / `dex-sidecar-host-verify-only` / `sqlite-sidecar-host-verify-only` and must not record the process-start vectors; `.pyc` annotates `pyc-sidecar-host-verify-only` until the import-time hook is actually written; optional JS-loader hook `generateSidecarWasmLoaderSnippet` / `prependWasmSidecarLoaderHook` / `entangleWasmLoaderWithSidecars` landed 2026-11-15, auto-wired 2026-11-16 into a same-directory Node CJS loader, and extended 2026-11-17 to a same-directory Node ESM loader with Node-builtin-only static imports (`detectWasmJsLoaderCarrier` / `esmStaticImportsAreNodeBuiltinsOnly`; only then record `wasm-sidecar-entangle` / clear `wasm-sidecar-host-verify-only`); **2026-11-16 honesty:** ESM static `import` / `export-from` of user modules evaluate before the prepended JS IIFE — annotate `esm-import-hoist-before-hooks` (soft-cap 93); do not treat entry-body fail-closed as process-start for the hoisted graph. **2026-11-17 honesty:** Node builtin-only imports are not a user hoisted graph. Optional `.pyc` import-time hook `generateSidecarPycLoaderSnippet` / `prependPycSidecarLoaderHook` / `entanglePycLoaderWithSidecars` landed 2026-11-18, auto-wired into a same-directory (or parent-of-`__pycache__`) Python loader named `loader.py` / `__init__.py` / `__main__.py` that already mentions the module (`selectPythonPycLoader` / `discoverPythonPycLoader` / `isSafePythonPycLoader`; only then record `pyc-sidecar-entangle` / clear `pyc-sidecar-host-verify-only`). The hook does not rewrite the `.pyc`. Remaining: GGUF/DEX/Safetensors/SQLite load-time hooks; browser WASM loader auto-wire; ESM hook that runs before hoisted user imports. +Status: **JS + Python + WASM + Bun companion + ASAR archive-note + ASAR main JS preamble strip⇒fail; .pyc marshal-note + import-time hook (when auto-wired); GGUF STRING metadata note + load-time hook (when auto-wired)** — binding digest + JS/Python preambles + WASM `bla.sidecar.v1` custom-section note + Bun sibling `.bla-sidecar.v1.json` companion note (byte-exact binary) + ASAR in-archive `.bla/sidecar.v1.json` note + optional Electron main JS preamble (`generateSidecarAsarMainJsPreamble`) + CPython `.pyc` TYPE_ASCII marshal trailer (`generateSidecarPycMarshalNote` / `verifyPycSidecarMarshalNote`, importlib-ignored) + GGUF `bla.sidecar.v1` STRING metadata KV (`generateSidecarGgufMetadataNote` / `verifyGgufSidecarMetadataNote`, alignment-safe rewrite keeps tensor bytes bit-exact; loaders ignore unknown STRING keys) + protect/harden opt-in landed; defenseScore gap when `sidecarEntangle` is off landed; Safetensors / DEX / SQLite host-only notes annotate `safetensors-sidecar-host-verify-only` / `dex-sidecar-host-verify-only` / `sqlite-sidecar-host-verify-only` and must not record the process-start vectors; `.pyc` annotates `pyc-sidecar-host-verify-only` until the import-time hook is actually written; GGUF annotates `gguf-sidecar-host-verify-only` until the load-time hook is actually written; optional JS-loader hook `generateSidecarWasmLoaderSnippet` / `prependWasmSidecarLoaderHook` / `entangleWasmLoaderWithSidecars` landed 2026-11-15, auto-wired 2026-11-16 into a same-directory Node CJS loader, and extended 2026-11-17 to a same-directory Node ESM loader with Node-builtin-only static imports (`detectWasmJsLoaderCarrier` / `esmStaticImportsAreNodeBuiltinsOnly`; only then record `wasm-sidecar-entangle` / clear `wasm-sidecar-host-verify-only`); **2026-11-16 honesty:** ESM static `import` / `export-from` of user modules evaluate before the prepended JS IIFE — annotate `esm-import-hoist-before-hooks` (soft-cap 93); do not treat entry-body fail-closed as process-start for the hoisted graph. **2026-11-17 honesty:** Node builtin-only imports are not a user hoisted graph. Optional `.pyc` import-time hook `generateSidecarPycLoaderSnippet` / `prependPycSidecarLoaderHook` / `entanglePycLoaderWithSidecars` landed 2026-11-18, auto-wired into a same-directory (or parent-of-`__pycache__`) Python loader named `loader.py` / `__init__.py` / `__main__.py` that already mentions the module (`selectPythonPycLoader` / `discoverPythonPycLoader` / `isSafePythonPycLoader`; only then record `pyc-sidecar-entangle` / clear `pyc-sidecar-host-verify-only`). The hook does not rewrite the `.pyc`. Optional GGUF load-time hook `generateSidecarGgufLoaderSnippet` / `prependGgufSidecarLoaderHook` / `entangleGgufLoaderWithSidecars` landed 2026-11-19, auto-wired into a same-directory Python loader named `loader.py` / `__init__.py` / `__main__.py` that already mentions the `.gguf` basename (`selectPythonGgufLoader` / `discoverPythonGgufLoader` / `isSafePythonGgufLoader`; only then record `gguf-sidecar-entangle` / clear `gguf-sidecar-host-verify-only`). The hook does not rewrite tensor bytes, tensor infos, or relative offsets. Remaining: Safetensors/DEX/SQLite load-time hooks; browser WASM loader auto-wire; ESM hook that runs before hoisted user imports. ## Goal @@ -47,9 +47,9 @@ checks beat orphan honeypot files. - ~~Extend to ASAR archive-note companion~~ — 2026-11-09 nightly: `generateSidecarAsarArchiveNote` / `bindAsarSidecarEntanglement` / `entangleAsarEntryWithSidecars` / `verifyAsarSidecarArchiveNote`; protect + harden ASAR post-pass plant `.bla/sidecar.v1.json` binding `enclave/hardware-tee-bridge.js`. - ~~ASAR main JS preamble (Electron fs-patched re-measure)~~ — 2026-11-10 nightly: `generateSidecarAsarMainJsPreamble` / `selectAsarMainJsMember` / `entangleAsarMainJsWithSidecars`; protect + harden prepend strip⇒fail preamble to package.json#main (or heuristic entry). - ~~CPython `.pyc` marshal trailer note (host verify)~~ — 2026-11-11 nightly: `generateSidecarPycMarshalNote` / `bindPycSidecarEntanglement` / `entanglePycEntryWithSidecars` / `verifyPycSidecarMarshalNote`; TYPE_ASCII trailer after root (importlib-safe). ~~Import-time hook~~ — 2026-11-18 nightly: `generateSidecarPycLoaderSnippet` / `prependPycSidecarLoaderHook` / `entanglePycLoaderWithSidecars`; auto-wired into `loader.py` / `__init__.py` / `__main__.py` (never the `.py` source of the `.pyc`). **Honesty:** record `pyc-sidecar-entangle` / clear `pyc-sidecar-host-verify-only` only when the hook was actually written. Default OFF. -- ~~GGUF STRING metadata note (host verify)~~ — 2026-11-12 nightly: `generateSidecarGgufMetadataNote` / `bindGgufSidecarEntanglement` / `entangleGgufEntryWithSidecars` / `verifyGgufSidecarMetadataNote`; `bla.sidecar.v1` STRING KV via alignment-safe `rebuildGguf` (tensor bytes bit-exact). **Honesty:** do not record `gguf-sidecar-entangle` until a load-time hook lands; defenseScore gap `gguf-sidecar-host-verify-only`. Also fixed `protectGgufBuffer` to use the same rewriter so tensor-bearing GGUFs no longer silently shift the data section. Default OFF. Remaining: GGUF/DEX/Safetensors/SQLite load-time hooks. +- ~~GGUF STRING metadata note (host verify)~~ — 2026-11-12 nightly: `generateSidecarGgufMetadataNote` / `bindGgufSidecarEntanglement` / `entangleGgufEntryWithSidecars` / `verifyGgufSidecarMetadataNote`; `bla.sidecar.v1` STRING KV via alignment-safe `rebuildGguf` (tensor bytes bit-exact). Also fixed `protectGgufBuffer` to use the same rewriter so tensor-bearing GGUFs no longer silently shift the data section. ~~Load-time hook~~ — 2026-11-19 nightly: `generateSidecarGgufLoaderSnippet` / `prependGgufSidecarLoaderHook` / `entangleGgufLoaderWithSidecars`; auto-wired into a same-directory `loader.py` / `__init__.py` / `__main__.py` that already mentions the `.gguf` basename. **Honesty:** record `gguf-sidecar-entangle` / clear `gguf-sidecar-host-verify-only` only when the hook was actually written. The hook does not rewrite tensor bytes, tensor infos, or relative offsets. Default OFF. - ~~JS runtime carrier honesty~~ — 2026-11-13: `.mjs`, `import`/`export`, top-level `await`, `import.meta`, and `.js` under nearest `package.json` `"type": "module"` are detected via Babel `program.sourceType` + `refineJsCarrierForRuntimePaths` and now receive an ESM-native strip⇒fail preamble (`import { createRequire } from "node:module"` + `import.meta.url`), with matching ESM `self-seal` support. - **ESM import-hoist honesty** — 2026-11-16: the ESM-native preamble fail-closes the entry body, but static `import` / `export-from` bindings are instantiated first, so imported module side effects still run after sidecar strip or hook failure (`jsModuleHasHoistedDependencyGraph`; the BLA `createRequire` preamble import is ignored). Annotate defenseScore gap `esm-import-hoist-before-hooks` (soft-cap 93) when sidecar or self-seal is on. CJS `require()` after the preamble remains process-start. A true pre-import hook (custom ESM loader / `module.register` before the graph loads) is still TBD. - ~~Safetensors `__metadata__` note (host verify)~~ — 2026-11-13: `generateSidecarSafetensorsMetadataNote` / `bindSafetensorsSidecarEntanglement` / `entangleSafetensorsEntryWithSidecars` / `verifySafetensorsSidecarMetadataNote`; `bla.sidecar.v1` `__metadata__` key keeps tensor bytes bit-exact; `defenseScore` annotates `safetensors-sidecar-host-verify-only` and `wasm-sidecar-host-verify-only` (soft-cap 93). - ~~DEX string-pool note (host verify)~~ — 2026-11-13 nightly: `generateSidecarDexStringPoolNote` / `bindDexSidecarEntanglement` / `entangleDexEntryWithSidecars` / `verifyDexSidecarStringPoolNote`; unreferenced sort-last `bla.sidecar.v1=` pool string via `parseDexLayout` / `rebuildDex` (existing string indices + instruction bytes unchanged; trailers preserved). **Honesty:** do not record `dex-sidecar-entangle` until a load-time hook lands; defenseScore gap `dex-sidecar-host-verify-only`. Default OFF. -- ~~SQLite schema-comment / reserved-VIEW note (host verify)~~ — 2026-11-14 nightly: `generateSidecarSqliteSchemaComment` / `bindSqliteSidecarEntanglement` / `entangleSqliteEntryWithSidecars` / `verifySqliteSidecarSchemaComment`; unused `bla_sidecar_v1` VIEW (`SELECT '' AS "bla.sidecar.v1" WHERE 0`) keeps user tables, rows, query results, WAL/FK, and application_id/user_version unchanged. **Honesty:** do not record `sqlite-sidecar-entangle` until a load-time hook lands; defenseScore gap `sqlite-sidecar-host-verify-only`. Default OFF. Remaining: GGUF/DEX/Safetensors/SQLite load-time hooks. +- ~~SQLite schema-comment / reserved-VIEW note (host verify)~~ — 2026-11-14 nightly: `generateSidecarSqliteSchemaComment` / `bindSqliteSidecarEntanglement` / `entangleSqliteEntryWithSidecars` / `verifySqliteSidecarSchemaComment`; unused `bla_sidecar_v1` VIEW (`SELECT '' AS "bla.sidecar.v1" WHERE 0`) keeps user tables, rows, query results, WAL/FK, and application_id/user_version unchanged. **Honesty:** do not record `sqlite-sidecar-entangle` until a load-time hook lands; defenseScore gap `sqlite-sidecar-host-verify-only`. Default OFF. Remaining: Safetensors/DEX/SQLite load-time hooks. diff --git a/src/core/types.ts b/src/core/types.ts index bda7c4a..e753372 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -289,6 +289,8 @@ export interface ProtectionResult { readonly wasmSidecarLoaderHookApplied?: boolean; /** Present when a same-directory Python loader was auto-wired with the `.pyc` import-time hook. */ readonly pycSidecarLoaderHookApplied?: boolean; + /** Present when a same-directory Python loader was auto-wired with the GGUF load-time hook. */ + readonly ggufSidecarLoaderHookApplied?: boolean; /** Present when WASM export rename actually rewrote at least one non-preserved export (#40.3). */ readonly wasmExportRenameApplied?: boolean; /** diff --git a/src/engine/defense-score-honesty.ts b/src/engine/defense-score-honesty.ts index 6b6f607..c6d62fa 100644 --- a/src/engine/defense-score-honesty.ts +++ b/src/engine/defense-score-honesty.ts @@ -67,6 +67,12 @@ export interface DefenseScoreHonestyInput { * the .pyc). Clears `pyc-sidecar-host-verify-only`. */ readonly pycSidecarLoaderHookApplied?: boolean; + /** + * True when a same-directory Python loader was auto-wired with + * `entangleGgufLoaderWithSidecars` (or the hook is already present next to + * the .gguf). Clears `gguf-sidecar-host-verify-only`. + */ + readonly ggufSidecarLoaderHookApplied?: boolean; /** * How a SQLite database's generated decoy columns are defined: `"eval"` * when they call the function no stock connection has (opt-in), otherwise @@ -170,12 +176,18 @@ export function collectDefenseScoreGaps( }); } - // GGUF STRING KV is host-verify only (loaders ignore unknown metadata keys). - if (input.detectedFormat === "model-gguf" && input.sidecarEntangleEnabled === true) { + // GGUF STRING KV is host-verify only (loaders ignore unknown metadata keys) + // until a same-directory Python load-time hook is actually auto-wired (or + // already present) beside the model. + if ( + input.detectedFormat === "model-gguf" && + input.sidecarEntangleEnabled === true && + input.ggufSidecarLoaderHookApplied !== true + ) { gaps.push({ id: "gguf-sidecar-host-verify-only", summary: - "sidecarEntangle planted a GGUF bla.sidecar.v1 STRING metadata note verified only by host verifyGgufSidecarMetadataNote; GGUF loaders still load after strip or sibling delete. Load-time hook TBD — do not treat this as process-start strip⇒fail (issue #40 / DESIGN).", + "sidecarEntangle planted a GGUF bla.sidecar.v1 STRING metadata note verified by host verifyGgufSidecarMetadataNote; GGUF loaders still load after strip or sibling delete unless a same-directory Python loader (loader.py / __init__.py / __main__.py) was auto-wired with the load-time hook (issue #40 / DESIGN).", scoreCap: 93, }); } diff --git a/src/engine/harden.ts b/src/engine/harden.ts index 0e67c49..da11f6c 100644 --- a/src/engine/harden.ts +++ b/src/engine/harden.ts @@ -95,6 +95,13 @@ import { extractPycSidecarMarshalNote, relativePycPathFromLoader, PYC_SIDECAR_LOADER_MARKER, + generateSidecarGgufLoaderSnippet, + prependGgufSidecarLoaderHook, + selectPythonGgufLoader, + computeGgufSidecarCodeIdentity, + extractGgufSidecarMetadataNote, + relativeGgufPathFromLoader, + GGUF_SIDECAR_LOADER_MARKER, verifyAsarSidecarArchiveNote, } from "../shields/runtime/sidecar-entangle.js"; import { packAsarBuffer, unpackAsarBuffer } from "../shields/formats/asar.js"; @@ -3102,6 +3109,114 @@ function hardenProjectCore( // protectTarget honesty; the loader rewrite above is the act. } + const ggufPlantedAbs = new Set(); + const ggufLikeEarly = bindableSummaries.filter( + (s) => + s.success && + (s.format === "model-gguf" || + /\.gguf$/i.test(s.relativePath.replaceAll("\\", "/"))), + ); + for (const ggufSum of ggufLikeEarly) { + const ggufAbs = join(outputRoot, ggufSum.relativePath); + if (!existsSync(ggufAbs)) continue; + const ggufDir = dirname(ggufAbs); + const searchDirs = [ggufDir]; + const loaderCandidates: { + name: string; + source: string; + absPath: string; + }[] = []; + for (const s of pyLike) { + const outAbs = join(outputRoot, s.relativePath); + if (!searchDirs.includes(dirname(outAbs))) continue; + if (!isPythonPycLoaderFilename(s.relativePath)) continue; + let predicateSource: string | null = null; + try { + if (s.inputPath && existsSync(s.inputPath)) { + predicateSource = readFileSync(s.inputPath, "utf8"); + } + } catch { + predicateSource = null; + } + if (predicateSource === null && existsSync(outAbs)) { + try { + predicateSource = readFileSync(outAbs, "utf8"); + } catch { + predicateSource = null; + } + } + if (predicateSource === null) continue; + loaderCandidates.push({ + name: basename(outAbs), + source: predicateSource, + absPath: outAbs, + }); + } + const selectedName = selectPythonGgufLoader( + basename(ggufAbs), + loaderCandidates, + ); + const found = selectedName + ? loaderCandidates.find((c) => c.name === selectedName) + : undefined; + if (!found) continue; + const alreadyNoted = + extractGgufSidecarMetadataNote(readFileSync(ggufAbs)) !== null; + const otherPrimary = + (jsEntryAbs !== undefined && found.absPath !== jsEntryAbs) || + (pyEntryAbs !== undefined && found.absPath !== pyEntryAbs); + // A foreign JS/Python entry may already bake a digest of this + // model. Hooking without a note would fail-closed at import; + // planting a new metadata KV would break that digest. + if (otherPrimary && !alreadyNoted) continue; + + if (!otherPrimary) { + const sidecars: { pathLabel: string; bytes: Buffer }[] = []; + for (const s of bindableSummaries) { + if (!s.success || s.relativePath === ggufSum.relativePath) continue; + const abs = join(outputRoot, s.relativePath); + if (jsEntryAbs && abs === jsEntryAbs) continue; + if (pyEntryAbs && abs === pyEntryAbs) continue; + if (abs === found.absPath) continue; + const rel = relative(ggufDir, abs).split(sep).join("/"); + if (rel.length === 0 || rel.startsWith("..") || !existsSync(abs)) continue; + sidecars.push({ pathLabel: rel, bytes: readFileSync(abs) }); + } + const bound = entangleGgufEntryWithSidecars({ + entryPath: basename(ggufAbs), + entryBytes: readFileSync(ggufAbs), + sidecars, + seedTag: derivePublicSidecarSeedTag(String(options.seed ?? runId)), + allowEmptySidecars: true, + }); + if (bound) { + writeFileAtomicSync(ggufAbs, bound.bytes, undefined, { + allowLinkWriteThrough: false, + }); + ggufPlantedAbs.add(ggufAbs); + } + } + + const identity = computeGgufSidecarCodeIdentity(readFileSync(ggufAbs)); + const existing = readFileSync(found.absPath, "utf8"); + const already = existing.includes(GGUF_SIDECAR_LOADER_MARKER); + const ggufRel = relativeGgufPathFromLoader(found.absPath, ggufAbs); + const inject = already + ? null + : generateSidecarGgufLoaderSnippet({ + codeIdentityHex: identity, + ggufRelPaths: [ggufRel], + }); + const next = inject + ? prependGgufSidecarLoaderHook(existing, inject.preamble) + : existing; + if (next !== existing) { + writeFileAtomicSync(found.absPath, next, "utf8", { + allowLinkWriteThrough: false, + }); + } + } + if (jsEntry) { const entryAbs = join(outputRoot, jsEntry.relativePath); if (existsSync(entryAbs)) { @@ -3289,7 +3404,7 @@ function hardenProjectCore( : undefined; if (ggufEntry) { const entryAbs = join(outputRoot, ggufEntry.relativePath); - if (existsSync(entryAbs)) { + if (existsSync(entryAbs) && !ggufPlantedAbs.has(entryAbs)) { const entryDir = dirname(entryAbs); const sidecars: { pathLabel: string; bytes: Buffer }[] = []; for (const s of bindableSummaries) { diff --git a/src/engine/protector.ts b/src/engine/protector.ts index c9447c7..ad5cbc0 100644 --- a/src/engine/protector.ts +++ b/src/engine/protector.ts @@ -116,6 +116,11 @@ import { listPycLoaderSearchDirectories, relativePycPathFromLoader, PYC_SIDECAR_LOADER_MARKER, + entangleGgufLoaderWithSidecars, + discoverPythonGgufLoader, + listGgufLoaderSearchDirectories, + relativeGgufPathFromLoader, + GGUF_SIDECAR_LOADER_MARKER, } from "../shields/runtime/sidecar-entangle.js"; /** @@ -455,6 +460,59 @@ function applySidecarEntangleToGguf( return { buffer: bound.bytes, applied: true }; } +function applySidecarGgufLoaderAutoWire(options: { + readonly ggufBytes: Uint8Array; + readonly ggufBasenames: readonly string[]; + readonly outputDir: string; + readonly sourceDir: string; + readonly outputGgufPath: string; +}): { applied: boolean; destPath?: string } { + const search: string[] = []; + const seen = new Set(); + const addDir = (dir: string): void => { + if (!dir || seen.has(dir)) return; + seen.add(dir); + search.push(dir); + }; + addDir(options.outputDir); + if (options.sourceDir !== options.outputDir) addDir(options.sourceDir); + for (const extra of listGgufLoaderSearchDirectories(options.outputGgufPath)) { + addDir(extra); + } + + const names = [...new Set(options.ggufBasenames.filter((n) => /\.gguf$/i.test(n)))]; + let found: ReturnType = null; + for (const ggufBasename of names) { + found = discoverPythonGgufLoader({ + ggufBasename, + searchDirectories: search, + }); + if (found) break; + } + if (!found) return { applied: false }; + let destPath = found.absPath; + const foundDir = dirname(found.absPath); + if (foundDir !== options.outputDir && options.sourceDir !== options.outputDir) { + destPath = join(options.outputDir, found.name); + if (existsSync(destPath) && destPath !== found.absPath) { + return { applied: false }; + } + } + const ggufRel = relativeGgufPathFromLoader(destPath, options.outputGgufPath); + const entangled = entangleGgufLoaderWithSidecars({ + loaderSource: found.source, + ggufBytes: options.ggufBytes, + ggufRelPath: ggufRel, + }); + if (entangled.code === found.source && destPath === found.absPath) { + return found.source.includes(GGUF_SIDECAR_LOADER_MARKER) + ? { applied: true, destPath } + : { applied: false }; + } + writeFileAtomicSync(destPath, entangled.code, "utf8"); + return { applied: true, destPath }; +} + function applySidecarEntangleToSafetensors( safetensorsBytes: Buffer, options: ProtectionOptions, @@ -799,6 +857,7 @@ function protectTargetInner(options: ProtectionOptions): ProtectionResult { let sidecarEntangleApplied = false; let wasmSidecarLoaderHookApplied = false; let pycSidecarLoaderHookApplied = false; + let ggufSidecarLoaderHookApplied = false; const counteredVectors = new Set(); if ( @@ -1446,10 +1505,17 @@ function protectTargetInner(options: ProtectionOptions): ProtectionResult { finalGguf = entangled.buffer; if (entangled.applied) { sidecarEntangleApplied = true; - // Host-verify STRING KV only: GGUF loaders ignore unknown metadata - // keys, so strip / sibling delete still load. Do not record - // gguf-sidecar-entangle until a load-time hook lands (same honesty - // bar as .pyc marshal trailer / ASAR archive-note-only). + const hook = applySidecarGgufLoaderAutoWire({ + ggufBytes: finalGguf, + ggufBasenames: [basename(absInput), basename(absOutput)], + outputDir: dirname(absOutput), + sourceDir: dirname(absInput), + outputGgufPath: absOutput, + }); + if (hook.applied) { + ggufSidecarLoaderHookApplied = true; + counteredVectors.add("gguf-sidecar-entangle"); + } } } writeProtectedOutput(absOutput, finalGguf); @@ -1720,6 +1786,7 @@ function protectTargetInner(options: ProtectionOptions): ProtectionResult { ...(sidecarEntangleApplied ? { sidecarEntangle: true as const } : {}), ...(wasmSidecarLoaderHookApplied ? { wasmSidecarLoaderHookApplied: true as const } : {}), ...(pycSidecarLoaderHookApplied ? { pycSidecarLoaderHookApplied: true as const } : {}), + ...(ggufSidecarLoaderHookApplied ? { ggufSidecarLoaderHookApplied: true as const } : {}), ...(wasmExportsRenamed > 0 ? { wasmExportRenameApplied: true as const } : {}), ...(wasmExportRenames.length > 0 ? { wasmExportRenames } : {}), ...(wasmExportOrderShuffled ? { wasmExportOrderShuffled: true as const } : {}), diff --git a/src/engine/verifier.ts b/src/engine/verifier.ts index 51a22f3..6a2fcfc 100644 --- a/src/engine/verifier.ts +++ b/src/engine/verifier.ts @@ -15,7 +15,7 @@ import { inspectNativeBinary } from "../shields/formats/native.js"; import { inspectSqliteBuffer } from "../shields/formats/sqlite.js"; import { verifySelfSealSelfDigest } from "../shields/runtime/self-seal.js"; import { jsModuleHasHoistedDependencyGraph } from "../shields/javascript/ast-shield.js"; -import { isWasmSidecarLoaderHookWired, isPycSidecarLoaderHookWired } from "../shields/runtime/sidecar-entangle.js"; +import { isWasmSidecarLoaderHookWired, isPycSidecarLoaderHookWired, isGgufSidecarLoaderHookWired } from "../shields/runtime/sidecar-entangle.js"; export interface VerifyTargetOptions { readonly selfSealEnabled?: boolean; @@ -194,6 +194,18 @@ export function verifyTarget( } } + let ggufSidecarLoaderHookApplied = false; + if (afterAudit.detectedFormat === "model-gguf") { + try { + ggufSidecarLoaderHookApplied = isGgufSidecarLoaderHookWired({ + ggufBytes: readFileSync(absTarget), + ggufPath: absTarget, + }); + } catch { + ggufSidecarLoaderHookApplied = false; + } + } + let esmImportHoistBeforeHooks = false; if ( afterAudit.detectedFormat === "javascript" || @@ -231,6 +243,7 @@ export function verifyTarget( ...(esmImportHoistBeforeHooks ? { esmImportHoistBeforeHooks: true } : {}), ...(wasmSidecarLoaderHookApplied ? { wasmSidecarLoaderHookApplied: true } : {}), ...(pycSidecarLoaderHookApplied ? { pycSidecarLoaderHookApplied: true } : {}), + ...(ggufSidecarLoaderHookApplied ? { ggufSidecarLoaderHookApplied: true } : {}), }); const agentActions: VerificationAgentAction[] = []; diff --git a/src/identity.ts b/src/identity.ts index 7ed6552..8bebf50 100644 --- a/src/identity.ts +++ b/src/identity.ts @@ -354,4 +354,33 @@ export const REA_INTERNAL_LIMITS = Object.freeze({ */ ANDROID_PACKED_TABLE_BYTES: 16_777_216, MAX_PE_OPTIONAL_HEADER_BYTES: 4_096, + /** + * New REA ceilings on tip c84fde05e (345 commits after 31aaec6b4; 230 fix / + * 41 refactor / 22 test / 19 perf / 14 docs / 9 ci / 10 feat). The feat + * commits add inspection capability (FDO packaging/dlopen notes #7237, + * Ghidra prototype conflicts #8161, MIPS32 PIC shared images, RISC-V + * initializers / PIC, cargo-auditable dependencies from WASM #5069, + * DOS COM import, Ghidra 12.2, EM_PPC64 classification) — not plantable + * densify trip targets. Honesty catalog only: + * + * - `FDO_ALLOWANCE` (64 KiB) is a shared per-report JSON decode cap for + * ELF FDO packaging / dlopen notes (`ElfFdoNote.ts`, #7237). Planting + * dense 64 KiB notes would bloat the ELF; overflow is left undecoded. + * - `BUNDLE_INFO_PLIST_INPUT_BYTES` (64 MiB) and `INFO_PLIST_DECODE_MIB` + * (16) cap Info.plist admission / decode (`AppBundleExecutable.ts`). + * A 64 MiB property list would bloat or break the app bundle. + * - `MAX_WASM_SECTIONS` (65,536) caps cargo-auditable WASM walks + * (`RustWasmImage.ts`, #5069). Planting 65,536 sections would bloat + * or break the module. + * - Also noted, not catalogued as new trip targets: `MIPS_GNU_ATTRIBUTES_MAX_BYTES` + * (4096, format validation), PE resource `METADATA_BYTES` (8 MiB), + * `ASN1_DEPTH_BUDGET` (64, Authenticode nesting — planting would + * invalidate the certificate), `GUARDIAN_READINESS_BYTES` (4096, JADX + * workspace, not plantable). `MAX_VENDOR_HEADER_SIZE` (1 MiB) was + * already catalogued as `ANDROID_BOOT_MAX_HEADER_SIZE`. + */ + FDO_ALLOWANCE: 65_536, + BUNDLE_INFO_PLIST_INPUT_BYTES: 64 * 1024 * 1024, + INFO_PLIST_DECODE_MIB: 16, + MAX_WASM_SECTIONS: 65_536, }); diff --git a/src/index.ts b/src/index.ts index a7b66ce..d22a1f5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -557,6 +557,19 @@ export { pycModuleStemFromBasename, relativePycPathFromLoader, PYC_SIDECAR_LOADER_MARKER, + generateSidecarGgufLoaderSnippet, + prependGgufSidecarLoaderHook, + entangleGgufLoaderWithSidecars, + computeGgufSidecarCodeIdentity, + sourceMentionsGgufFile, + isSafePythonGgufLoader, + selectPythonGgufLoader, + listSameDirPythonGgufLoaderCandidates, + listGgufLoaderSearchDirectories, + discoverPythonGgufLoader, + isGgufSidecarLoaderHookWired, + relativeGgufPathFromLoader, + GGUF_SIDECAR_LOADER_MARKER, generateSidecarGgufMetadataNote, asarMemberPathFromMain, selectAsarMainJsMember, diff --git a/src/shields/runtime/sidecar-entangle.ts b/src/shields/runtime/sidecar-entangle.ts index 38f241b..27f56c2 100644 --- a/src/shields/runtime/sidecar-entangle.ts +++ b/src/shields/runtime/sidecar-entangle.ts @@ -3419,7 +3419,7 @@ const MAX_PYC_LOADER_CANDIDATE_BYTES = 2 * 1024 * 1024; const MAX_PYC_LOADER_AUTOWIRE_CHARS = 64 * 1024; const PYC_LOADER_EXTENSIONS = [".py"] as const; const ALREADY_ENTANGLED_PY_RE = - /bla-self-seal\/v1|_verify_sibling_digest_|integrity binding mismatch|_installPycSidecarLoaderHook|_installWasmSidecarLoaderHook/; + /bla-self-seal\/v1|_verify_sibling_digest_|integrity binding mismatch|_installPycSidecarLoaderHook|_installWasmSidecarLoaderHook|_installGgufSidecarLoaderHook/; /** Module stem of a CPython cache filename (`sample.cpython-312.opt-1.pyc` → `sample`). */ export function pycModuleStemFromBasename(filename: string): string { @@ -3794,7 +3794,7 @@ export function bindGgufSidecarEntanglement( ...plan, entryHookKind: "gguf-metadata-note", status: "bound", - note: "Bound — GGUF bla.sidecar.v1 STRING metadata note planted; ship sidecars alongside and verify via verifyGgufSidecarMetadataNote. Host verify fails closed on strip/alter; GGUF loaders ignore unknown STRING KVs (no process-start fail until a load-time hook lands). Tensor bytes stay bit-exact.", + note: "Bound — GGUF bla.sidecar.v1 STRING metadata note planted; ship sidecars alongside and verify via verifyGgufSidecarMetadataNote. Host verify fails closed on strip/alter; GGUF loaders ignore unknown STRING KVs unless a same-directory Python loader was auto-wired with the load-time hook. Tensor bytes stay bit-exact.", }; return { plan: boundPlan, @@ -3814,8 +3814,9 @@ export function entangleGgufEntryWithSidecars(options: { readonly sidecars: readonly SidecarBindingPart[]; readonly seedTag?: string; readonly softFold?: boolean; + readonly allowEmptySidecars?: boolean; }): SidecarGgufBindResult | null { - if (options.sidecars.length === 0) return null; + if (options.sidecars.length === 0 && options.allowEmptySidecars !== true) return null; return bindGgufSidecarEntanglement({ entryPath: options.entryPath, sidecarPaths: options.sidecars.map((s) => s.pathLabel), @@ -3912,8 +3913,9 @@ export function stripGgufSidecarMetadataNote(entryBytes: Uint8Array): Buffer { * digest. Missing note / missing sibling / altered sibling ⇒ ok:false. * * DEFENSIVE ONLY — for tests, loaders, and harden post-pass verification. - * GGUF loaders ignore unknown STRING KVs, so strip does not break load until - * a load-time hook lands (same honesty bar as .pyc marshal trailer). + * GGUF loaders ignore unknown STRING KVs, so strip does not break load unless + * a same-directory Python loader was auto-wired with the load-time hook + * (same honesty bar as the .pyc import-time hook). */ export function verifyGgufSidecarMetadataNote( options: GgufSidecarVerifyOptions, @@ -3971,6 +3973,552 @@ export function verifyGgufSidecarMetadataNote( }; } + +/** + * SHA-256 of a GGUF container with the `bla.sidecar.v1` STRING KV stripped + * (alignment-safe rebuild). Files that do not carry the note hash as-is, so + * the identity is stable across planting for canonically padded inputs. + * + * DEFENSIVE ONLY. + */ +export function computeGgufSidecarCodeIdentity(bytes: Uint8Array): string { + let stripped: Buffer; + try { + stripped = stripGgufSidecarMetadataNote(bytes); + } catch { + stripped = Buffer.from(bytes); + } + return createHash("sha256").update(stripped).digest("hex"); +} + +/** Idempotence marker inserted with the optional GGUF load-time hook. */ +export const GGUF_SIDECAR_LOADER_MARKER = "_installGgufSidecarLoaderHook" as const; + +export interface SidecarGgufLoaderInjectOptions { + /** + * SHA-256 hex (or list) of the protected GGUF with the sidecar metadata + * note stripped. Files whose identity matches MUST present a valid + * `bla.sidecar.v1` STRING KV; other GGUF files pass through unchanged. + */ + readonly codeIdentityHex: string | readonly string[]; + /** + * Paths to the protected `.gguf`, relative to the loader file's directory + * (`Path(__file__).resolve().parent`). Sidecar paths inside the note are + * resolved against the `.gguf` file's own directory. + */ + readonly ggufRelPaths: readonly string[]; + readonly bindingName?: string; +} + +export interface SidecarGgufLoaderInjectResult { + readonly schema: typeof SIDECAR_ENTANGLE_SCHEMA; + readonly preamble: string; + readonly bindingName: string; + readonly codeIdentityHexes: readonly string[]; + readonly ggufRelPaths: readonly string[]; +} + +function normalizeGgufCodeIdentity(hex: string): string { + const value = hex.toLowerCase(); + if (!/^[0-9a-f]{64}$/.test(value)) { + throw new Error("codeIdentityHex must be a 64-char lowercase hex SHA-256 digest"); + } + return value; +} + +/** + * Emit a self-contained host Python snippet that, at import time of the + * loader it is inserted into, reads the sibling `.gguf`, and when the + * stripped-body identity matches, verifies the `bla.sidecar.v1` STRING + * metadata note and sibling sidecar digest. Strip or sibling delete fails + * closed. Unrelated GGUF files (identity mismatch) and a missing file pass + * through so a later open can fail naturally. + * + * The hook does not rewrite the `.gguf` (tensor infos, relative tensor + * offsets, and tensor bytes stay untouched; the existing alignment-safe + * metadata note is the only GGUF mutation, and it is planted separately). + * It is host-opt-in: `protectTarget` / `hardenProject` auto-wire it only + * into a known same-directory Python loader (`loader.py`, `__init__.py`, + * `__main__.py`) that already mentions the `.gguf` basename — never a + * test / already-entangled / self-seal file. Only then is + * `gguf-sidecar-entangle` recorded and `gguf-sidecar-host-verify-only` + * cleared. + * + * DEFENSIVE ONLY. Opt-in. No crack / RE advice. + */ +export function generateSidecarGgufLoaderSnippet( + options: SidecarGgufLoaderInjectOptions, +): SidecarGgufLoaderInjectResult { + const raw = options.codeIdentityHex; + const list = (Array.isArray(raw) ? raw : [raw]).map((hex) => + normalizeGgufCodeIdentity(hex), + ); + if (list.length === 0) { + throw new Error("codeIdentityHex must contain at least one digest"); + } + const relPaths = (options.ggufRelPaths ?? []).filter( + (p) => typeof p === "string" && p.length > 0, + ); + if (relPaths.length === 0) { + throw new Error("ggufRelPaths must contain at least one path"); + } + const tag = list[0]!.slice(0, 8); + const bindingName = options.bindingName ?? `${GGUF_SIDECAR_LOADER_MARKER}_${tag}`; + const idsLiteral = JSON.stringify(list); + const pathsLiteral = JSON.stringify(relPaths); + const keyLiteral = JSON.stringify(GGUF_SIDECAR_METADATA_KEY); + const schemaLiteral = JSON.stringify(SIDECAR_ENTANGLE_SCHEMA); + const kindLiteral = JSON.stringify(GGUF_SIDECAR_METADATA_KIND); + + const lines: string[] = []; + lines.push(`# ${GGUF_SIDECAR_LOADER_MARKER}`); + lines.push(`def ${bindingName}():`); + lines.push(` import hashlib as _sc_hashlib_${tag}`); + lines.push(` import json as _sc_json_${tag}`); + lines.push(` import pathlib as _sc_pathlib_${tag}`); + lines.push(` _sc_ids_${tag} = ${idsLiteral}`); + lines.push(` _sc_idset_${tag} = set(_sc_ids_${tag})`); + lines.push(` _sc_paths_${tag} = ${pathsLiteral}`); + lines.push(` _sc_key_${tag} = ${keyLiteral}`); + lines.push(` _sc_schema_${tag} = ${schemaLiteral}`); + lines.push(` _sc_kind_${tag} = ${kindLiteral}`); + lines.push(` _sc_base_${tag} = _sc_pathlib_${tag}.Path(__file__).resolve().parent`); + lines.push(` def _sc_skip_${tag}(_sc_buf_${tag}, _sc_off_${tag}, _sc_ty_${tag}, _sc_depth_${tag}):`); + lines.push(` if _sc_depth_${tag} > 64 or _sc_off_${tag} > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` if _sc_ty_${tag} in (0, 1, 7):`); + lines.push(` return _sc_off_${tag} + 1 if _sc_off_${tag} + 1 <= len(_sc_buf_${tag}) else None`); + lines.push(` if _sc_ty_${tag} in (2, 3):`); + lines.push(` return _sc_off_${tag} + 2 if _sc_off_${tag} + 2 <= len(_sc_buf_${tag}) else None`); + lines.push(` if _sc_ty_${tag} in (4, 5, 6):`); + lines.push(` return _sc_off_${tag} + 4 if _sc_off_${tag} + 4 <= len(_sc_buf_${tag}) else None`); + lines.push(` if _sc_ty_${tag} in (10, 11, 12):`); + lines.push(` return _sc_off_${tag} + 8 if _sc_off_${tag} + 8 <= len(_sc_buf_${tag}) else None`); + lines.push(` if _sc_ty_${tag} == 8:`); + lines.push(` if _sc_off_${tag} + 8 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_ln_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_off_${tag}:_sc_off_${tag} + 8], "little")`); + lines.push(` _sc_end_${tag} = _sc_off_${tag} + 8 + _sc_ln_${tag}`); + lines.push(` return _sc_end_${tag} if _sc_end_${tag} <= len(_sc_buf_${tag}) else None`); + lines.push(` if _sc_ty_${tag} == 9:`); + lines.push(` if _sc_off_${tag} + 12 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_elem_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_off_${tag}:_sc_off_${tag} + 4], "little")`); + lines.push(` _sc_cnt_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_off_${tag} + 4:_sc_off_${tag} + 12], "little")`); + lines.push(` if _sc_cnt_${tag} > len(_sc_buf_${tag}) - (_sc_off_${tag} + 12):`); + lines.push(` return None`); + lines.push(` _sc_cur_${tag} = _sc_off_${tag} + 12`); + lines.push(` for _sc_i_${tag} in range(_sc_cnt_${tag}):`); + lines.push(` _sc_nxt_${tag} = _sc_skip_${tag}(_sc_buf_${tag}, _sc_cur_${tag}, _sc_elem_${tag}, _sc_depth_${tag} + 1)`); + lines.push(` if _sc_nxt_${tag} is None:`); + lines.push(` return None`); + lines.push(` _sc_cur_${tag} = _sc_nxt_${tag}`); + lines.push(` return _sc_cur_${tag}`); + lines.push(` return None`); + lines.push(` def _sc_parse_${tag}(_sc_buf_${tag}):`); + lines.push(` if len(_sc_buf_${tag}) < 24 or _sc_buf_${tag}[0:4] != b"GGUF":`); + lines.push(` return None`); + lines.push(` _sc_ver_${tag} = int.from_bytes(_sc_buf_${tag}[4:8], "little")`); + lines.push(` if _sc_ver_${tag} not in (2, 3):`); + lines.push(` return None`); + lines.push(` _sc_tc_${tag} = int.from_bytes(_sc_buf_${tag}[8:16], "little")`); + lines.push(` _sc_kc_${tag} = int.from_bytes(_sc_buf_${tag}[16:24], "little")`); + lines.push(` if _sc_kc_${tag} > 100000 or _sc_tc_${tag} > 1000000:`); + lines.push(` return None`); + lines.push(` _sc_off_${tag} = 24`); + lines.push(` _sc_kvs_${tag} = []`); + lines.push(` _sc_align_${tag} = 32`); + lines.push(` for _sc_i_${tag} in range(_sc_kc_${tag}):`); + lines.push(` _sc_start_${tag} = _sc_off_${tag}`); + lines.push(` if _sc_off_${tag} + 8 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_klen_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_off_${tag}:_sc_off_${tag} + 8], "little")`); + lines.push(` _sc_kend_${tag} = _sc_off_${tag} + 8 + _sc_klen_${tag}`); + lines.push(` if _sc_kend_${tag} + 4 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_key_${tag}2 = _sc_buf_${tag}[_sc_off_${tag} + 8:_sc_kend_${tag}].decode("utf-8", "replace")`); + lines.push(` _sc_vty_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_kend_${tag}:_sc_kend_${tag} + 4], "little")`); + lines.push(` _sc_nxt_${tag} = _sc_skip_${tag}(_sc_buf_${tag}, _sc_kend_${tag} + 4, _sc_vty_${tag}, 0)`); + lines.push(` if _sc_nxt_${tag} is None:`); + lines.push(` return None`); + lines.push(` if _sc_key_${tag}2 == "general.alignment" and _sc_vty_${tag} == 4:`); + lines.push(` _sc_a_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_kend_${tag} + 4:_sc_kend_${tag} + 8], "little")`); + lines.push(` if _sc_a_${tag} > 0 and _sc_a_${tag} % 8 == 0:`); + lines.push(` if _sc_a_${tag} > 1048576:`); + lines.push(` return None`); + lines.push(` _sc_align_${tag} = _sc_a_${tag}`); + lines.push(` _sc_kvs_${tag}.append((_sc_key_${tag}2, _sc_start_${tag}, _sc_nxt_${tag}, _sc_vty_${tag}))`); + lines.push(` _sc_off_${tag} = _sc_nxt_${tag}`); + lines.push(` _sc_kvend_${tag} = _sc_off_${tag}`); + lines.push(` for _sc_t_${tag} in range(_sc_tc_${tag}):`); + lines.push(` if _sc_off_${tag} + 8 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_nlen_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_off_${tag}:_sc_off_${tag} + 8], "little")`); + lines.push(` _sc_nend_${tag} = _sc_off_${tag} + 8 + _sc_nlen_${tag}`); + lines.push(` if _sc_nend_${tag} + 4 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_off_${tag} = _sc_nend_${tag}`); + lines.push(` _sc_ndims_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_off_${tag}:_sc_off_${tag} + 4], "little")`); + lines.push(` if _sc_ndims_${tag} > 8:`); + lines.push(` return None`); + lines.push(` _sc_off_${tag} += 4`); + lines.push(` _sc_tail_${tag} = _sc_ndims_${tag} * 8 + 4 + 8`); + lines.push(` if _sc_off_${tag} + _sc_tail_${tag} > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_off_${tag} += _sc_tail_${tag}`); + lines.push(` _sc_tiend_${tag} = _sc_off_${tag}`); + lines.push(` _sc_has_${tag} = len(_sc_buf_${tag}) > _sc_tiend_${tag}`); + lines.push(` _sc_rem_${tag} = _sc_tiend_${tag} % _sc_align_${tag}`); + lines.push(` _sc_dstart_${tag} = _sc_tiend_${tag} if _sc_rem_${tag} == 0 else _sc_tiend_${tag} + (_sc_align_${tag} - _sc_rem_${tag})`); + lines.push(` return (_sc_ver_${tag}, _sc_tc_${tag}, _sc_kvs_${tag}, _sc_kvend_${tag}, _sc_tiend_${tag}, _sc_align_${tag}, _sc_dstart_${tag}, _sc_has_${tag})`); + lines.push(` def _sc_strip_${tag}(_sc_buf_${tag}):`); + lines.push(` _sc_lay_${tag} = _sc_parse_${tag}(_sc_buf_${tag})`); + lines.push(` if _sc_lay_${tag} is None:`); + lines.push(` return _sc_buf_${tag}`); + lines.push(` _sc_ver_${tag}, _sc_tc_${tag}, _sc_kvs_${tag}, _sc_kvend_${tag}, _sc_tiend_${tag}, _sc_align_${tag}, _sc_dstart_${tag}, _sc_has_${tag} = _sc_lay_${tag}`); + lines.push(` if not any(_sc_kv_${tag}[0] == _sc_key_${tag} for _sc_kv_${tag} in _sc_kvs_${tag}):`); + lines.push(` return _sc_buf_${tag}`); + lines.push(` _sc_kept_${tag} = [_sc_kv_${tag} for _sc_kv_${tag} in _sc_kvs_${tag} if _sc_kv_${tag}[0] != _sc_key_${tag}]`); + lines.push(` _sc_hdr_${tag} = bytearray(24)`); + lines.push(` _sc_hdr_${tag}[0:4] = b"GGUF"`); + lines.push(` _sc_hdr_${tag}[4:8] = int(_sc_ver_${tag}).to_bytes(4, "little")`); + lines.push(` _sc_hdr_${tag}[8:16] = int(_sc_tc_${tag}).to_bytes(8, "little")`); + lines.push(` _sc_hdr_${tag}[16:24] = int(len(_sc_kept_${tag})).to_bytes(8, "little")`); + lines.push(` _sc_parts_${tag} = [bytes(_sc_hdr_${tag})]`); + lines.push(` for _sc_kv_${tag} in _sc_kept_${tag}:`); + lines.push(` _sc_parts_${tag}.append(_sc_buf_${tag}[_sc_kv_${tag}[1]:_sc_kv_${tag}[2]])`); + lines.push(` _sc_parts_${tag}.append(_sc_buf_${tag}[_sc_kvend_${tag}:_sc_tiend_${tag}])`); + lines.push(` _sc_meta_${tag} = b"".join(_sc_parts_${tag})`); + lines.push(` if not _sc_has_${tag}:`); + lines.push(` return _sc_meta_${tag}`); + lines.push(` if _sc_dstart_${tag} > len(_sc_buf_${tag}):`); + lines.push(` return _sc_buf_${tag}`); + lines.push(` _sc_prem_${tag} = len(_sc_meta_${tag}) % _sc_align_${tag}`); + lines.push(` _sc_pad_${tag} = 0 if _sc_prem_${tag} == 0 else _sc_align_${tag} - _sc_prem_${tag}`); + lines.push(` return _sc_meta_${tag} + (b"\\x00" * _sc_pad_${tag}) + _sc_buf_${tag}[_sc_dstart_${tag}:]`); + lines.push(` def _sc_extract_${tag}(_sc_buf_${tag}):`); + lines.push(` _sc_lay_${tag} = _sc_parse_${tag}(_sc_buf_${tag})`); + lines.push(` if _sc_lay_${tag} is None:`); + lines.push(` return None`); + lines.push(` _sc_kvs_${tag} = _sc_lay_${tag}[2]`); + lines.push(` _sc_hit_${tag} = None`); + lines.push(` for _sc_kv_${tag} in _sc_kvs_${tag}:`); + lines.push(` if _sc_kv_${tag}[0] == _sc_key_${tag} and _sc_kv_${tag}[3] == 8:`); + lines.push(` _sc_hit_${tag} = _sc_kv_${tag}`); + lines.push(` break`); + lines.push(` if _sc_hit_${tag} is None:`); + lines.push(` return None`); + lines.push(` _sc_start_${tag} = _sc_hit_${tag}[1]`); + lines.push(` _sc_klen_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_start_${tag}:_sc_start_${tag} + 8], "little")`); + lines.push(` _sc_vstart_${tag} = _sc_start_${tag} + 8 + _sc_klen_${tag} + 4`); + lines.push(` if _sc_vstart_${tag} + 8 > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` _sc_vlen_${tag} = int.from_bytes(_sc_buf_${tag}[_sc_vstart_${tag}:_sc_vstart_${tag} + 8], "little")`); + lines.push(` if _sc_vstart_${tag} + 8 + _sc_vlen_${tag} > len(_sc_buf_${tag}):`); + lines.push(` return None`); + lines.push(` try:`); + lines.push(` _sc_obj_${tag} = _sc_json_${tag}.loads(_sc_buf_${tag}[_sc_vstart_${tag} + 8:_sc_vstart_${tag} + 8 + _sc_vlen_${tag}].decode("utf-8"))`); + lines.push(` except Exception:`); + lines.push(` return None`); + lines.push(` if _sc_obj_${tag}.get("schema") != _sc_schema_${tag} or _sc_obj_${tag}.get("kind") != _sc_kind_${tag}:`); + lines.push(` return None`); + lines.push(` _sc_dig_${tag} = _sc_obj_${tag}.get("digest")`); + lines.push(` if not isinstance(_sc_dig_${tag}, str) or len(_sc_dig_${tag}) != 64:`); + lines.push(` return None`); + lines.push(` return _sc_obj_${tag}`); + lines.push(` def _sc_digest_${tag}(_sc_entry_${tag}, _sc_parts_${tag}, _sc_seed_${tag}):`); + lines.push(` _sc_h_${tag} = _sc_hashlib_${tag}.sha256()`); + lines.push(` _sc_h_${tag}.update(_sc_entry_${tag}.encode("utf-8"))`); + lines.push(` _sc_h_${tag}.update(b"\\x00")`); + lines.push(` for _sc_sp_${tag}, _sc_data_${tag} in _sc_parts_${tag}:`); + lines.push(` _sc_h_${tag}.update(_sc_sp_${tag}.encode("utf-8"))`); + lines.push(` _sc_h_${tag}.update(b"\\x00")`); + lines.push(` _sc_h_${tag}.update(_sc_data_${tag})`); + lines.push(` _sc_h_${tag}.update(b"\\x00")`); + lines.push(` _sc_h_${tag}.update(_sc_seed_${tag}.encode("utf-8"))`); + lines.push(` return _sc_h_${tag}.hexdigest()`); + lines.push(` for _sc_rel_${tag} in _sc_paths_${tag}:`); + lines.push(` _sc_path_${tag} = _sc_pathlib_${tag}.Path(_sc_rel_${tag})`); + lines.push(` if not _sc_path_${tag}.is_absolute():`); + lines.push(` _sc_path_${tag} = _sc_base_${tag} / _sc_rel_${tag}`); + lines.push(` try:`); + lines.push(` _sc_raw_${tag} = _sc_path_${tag}.read_bytes()`); + lines.push(` except Exception:`); + lines.push(` continue`); + lines.push(` _sc_ident_${tag} = _sc_hashlib_${tag}.sha256(_sc_strip_${tag}(_sc_raw_${tag})).hexdigest()`); + lines.push(` if _sc_ident_${tag} not in _sc_idset_${tag}:`); + lines.push(` continue`); + lines.push(` _sc_note_${tag} = _sc_extract_${tag}(_sc_raw_${tag})`); + lines.push(` if _sc_note_${tag} is None:`); + lines.push(` raise RuntimeError("integrity binding mismatch")`); + lines.push(` _sc_parts_${tag} = []`); + lines.push(` _sc_dir_${tag} = _sc_path_${tag}.parent`); + lines.push(` _sc_sp_list_${tag} = _sc_note_${tag}.get("sidecarPaths") or []`); + lines.push(` if not isinstance(_sc_sp_list_${tag}, list):`); + lines.push(` raise RuntimeError("integrity binding mismatch")`); + lines.push(` for _sc_sp_${tag} in _sc_sp_list_${tag}:`); + lines.push(` if not isinstance(_sc_sp_${tag}, str) or len(_sc_sp_${tag}) == 0:`); + lines.push(` continue`); + lines.push(` _sc_sib_${tag} = _sc_pathlib_${tag}.Path(_sc_sp_${tag})`); + lines.push(` if not _sc_sib_${tag}.is_absolute():`); + lines.push(` _sc_sib_${tag} = _sc_dir_${tag} / _sc_sp_${tag}`); + lines.push(` try:`); + lines.push(` _sc_parts_${tag}.append((_sc_sp_${tag}, _sc_sib_${tag}.read_bytes()))`); + lines.push(` except Exception:`); + lines.push(` raise RuntimeError("integrity binding mismatch") from None`); + lines.push(` _sc_seed_${tag} = _sc_note_${tag}.get("seedTag")`); + lines.push(` if not isinstance(_sc_seed_${tag}, str) or len(_sc_seed_${tag}) == 0:`); + lines.push(` _sc_seed_${tag} = "bla-sidecar-v1"`); + lines.push(` _sc_entry_${tag} = _sc_note_${tag}.get("entryPath")`); + lines.push(` if not isinstance(_sc_entry_${tag}, str):`); + lines.push(` _sc_entry_${tag} = ""`); + lines.push(` if _sc_digest_${tag}(_sc_entry_${tag}, _sc_parts_${tag}, _sc_seed_${tag}) != str(_sc_note_${tag}.get("digest")).lower():`); + lines.push(` raise RuntimeError("integrity binding mismatch")`); + lines.push(` return`); + lines.push(`try:`); + lines.push(` ${bindingName}()`); + lines.push(`finally:`); + lines.push(` del ${bindingName}`); + + return { + schema: SIDECAR_ENTANGLE_SCHEMA, + preamble: lines.join("\n"), + bindingName, + codeIdentityHexes: list, + ggufRelPaths: relPaths, + }; +} + +/** + * Insert a GGUF load-time hook after shebang / encoding / docstring / + * `from __future__`, idempotent when {@link GGUF_SIDECAR_LOADER_MARKER} is + * already present. Preserves surrounding newlines the same way + * `bindPythonSidecarEntanglement` does. + */ +export function prependGgufSidecarLoaderHook( + entrySource: string, + preamble: string, +): string { + if (entrySource.includes(GGUF_SIDECAR_LOADER_MARKER)) return entrySource; + const eol = entrySource.includes("\r\n") ? "\r\n" : "\n"; + const analysis = findPythonPrologueInsertionOffset(entrySource); + const headRaw = entrySource.slice(0, analysis.insertOffset); + const tailRaw = entrySource.slice(analysis.insertOffset); + const head = headRaw.replace(/(?:\r?\n)+$/, ""); + const tail = tailRaw.replace(/^(?:\r?\n)+/, ""); + const body = preamble.replace(/\r?\n/g, eol).replace(/(?:\r?\n)+$/, ""); + const parts: string[] = []; + if (head.length > 0) parts.push(head); + parts.push(body); + if (tail.length > 0) parts.push(tail); + return ( + parts.join(eol) + + (entrySource.endsWith("\n") || entrySource.endsWith("\r\n") ? eol : "") + ); +} + +/** + * Convenience: prepend a loader hook bound to this GGUF file's stripped + * identity. Idempotent when the marker is already present. + * + * DEFENSIVE ONLY. Opt-in. No crack / RE advice. + */ +export function entangleGgufLoaderWithSidecars(options: { + readonly loaderSource: string; + readonly ggufBytes: Uint8Array; + readonly ggufRelPath: string | readonly string[]; +}): { readonly code: string; readonly identity: string } { + const identity = computeGgufSidecarCodeIdentity(options.ggufBytes); + const paths = ( + Array.isArray(options.ggufRelPath) ? options.ggufRelPath : [options.ggufRelPath] + ).filter((p) => p.length > 0); + const inject = generateSidecarGgufLoaderSnippet({ + codeIdentityHex: identity, + ggufRelPaths: paths, + }); + return { + code: prependGgufSidecarLoaderHook(options.loaderSource, inject.preamble), + identity, + }; +} + +const MAX_GGUF_LOADER_CANDIDATE_BYTES = 2 * 1024 * 1024; +const MAX_GGUF_LOADER_AUTOWIRE_CHARS = 64 * 1024; + +/** + * True when `source` mentions `ggufBasename` in a quoted string / template. + */ +export function sourceMentionsGgufFile(source: string, ggufBasename: string): boolean { + const name = (ggufBasename.replaceAll("\\", "/").split("/").pop() ?? ""); + if (!name || !name.toLowerCase().endsWith(".gguf")) return false; + const quoted = new RegExp( + `['"\`](?:[^'"\`\\\\]|\\\\.)*${escapeRegExpLiteral(name)}['"\`]`, + ); + return quoted.test(source); +} + +/** + * Conservative auto-wire predicate: a Python file with a known loader name + * (`loader.py`, `__init__.py`, `__main__.py`) that already mentions the + * target `.gguf` basename. Test, already-entangled, and oversized files + * return false. + * + * DEFENSIVE ONLY. Opt-in. No crack / RE advice. + */ +export function isSafePythonGgufLoader(options: { + readonly source: string; + readonly filename: string; + readonly ggufBasename: string; +}): boolean { + if (!isPythonPycLoaderFilename(options.filename)) return false; + if (options.source.length > MAX_GGUF_LOADER_AUTOWIRE_CHARS) return false; + if (ALREADY_ENTANGLED_PY_RE.test(options.source)) return false; + return sourceMentionsGgufFile(options.source, options.ggufBasename); +} + +function ggufLoaderPreference(loaderName: string): number { + const base = basename(loaderName); + if (/^loader\.py$/i.test(base)) return 0; + if (/^__init__\.py$/i.test(base)) return 1; + if (/^__main__\.py$/i.test(base)) return 2; + return 3; +} + +export interface PythonGgufLoaderCandidate { + readonly name: string; + readonly source: string; +} + +/** + * Pick a single same-directory Python loader, or `null` when none qualify + * or several unranked files all qualify (do not guess). Preferred names: + * `loader.py`, then `__init__.py`, then `__main__.py`. + * + * DEFENSIVE ONLY. Opt-in. No crack / RE advice. + */ +export function selectPythonGgufLoader( + ggufBasename: string, + candidates: readonly PythonGgufLoaderCandidate[], +): string | null { + const qualifying = candidates.filter((c) => + isSafePythonGgufLoader({ + source: c.source, + filename: c.name, + ggufBasename, + }), + ); + if (qualifying.length === 0) return null; + const scored = qualifying.map((c) => ({ + c, + score: ggufLoaderPreference(c.name), + })); + scored.sort((a, b) => a.score - b.score || a.c.name.localeCompare(b.c.name)); + const best = scored[0]!; + if (best.score >= 3) return null; + const ties = scored.filter((s) => s.score === best.score); + if (ties.length > 1) return null; + return best.c.name; +} + +export interface DiscoveredPythonGgufLoader { + readonly absPath: string; + readonly name: string; + readonly source: string; +} + +function readGgufLoaderCandidateSource(absPath: string): string | null { + try { + const st = statSync(absPath); + if (!st.isFile() || st.size <= 0 || st.size > MAX_GGUF_LOADER_CANDIDATE_BYTES) { + return null; + } + return readFileSync(absPath, "utf8"); + } catch { + return null; + } +} + +/** Directories to search for a Python loader next to a `.gguf`. */ +export function listGgufLoaderSearchDirectories(ggufPath: string): string[] { + return [dirname(ggufPath)]; +} + +/** + * List same-directory `.py` candidates (no recursion). Unreadable or + * oversized files are skipped. + */ +export function listSameDirPythonGgufLoaderCandidates( + directory: string, +): DiscoveredPythonGgufLoader[] { + let names: string[] = []; + try { + names = readdirSync(directory); + } catch { + return []; + } + const out: DiscoveredPythonGgufLoader[] = []; + for (const name of names) { + if (!isPythonPycLoaderFilename(name)) continue; + const absPath = join(directory, name); + const source = readGgufLoaderCandidateSource(absPath); + if (source === null) continue; + out.push({ absPath, name, source }); + } + return out; +} + +/** + * Discover the unique safe Python loader for `ggufBasename` in the first + * search directory that yields a selection. Later directories are not + * consulted once one directory produces a hit (output dir wins over source). + * + * DEFENSIVE ONLY. Opt-in. No crack / RE advice. + */ +export function discoverPythonGgufLoader(options: { + readonly ggufBasename: string; + readonly searchDirectories: readonly string[]; +}): DiscoveredPythonGgufLoader | null { + const seen = new Set(); + for (const directory of options.searchDirectories) { + if (typeof directory !== "string" || directory.length === 0) continue; + if (seen.has(directory)) continue; + seen.add(directory); + const candidates = listSameDirPythonGgufLoaderCandidates(directory); + const selected = selectPythonGgufLoader(options.ggufBasename, candidates); + if (!selected) continue; + const hit = candidates.find((c) => c.name === selected || c.absPath === selected); + if (hit) return hit; + } + return null; +} + +/** + * True when a same-directory Python file already carries the GGUF load-time + * hook bound to this file's code identity (used by defenseScore honesty). + * + * DEFENSIVE ONLY. + */ +export function isGgufSidecarLoaderHookWired(options: { + readonly ggufBytes: Uint8Array; + readonly ggufPath: string; +}): boolean { + const identity = computeGgufSidecarCodeIdentity(options.ggufBytes); + const directories = listGgufLoaderSearchDirectories(options.ggufPath); + return directories.some((directory) => + listSameDirPythonGgufLoaderCandidates(directory).some( + (c) => + c.source.includes(GGUF_SIDECAR_LOADER_MARKER) && c.source.includes(identity), + ), + ); +} + +/** Relative path from a loader file's directory to the protected `.gguf`. */ +export function relativeGgufPathFromLoader(loaderPath: string, ggufPath: string): string { + const rel = relative(dirname(loaderPath), ggufPath).split("\\").join("/"); + return rel.length === 0 ? basename(ggufPath) : rel; +} + + /** Test/helper: strip the sidecar note section from a WASM binary. */ export function stripWasmSidecarCustomSection(bytes: Uint8Array): Buffer { return stripWasmCustomSectionByName(Buffer.from(bytes), WASM_SIDECAR_SECTION_NAME); diff --git a/tests/defense-score-honesty.test.ts b/tests/defense-score-honesty.test.ts index 0b3fc28..3f3ab1f 100644 --- a/tests/defense-score-honesty.test.ts +++ b/tests/defense-score-honesty.test.ts @@ -105,6 +105,14 @@ describe("Issue #40.6: defenseScore honesty", () => { expect(hostOnly.gaps.map((g) => g.id)).toContain("gguf-sidecar-host-verify-only"); expect(hostOnly.gaps.map((g) => g.id)).not.toContain("sidecar-entangle-not-enabled"); expect(hostOnly.cappedScore).toBe(93); + + const hooked = applyDefenseScoreHonesty(99, { + detectedFormat: "model-gguf", + sidecarEntangleEnabled: true, + ggufSidecarLoaderHookApplied: true, + }); + expect(hooked.gaps.map((g) => g.id)).not.toContain("gguf-sidecar-host-verify-only"); + expect(hooked.cappedScore).toBe(99); }); it("keeps a SQLite host-only schema-comment note from closing the strip⇒fail gap", () => { diff --git a/tests/sidecar-gguf-loader-hook.test.ts b/tests/sidecar-gguf-loader-hook.test.ts new file mode 100644 index 0000000..b478cf3 --- /dev/null +++ b/tests/sidecar-gguf-loader-hook.test.ts @@ -0,0 +1,437 @@ +import { describe, expect, it } from "vitest"; +import { + mkdtempSync, + mkdirSync, + writeFileSync, + rmSync, + readFileSync, + unlinkSync, +} from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { + bindGgufSidecarEntanglement, + computeGgufSidecarCodeIdentity, + entangleGgufLoaderWithSidecars, + extractGgufSidecarMetadataNote, + generateSidecarGgufLoaderSnippet, + isGgufSidecarLoaderHookWired, + isSafePythonGgufLoader, + prependGgufSidecarLoaderHook, + GGUF_SIDECAR_LOADER_MARKER, + GGUF_SIDECAR_METADATA_KEY, + selectPythonGgufLoader, + SIDECAR_ENTANGLE_SCHEMA, + sourceMentionsGgufFile, + stripGgufSidecarMetadataNote, +} from "../src/shields/runtime/sidecar-entangle.js"; +import { + encodeGgufString, + encodeGgufStringKv, + parseGgufLayout, +} from "../src/shields/formats/gguf-layout.js"; +import { collectDefenseScoreGaps } from "../src/engine/defense-score-honesty.js"; +import { protectTarget } from "../src/engine/protector.js"; +import { hardenProject } from "../src/engine/harden.js"; +import { verifyTarget } from "../src/engine/verifier.js"; + +const PY3 = (() => { + const r = spawnSync("python3", ["--version"], { encoding: "utf8" }); + return r.status === 0; +})(); + +function buildMinimalGguf(): Buffer { + const kv = encodeGgufStringKv("general.architecture", "llama"); + const name = encodeGgufString("w"); + const rest = Buffer.alloc(4 + 8 + 4 + 8); + rest.writeUInt32LE(1, 0); + rest.writeBigUInt64LE(1n, 4); + rest.writeUInt32LE(0, 12); + rest.writeBigUInt64LE(0n, 16); + const header = Buffer.alloc(24); + header.write("GGUF", 0, "ascii"); + header.writeUInt32LE(3, 4); + header.writeBigUInt64LE(1n, 8); + header.writeBigUInt64LE(1n, 16); + const metaAndInfos = Buffer.concat([header, kv, name, rest]); + const align = 32; + const rem = metaAndInfos.length % align; + const pad = rem === 0 ? Buffer.alloc(0) : Buffer.alloc(align - rem, 0); + const data = Buffer.from([0xde, 0xad, 0xbe, 0xef]); + return Buffer.concat([metaAndInfos, pad, data]); +} + +function tensorDataDigest(bytes: Buffer): string { + const layout = parseGgufLayout(bytes)!; + return createHash("sha256") + .update(bytes.subarray(layout.dataStart)) + .digest("hex"); +} + +const LOADER_BODY = `import pathlib +_gguf = pathlib.Path(__file__).resolve().parent / "model.gguf" +_raw = _gguf.read_bytes() +assert _raw[0:4] == b"GGUF" +print("ok", len(_raw)) +`; + +const FUTURE_LOADER_BODY = `#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""Loader docstring.""" +from __future__ import annotations +import pathlib +_gguf = pathlib.Path(__file__).resolve().parent / "model.gguf" +_raw = _gguf.read_bytes() +assert _raw[0:4] == b"GGUF" +print("ok", len(_raw)) +`; + +function runLoader( + dir: string, + source: string, + filename = "loader.py", +): { status: number; stdout: string; stderr: string } { + writeFileSync(join(dir, filename), source, "utf8"); + const r = spawnSync("python3", [filename], { + cwd: dir, + encoding: "utf8", + timeout: 15_000, + }); + return { + status: r.status ?? 1, + stdout: r.stdout ?? "", + stderr: r.stderr ?? "", + }; +} + +describe("sidecar GGUF load-time hook", () => { + it("generateSidecarGgufLoaderSnippet rejects non-hex identities", () => { + expect(() => + generateSidecarGgufLoaderSnippet({ + codeIdentityHex: "deadbeef", + ggufRelPaths: ["model.gguf"], + }), + ).toThrow(/64-char/); + }); + + it("computeGgufSidecarCodeIdentity is stable across planting the note", () => { + const entry = buildMinimalGguf(); + const unbound = computeGgufSidecarCodeIdentity(entry); + const bound = bindGgufSidecarEntanglement({ + entryPath: "model.gguf", + sidecarPaths: ["sib.bin"], + seedTag: "loader-id-v1", + entryBytes: entry, + sidecarBytes: [new TextEncoder().encode("sidecar-payload\n")], + }); + expect(computeGgufSidecarCodeIdentity(bound.bytes)).toBe(unbound); + expect(unbound).toMatch(/^[0-9a-f]{64}$/); + const stripped = stripGgufSidecarMetadataNote(bound.bytes); + expect(computeGgufSidecarCodeIdentity(stripped)).toBe(unbound); + expect(Buffer.compare(stripped, entry)).toBe(0); + expect(tensorDataDigest(bound.bytes)).toBe(tensorDataDigest(entry)); + }); + + it("prepend is idempotent and preserves shebang / encoding / docstring / futures", () => { + const inject = generateSidecarGgufLoaderSnippet({ + codeIdentityHex: "a".repeat(64), + ggufRelPaths: ["model.gguf"], + }); + expect(inject.schema).toBe(SIDECAR_ENTANGLE_SCHEMA); + expect(inject.preamble).toContain(GGUF_SIDECAR_LOADER_MARKER); + const once = prependGgufSidecarLoaderHook(FUTURE_LOADER_BODY, inject.preamble); + expect(once.startsWith("#!/usr/bin/env python3\n")).toBe(true); + expect(once).toContain("# -*- coding: utf-8 -*-"); + expect(once).toContain('"""Loader docstring."""'); + expect(once).toContain("from __future__ import annotations"); + const hookAt = once.indexOf(GGUF_SIDECAR_LOADER_MARKER); + const futureAt = once.indexOf("from __future__ import annotations"); + expect(hookAt).toBeGreaterThan(futureAt); + const twice = prependGgufSidecarLoaderHook(once, inject.preamble); + expect(twice).toBe(once); + }); + + it("isSafe / select only admit known loaders that mention the .gguf", () => { + const loader = LOADER_BODY; + const noMention = "print('hello')\n"; + const helper = + 'import pathlib\n_gguf = pathlib.Path("model.gguf")\n'; + const testLoader = + 'import pathlib\n_gguf = pathlib.Path(__file__).parent / "model.gguf"\n'; + const already = `${loader}\n# _verify_sibling_digest_deadbeef\n`; + + expect(sourceMentionsGgufFile(loader, "model.gguf")).toBe(true); + expect(sourceMentionsGgufFile(noMention, "model.gguf")).toBe(false); + + expect( + isSafePythonGgufLoader({ + source: loader, + filename: "loader.py", + ggufBasename: "model.gguf", + }), + ).toBe(true); + expect( + isSafePythonGgufLoader({ + source: testLoader, + filename: "test_loader.py", + ggufBasename: "model.gguf", + }), + ).toBe(false); + expect( + isSafePythonGgufLoader({ + source: already, + filename: "loader.py", + ggufBasename: "model.gguf", + }), + ).toBe(false); + expect( + isSafePythonGgufLoader({ + source: helper, + filename: "helper.py", + ggufBasename: "model.gguf", + }), + ).toBe(true); + expect( + isSafePythonGgufLoader({ + source: noMention, + filename: "loader.py", + ggufBasename: "model.gguf", + }), + ).toBe(false); + + expect( + selectPythonGgufLoader("model.gguf", [ + { name: "helper.py", source: helper }, + { name: "loader.py", source: loader }, + { name: "__init__.py", source: loader }, + ]), + ).toBe("loader.py"); + expect( + selectPythonGgufLoader("model.gguf", [ + { name: "__init__.py", source: loader }, + { name: "__main__.py", source: loader }, + ]), + ).toBe("__init__.py"); + expect( + selectPythonGgufLoader("model.gguf", [ + { name: "helper.py", source: helper }, + ]), + ).toBeNull(); + }); + + it.skipIf(!PY3)("load-time hook: intact model opens and keeps tensor bytes", () => { + const dir = mkdtempSync(join(tmpdir(), "bla-gguf-loader-ok-")); + try { + const entry = buildMinimalGguf(); + const data0 = tensorDataDigest(entry); + const snap = Buffer.from("sidecar-payload\n"); + const bound = bindGgufSidecarEntanglement({ + entryPath: "model.gguf", + sidecarPaths: ["sib.bin"], + seedTag: "gguf-loader-ok", + entryBytes: entry, + sidecarBytes: [snap], + }); + writeFileSync(join(dir, "model.gguf"), bound.bytes); + writeFileSync(join(dir, "sib.bin"), snap); + const hooked = entangleGgufLoaderWithSidecars({ + loaderSource: LOADER_BODY, + ggufBytes: bound.bytes, + ggufRelPath: "model.gguf", + }); + expect(hooked.code).toContain(GGUF_SIDECAR_LOADER_MARKER); + expect(tensorDataDigest(bound.bytes)).toBe(data0); + const ran = runLoader(dir, hooked.code); + expect(ran.status, ran.stderr || ran.stdout).toBe(0); + expect(ran.stdout).toMatch(/ok /); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it.skipIf(!PY3)("load-time hook fails closed on metadata strip or sibling delete", () => { + const dir = mkdtempSync(join(tmpdir(), "bla-gguf-loader-fail-")); + try { + const entry = buildMinimalGguf(); + const snap = Buffer.from("sidecar-payload\n"); + const bound = bindGgufSidecarEntanglement({ + entryPath: "model.gguf", + sidecarPaths: ["sib.bin"], + seedTag: "gguf-loader-fail", + entryBytes: entry, + sidecarBytes: [snap], + }); + writeFileSync(join(dir, "sib.bin"), snap); + const hooked = entangleGgufLoaderWithSidecars({ + loaderSource: LOADER_BODY, + ggufBytes: bound.bytes, + ggufRelPath: "model.gguf", + }); + + writeFileSync(join(dir, "model.gguf"), bound.bytes); + const ok = runLoader(dir, hooked.code); + expect(ok.status, ok.stderr || ok.stdout).toBe(0); + + writeFileSync(join(dir, "model.gguf"), stripGgufSidecarMetadataNote(bound.bytes)); + const stripped = runLoader(dir, hooked.code); + expect(stripped.status).not.toBe(0); + expect(stripped.stderr + stripped.stdout).toMatch(/integrity binding mismatch/); + + writeFileSync(join(dir, "model.gguf"), bound.bytes); + unlinkSync(join(dir, "sib.bin")); + const deleted = runLoader(dir, hooked.code); + expect(deleted.status).not.toBe(0); + expect(deleted.stderr + deleted.stdout).toMatch(/integrity binding mismatch/); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it.skipIf(!PY3)("protectTarget auto-wires a same-directory loader and records gguf-sidecar-entangle", () => { + const dir = mkdtempSync(join(tmpdir(), "bla-gguf-loader-protect-")); + try { + const entry = join(dir, "model.gguf"); + writeFileSync(entry, buildMinimalGguf()); + writeFileSync(join(dir, "sib.bin"), "sidecar-payload\n"); + writeFileSync(join(dir, "loader.py"), LOADER_BODY, "utf8"); + + const res = protectTarget({ + inputPath: entry, + outputPath: entry, + intensity: "balanced", + seed: "pt-gguf-loader-hook", + sidecarEntangle: true, + sidecarEntanglePaths: ["sib.bin"], + inPlace: true, + }); + expect(res.sidecarEntangle).toBe(true); + expect(res.ggufSidecarLoaderHookApplied).toBe(true); + expect(res.counteredReaVectors).toContain("gguf-sidecar-entangle"); + + const hooked = readFileSync(join(dir, "loader.py"), "utf8"); + expect(hooked).toContain(GGUF_SIDECAR_LOADER_MARKER); + const ggufBytes = readFileSync(entry); + expect( + isGgufSidecarLoaderHookWired({ + ggufBytes, + ggufPath: entry, + }), + ).toBe(true); + expect(extractGgufSidecarMetadataNote(ggufBytes)).not.toBeNull(); + expect(ggufBytes.includes(Buffer.from(GGUF_SIDECAR_METADATA_KEY))).toBe(true); + + const ran = runLoader(dir, hooked); + expect(ran.status, ran.stderr || ran.stdout).toBe(0); + expect(ran.stdout).toMatch(/ok /); + + const verification = verifyTarget(entry, undefined, { + sidecarEntangleEnabled: true, + }); + const gapIds = (verification.defenseScoreGaps ?? []).map((g) => g.id); + expect(gapIds).not.toContain("gguf-sidecar-host-verify-only"); + + const gaps = collectDefenseScoreGaps({ + detectedFormat: "model-gguf", + sidecarEntangleEnabled: true, + ggufSidecarLoaderHookApplied: true, + }); + expect(gaps.map((g) => g.id)).not.toContain("gguf-sidecar-host-verify-only"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it.skipIf(!PY3)("protectTarget does not rewrite when sidecarEntangle is off", () => { + const dir = mkdtempSync(join(tmpdir(), "bla-gguf-loader-off-")); + try { + const entry = join(dir, "model.gguf"); + writeFileSync(entry, buildMinimalGguf()); + writeFileSync(join(dir, "sib.bin"), "sidecar-payload\n"); + writeFileSync(join(dir, "loader.py"), LOADER_BODY, "utf8"); + + const res = protectTarget({ + inputPath: entry, + outputPath: entry, + intensity: "balanced", + seed: "pt-gguf-loader-off", + inPlace: true, + }); + expect(res.sidecarEntangle).not.toBe(true); + expect(res.ggufSidecarLoaderHookApplied).toBeUndefined(); + expect(res.counteredReaVectors).not.toContain("gguf-sidecar-entangle"); + expect(readFileSync(join(dir, "loader.py"), "utf8")).toBe(LOADER_BODY); + expect(extractGgufSidecarMetadataNote(readFileSync(entry))).toBeNull(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it.skipIf(!PY3)("protectTarget does not rewrite an unknown helper file", () => { + const dir = mkdtempSync(join(tmpdir(), "bla-gguf-loader-unsafe-")); + try { + const entry = join(dir, "model.gguf"); + writeFileSync(entry, buildMinimalGguf()); + writeFileSync(join(dir, "sib.bin"), "sidecar-payload\n"); + const helper = LOADER_BODY; + writeFileSync(join(dir, "helper.py"), helper, "utf8"); + + const res = protectTarget({ + inputPath: entry, + outputPath: entry, + intensity: "balanced", + seed: "pt-gguf-loader-unsafe", + sidecarEntangle: true, + sidecarEntanglePaths: ["sib.bin"], + inPlace: true, + }); + expect(res.sidecarEntangle).toBe(true); + expect(res.ggufSidecarLoaderHookApplied).toBeUndefined(); + expect(res.counteredReaVectors).not.toContain("gguf-sidecar-entangle"); + expect(readFileSync(join(dir, "helper.py"), "utf8")).toBe(helper); + expect(extractGgufSidecarMetadataNote(readFileSync(entry))).not.toBeNull(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it.skipIf(!PY3)("hardenProject auto-wires dist/loader.py beside model.gguf", () => { + const dir = mkdtempSync(join(tmpdir(), "bla-gguf-loader-harden-")); + try { + const dist = join(dir, "dist"); + mkdirSync(dist, { recursive: true }); + writeFileSync(join(dist, "model.gguf"), buildMinimalGguf()); + writeFileSync(join(dist, "sib.bin"), "sidecar-payload\n"); + writeFileSync(join(dist, "loader.py"), LOADER_BODY, "utf8"); + + const report = hardenProject({ + directory: dir, + intensity: "balanced", + seed: "harden-gguf-loader-hook", + sidecarEntangle: true, + }); + expect(report.overallSuccess).toBe(true); + expect(report.sidecarEntangleError).toBeUndefined(); + + const outDist = join(report.outputDirectory, "dist"); + const hooked = readFileSync(join(outDist, "loader.py"), "utf8"); + expect(hooked).toContain(GGUF_SIDECAR_LOADER_MARKER); + const ggufBytes = readFileSync(join(outDist, "model.gguf")); + expect(extractGgufSidecarMetadataNote(ggufBytes)).not.toBeNull(); + expect(ggufBytes.includes(Buffer.from(GGUF_SIDECAR_METADATA_KEY))).toBe(true); + expect( + isGgufSidecarLoaderHookWired({ + ggufBytes, + ggufPath: join(outDist, "model.gguf"), + }), + ).toBe(true); + + const ran = runLoader(outDist, hooked); + expect(ran.status, ran.stderr || ran.stdout).toBe(0); + expect(ran.stdout).toMatch(/ok /); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +});