From a076c3d72ff7e62cd30e09fe3d4824ec5598c390 Mon Sep 17 00:00:00 2001 From: Timo Zhang Date: Sun, 1 Nov 2026 23:05:13 +0800 Subject: [PATCH] feat(python): structural AST weave of decoys into customer defs (#53) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make MCP weave_decoys a real pipeline attribute for Python source: when true, after tarpit prepend, structurally walk customer FunctionDef bodies and insert shared-slot touch + opaque-false decoy calls (JS ast-shield §3 analog). Selection is indent-stack + tarpit name identity skip + seeded PRNG — not a business-name rule engine. When false, preserve #51 warm-init-only behavior. Update planner maturity (python-source → woven), MCP descriptions, docs, and CHANGELOG honesty. --- CHANGELOG.md | 16 +- docs/cli-and-mcp.md | 2 + docs/counter-rea-deception.md | 2 +- docs/weave-decoy-mcp.md | 30 +- src/core/types.ts | 7 +- src/engine/protector.ts | 6 +- src/mcp/server.ts | 6 +- src/shields/formats/python-ast-weave.ts | 473 ++++++++++++++++++++++++ src/shields/formats/python.ts | 55 ++- src/shields/tarpit/tarpit-generator.ts | 9 + src/shields/weave/weave-decoy.ts | 40 +- tests/python-ast-weave.test.ts | 168 +++++++++ tests/weave-decoy-mcp.test.ts | 10 +- 13 files changed, 774 insertions(+), 50 deletions(-) create mode 100644 src/shields/formats/python-ast-weave.ts create mode 100644 tests/python-ast-weave.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index dcfe540..a0a98e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,19 +12,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 theory pack [`docs/weave-decoy-mcp.md`](docs/weave-decoy-mcp.md) (zh-friendly), MCP resource `bla://directives/weave-decoy`, tool `bla_weave_plan` (read-only `bla-weave-plan/v1` JSON), prompt `bla-weave-decoy`, and opt-in `weave_decoys` - on `bla_harden` / `bla_defend` / `bla_protect`. Labels runs and documents that - JS AST already weaves while Python remains warm-init-only after #51 — full - customer-def AST weave is a follow-up. Defensive only; no dismissible noise layers. + on `bla_harden` / `bla_defend` / `bla_protect`. Defensive only; no dismissible noise layers. +- **Python structural decoy weave** (`src/shields/formats/python-ast-weave.ts`): + when `weave_decoys: true`, after tarpit prepend, walk customer `FunctionDef` + bodies (indent-stack / structural, skip injected decoy+warm by identity) and + insert always-executed shared-slot touch plus opaque-false decoy calls — + modeled on JS `ast-shield` §3. When false, Python stays #51 warm-init-only. + Not a rule engine (no business-name → snippet table). ### Changed - +- **`weave_decoys` is a real pipeline attribute** for Python source (not label-only): + planner maturity for `python-source` is `woven`; MCP/docs/`pipelineHook.todayBehaviorZh` + describe structural weave. `.pyc` remains warm-init / trailer oriented. - **Python tarpit densification** (`tarpit-generator.ts`, `python.ts`): decoys no longer look like empty `_bla_*` honeypots — mixed mundane/scare stems, heterogeneous always-executed bodies that mutate shared state, and a live module-level warm initializer so xref tools see real reachability. Dropped greppable `BLA-SHIELD` banner (auditor still matches `runtime attestation - boundary`). + boundary`). Exposes `sharedSlotName` / `stateVarName` / `warmInitName` for weave. - **Chaff vocabulary diversification** (`chaff-generator.ts`): alternate billing / cache / scheduler / observability / locale / storage mirages so `TPM2_` / `NitroEnclave` / `hardware_enclave_stub` is not the only family. diff --git a/docs/cli-and-mcp.md b/docs/cli-and-mcp.md index ae648df..f53b25f 100644 --- a/docs/cli-and-mcp.md +++ b/docs/cli-and-mcp.md @@ -27,6 +27,8 @@ - `bla_doctor`: Diagnostics for BLA engines and agent client registrations. - `bla_weave_plan`: Read-only weave-decoy planner (`path`, optional `max_targets`) → `bla-weave-plan/v1` JSON (defensive theory checklist; no crack/RE). +`weave_decoys: true` activates structural Python customer-def weave (shared-slot touch + opaque-false decoy calls) after the #51 tarpit; when false, Python stays warm-init-only. JS AST weave remains always-on. + ### MCP Resources - `bla://directives/weave-decoy` — weave-decoy theory + executor checklist (Markdown). diff --git a/docs/counter-rea-deception.md b/docs/counter-rea-deception.md index 2187cbb..388929f 100644 --- a/docs/counter-rea-deception.md +++ b/docs/counter-rea-deception.md @@ -33,7 +33,7 @@ customer entry to intact bytes remain a separate product gap (issue #40). Empty / homogeneous scare decoys are filtered as a separate layer by WorkBuddy-class agents. Prefer **weaving** decoys into real customer paths (shared state, entry reachability, mixed vocabulary, opaque-true happy path). -JS AST already weaves; Python after #51 is warm-init densify only. +JS AST already weaves; Python `.py` with `weave_decoys:true` structurally weaves decoys into customer defs (warm-init-only when the flag is false). Executable theory + MCP: [`weave-decoy-mcp.md`](./weave-decoy-mcp.md), resource `bla://directives/weave-decoy`, tool `bla_weave_plan`, flag `weave_decoys`. diff --git a/docs/weave-decoy-mcp.md b/docs/weave-decoy-mcp.md index 801e28f..09269f0 100644 --- a/docs/weave-decoy-mcp.md +++ b/docs/weave-decoy-mcp.md @@ -15,17 +15,19 @@ | 与业务无关的 sidecar「防护层」文件 | 「独立层」分类 | 难度不升反降 | | 可 grep 的 `BLA-SHIELD` 横幅 | 明确「这是防护」证据 | 方便一刀切 | -**目标**:诱饵必须**巧妙织入**客户真实路径,使「丢掉诱饵层」等于破坏业务或验证失败。 +**目标**:诱饵必须**巧妙织入**客户真实路径,使「丢掉诱饵层」等于破坏业务或验证失败。 +**构建方式必须是结构纠缠,而不是规则引擎**(禁止「匹配业务名 → 注入固定片段」对照表;禁止同质 marker 模板当策略)。 -## 2. JS 已织入 vs Python 仍 warm-init +## 2. JS 已织入 vs Python 结构织入(gated) | 管线 | 现状 | 含义 | | :--- | :--- | :--- | | **JavaScript / TypeScript** (`ast-shield.ts`) | **已织入** | opaque-true 包裹快乐路径语句;真实函数体织入步进器与 tarpit 互调;认知装甲与 seed 纠缠 | -| **Python** (`tarpit-generator.ts` + `python.ts`) | **warm-init-only**(#51) | 稠密异构体、共享状态、词干混用、模块级 warm-init 保 xref;**尚未**把诱饵调用织进客户 `def` | +| **Python `.py`** (`python-ast-weave.ts` + tarpit) | **woven(需 `weave_decoys: true`)** | #51 稠密异构体 + warm-init 之后,对客户 `FunctionDef` 做结构织入:共享槽 touch + opaque-false 诱饵调用;按 tarpit 名表**身份**跳过已注入 decoy/warm(非业务名规则) | +| **Python `.pyc`** | warm-init / trailer 为主 | 客户入口织入仍有限 | | 若干二进制 / sidecar 格式 | sidecar / 附加块 | 原则仍适用;完整入口纠缠列为后续 | -本 PR 把理论与 MCP 执行面备齐;**完整 Python(及更多格式)AST 织入客户定义**留给 follow-up PR。 +`weave_decoys: false` 时 Python 行为与 #51 一致(仅 warm-init,不改客户 `def` 体)。 ## 3. 织入原则(执行时必须遵守) @@ -33,7 +35,7 @@ 2. **客户入口可达** — 边必须从入口、路由、导出 API 进入诱饵;仅靠顶层 warm-init 不够对抗「独立层」过滤。 3. **词干混用** — 约 2/3 平凡业务名 + 约 1/3 安全诱饵名。 4. **快乐路径不透明真** — 真实语句用 opaque-true 守卫;禁止只把诱饵放进永不执行的死图。 -5. **调用图纠缠** — 在真实函数体内织入步进 / 互调 / 影子堆搅动。 +5. **调用图纠缠** — 在真实函数体内织入步进 / 互调 / 影子堆搅动(结构扫描,非规则表)。 6. **可选自密封** — 入口与完整字节 / digest 绑定(产品缺口见 issue #40)。 ## 4. 明确禁止 @@ -42,6 +44,7 @@ - 可 grep 的 `BLA-SHIELD` / `_bla_` 横幅(审计短语 `runtime attestation boundary` 可保留) - 仅 TEE 词表的同质包 - 永不调用的 `pass` 桩冒充「加固」 +- 规则式「名字匹配 → 注入」与同质 marker 模板当防护策略 - 任何进攻性内容 ## 5. 后续 AI 执行清单(不依赖聊天记录) @@ -50,16 +53,16 @@ 2. `bla_weave_plan` → `{ "path": "" }`,保存返回的 JSON 工单 3. `bla_harden` / `bla_defend` / `bla_protect` 传 **`weave_decoys: true`** 4. JS:验证现有 AST 织入,**不要**再堆 sidecar 噪声 -5. Python:保持稠密 + warm-init;**不要**用空 scare 模块填补缺口;标注 follow-up AST weave +5. Python `.py`:确认客户 `def` 体内出现共享槽 touch / 诱饵 xref;**不要**用空 scare 模块填补 6. 按 §6 验收;失败则回滚噪声改动 ## 6. 验收(apply 后必须通过) 1. **无禁止标记** — 无 `_bla_` / `BLA-SHIELD` 横幅 2. **词干混用** — 诱饵名非 100% TEE/enclave -3. **共享状态变异** — 诱饵体修改与业务相关的共享槽 -4. **客户入口边** — 静态 xref 从客户入口到达诱饵(JS 必须;Python warm-init 仅为过渡) -5. **opaque-true 缠绕** — 删除守卫或诱饵调用会改变行为或验证结果 +3. **共享状态变异** — 诱饵体与织入语句修改同一共享槽 +4. **客户入口边** — 静态 xref 从客户入口到达诱饵(JS;Python 在 `weave_decoys:true` 下经由客户 def 织入) +5. **opaque 缠绕** — 删除守卫或诱饵调用会改变行为或验证结果 6. **抗独立层过滤** — 不能仅凭空体 / 同质 scare / 未引用启发式整层丢弃 ## 7. MCP 表面 @@ -68,14 +71,13 @@ | :--- | :--- | | Resource `bla://directives/weave-decoy` | 返回本理论的可执行 Markdown | | Tool `bla_weave_plan` | 只读:分析路径 → `bla-weave-plan/v1` JSON | -| `bla_harden` / `bla_defend` / `bla_protect` 参数 `weave_decoys` | 标注本次加固并挂接未来 AST 织入钩子 | +| `bla_harden` / `bla_defend` / `bla_protect` 参数 `weave_decoys` | **真实管线属性**:激活 Python 客户 def 结构织入(JS 始终 AST 织入);非仅报告布尔标注 | -实现入口:`src/shields/weave/weave-decoy.ts`、`src/mcp/server.ts`。 +实现入口:`src/shields/weave/weave-decoy.ts`、`src/shields/formats/python-ast-weave.ts`、`src/mcp/server.ts`。 -## 8. Follow-up PR(本 PR 不做) +## 8. Follow-up -- 将诱饵调用 **AST 织入 Python 客户函数体**(超越 warm-init) -- 扩展更多格式的入口纠缠验收 +- `.pyc` / 更多格式的客户入口纠缠验收 - 可选 self-seal 与 `defenseScore` 对 onefile 诚实性(#40) ## 相关文档 diff --git a/src/core/types.ts b/src/core/types.ts index 35a1f98..2fcd0e2 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -157,8 +157,9 @@ export interface ProtectionOptions { */ readonly llmDirectives?: boolean; /** - * When true, label the run for weave-decoy hooks (JS AST already weaves; - * Python remains warm-init-only until follow-up). Default false. + * When true, activate weave-decoy entanglement: JS AST weave (always) plus + * structural Python customer-def weave (shared-slot + opaque-false decoy xref). + * When false, Python stays warm-init-only. Default false. */ readonly weaveDecoys?: boolean; } @@ -182,7 +183,7 @@ export interface ProtectionResult { readonly outputPath: string; readonly detectedFormat: ArtifactFormat; readonly intensity: ProtectionIntensity; - /** Present when protect was called with weaveDecoys:true (label / future hooks). */ + /** Present when protect was called with weaveDecoys:true (Python customer-def weave active). */ readonly weaveDecoys?: boolean; readonly seed: string; readonly originalSha256: string; diff --git a/src/engine/protector.ts b/src/engine/protector.ts index 73b3e9b..9b86a86 100644 --- a/src/engine/protector.ts +++ b/src/engine/protector.ts @@ -342,7 +342,11 @@ function protectTargetInner(options: ProtectionOptions): ProtectionResult { counteredVectors.add("obligation-ledger-cloning"); } else if (format === "python-source") { const rawPy = readFileSync(absInput, "utf8"); - const res = protectPythonSource(rawPy, { intensity, seed }); + const res = protectPythonSource(rawPy, { + intensity, + seed, + weaveDecoys: options.weaveDecoys === true, + }); writeFileSync(absOutput, res.code, "utf8"); decoyFunctionsInjected = res.decoyFunctionsInjected; fakeCrossReferencesCreated = res.crossReferenceEdges; diff --git a/src/mcp/server.ts b/src/mcp/server.ts index ffc189e..462674d 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -126,7 +126,7 @@ export const BLA_MCP_TOOLS: readonly McpToolDefinition[] = Object.freeze([ weave_decoys: { type: "boolean", description: - "Opt-in weave-decoy labeling/hooks: entangle decoys with real customer paths (JS already weaves; Python warm-init-only until follow-up AST weave). See bla://directives/weave-decoy. Default false.", + "Opt-in weave-decoy entanglement: JS AST weave (always) + structural Python customer-def weave (shared-slot + opaque-false decoy xref). When false, Python stays warm-init-only. See bla://directives/weave-decoy. Default false.", }, seed: { type: "string", @@ -167,7 +167,7 @@ export const BLA_MCP_TOOLS: readonly McpToolDefinition[] = Object.freeze([ weave_decoys: { type: "boolean", description: - "Opt-in weave-decoy labeling/hooks (shared-state / entry-reachable decoys; not dismissible noise). Default false.", + "Opt-in weave-decoy entanglement (shared-state / entry-reachable decoys into customer paths; not dismissible noise). Python .py structural weave when true. Default false.", }, seed: { type: "string", @@ -217,7 +217,7 @@ export const BLA_MCP_TOOLS: readonly McpToolDefinition[] = Object.freeze([ weave_decoys: { type: "boolean", description: - "Opt-in weave-decoy labeling/hooks (shared-state / entry-reachable decoys; not dismissible noise). Default false.", + "Opt-in weave-decoy entanglement (shared-state / entry-reachable decoys into customer paths; not dismissible noise). Python .py structural weave when true. Default false.", }, seed: { type: "string", diff --git a/src/shields/formats/python-ast-weave.ts b/src/shields/formats/python-ast-weave.ts new file mode 100644 index 0000000..7c812f8 --- /dev/null +++ b/src/shields/formats/python-ast-weave.ts @@ -0,0 +1,473 @@ +/** + * Structural Python decoy weave — entangle tarpit decoys into customer `def` bodies. + * + * Modeled on JS `ast-shield.ts` §3: walk real function bodies, always-touch shared + * state, and unshift opaque-false guarded decoy calls so xref from customer entry + * reaches decoys. Selection is structural (FunctionDef sites minus injected tarpit + * names by identity) + seeded PRNG — not keyword/business-name rule tables. + * + * No greppable `_bla_` / `BLA-SHIELD` banners. No Python AST dependency: indent-stack + * scan with string/comment awareness, preserving encoding and EOL. + */ + +import { SeededPrng } from "../../core/prng.js"; +import type { ProtectionIntensity } from "../../core/types.js"; +import { generatePythonOpaqueFalseExpression } from "../tarpit/opaque-predicates.js"; + +export interface PythonAstWeaveInput { + readonly source: string; + /** Injected tarpit / warm-init names — skipped by identity, not by regex heuristics. */ + readonly skipDefNames: readonly string[]; + readonly decoyFunctionNames: readonly string[]; + readonly sharedSlotName: string; + readonly stateVarName: string; + /** Warm-init def identity to exclude from the call pool. */ + readonly warmInitName?: string; + readonly prng: SeededPrng; + readonly intensity?: ProtectionIntensity; +} + +export interface PythonAstWeaveResult { + readonly code: string; + readonly customerDefsWoven: number; + readonly crossReferenceEdgesAdded: number; +} + +interface WeaveSite { + readonly name: string; + readonly insertOffset: number; + readonly bodyIndent: string; +} + +const VALID_STRING_PREFIXES = new Set([ + "", + "r", + "u", + "f", + "b", + "fr", + "rf", + "br", + "rb", +]); + +function matchStringStart( + source: string, + offset: number, +): { readonly quote: string; readonly contentStart: number } | null { + if (offset > 0 && /[A-Za-z0-9_]/.test(source[offset - 1]!)) return null; + const slice = source.slice(offset, offset + 5); + const m = /^([A-Za-z]{0,2})("""|'''|"|')/.exec(slice); + if (!m) return null; + const rawPrefix = m[1] ?? ""; + if (!VALID_STRING_PREFIXES.has(rawPrefix.toLowerCase())) return null; + const afterPrefix = offset + rawPrefix.length; + const quote = source.startsWith('"""', afterPrefix) + ? '"""' + : source.startsWith("'''", afterPrefix) + ? "'''" + : source[afterPrefix]!; + return { quote, contentStart: afterPrefix + quote.length }; +} + +function scanStringEnd(source: string, startOffset: number): number { + const match = matchStringStart(source, startOffset); + if (!match) return startOffset + 1; + const { quote, contentStart } = match; + let i = contentStart; + if (quote.length === 3) { + while (i < source.length) { + if (source.startsWith(quote, i)) return i + 3; + if (source[i] === "\\") { + if (i + 1 >= source.length) return source.length; + if (source[i + 1] === "\r" && source[i + 2] === "\n") i += 3; + else i += 2; + continue; + } + i += 1; + } + return source.length; + } + while (i < source.length) { + const ch = source[i]!; + if (ch === quote) return i + 1; + if (ch === "\n" || ch === "\r") return i; + if (ch === "\\") { + if (i + 1 >= source.length) return source.length; + if (source[i + 1] === "\r" && source[i + 2] === "\n") i += 3; + else i += 2; + continue; + } + i += 1; + } + return source.length; +} + +function matchKeywordAt(source: string, offset: number, keyword: string): boolean { + if (!source.startsWith(keyword, offset)) return false; + if (offset > 0 && /[A-Za-z0-9_]/.test(source[offset - 1]!)) return false; + const after = source[offset + keyword.length]; + if (after !== undefined && /[A-Za-z0-9_]/.test(after)) return false; + return true; +} + +function lineIndentAt(source: string, lineStart: number): string { + let i = lineStart; + while (i < source.length) { + const ch = source[i]!; + if (ch === " " || ch === "\t") i += 1; + else break; + } + return source.slice(lineStart, i); +} + +function advanceToNextLineStart(source: string, offset: number): number { + let i = offset; + while (i < source.length) { + if (source[i] === "\r" && source[i + 1] === "\n") return i + 2; + if (source[i] === "\n" || source[i] === "\r") return i + 1; + i += 1; + } + return source.length; +} + +function isBlankOrCommentLine(source: string, lineStart: number): boolean { + let i = lineStart; + while (i < source.length) { + const ch = source[i]!; + if (ch === " " || ch === "\t" || ch === "\f") { + i += 1; + continue; + } + if (ch === "#" || ch === "\n" || ch === "\r") return true; + return false; + } + return true; +} + +/** + * Find the colon that ends a `def` / `async def` header, respecting nested + * brackets and strings (structural, not a business-name regex table). + */ +function findHeaderColon(source: string, defKeywordOffset: number): number | null { + let i = defKeywordOffset; + let paren = 0; + let bracket = 0; + let brace = 0; + while (i < source.length) { + const str = matchStringStart(source, i); + if (str) { + i = scanStringEnd(source, i); + continue; + } + const ch = source[i]!; + if (ch === "#") { + i = advanceToNextLineStart(source, i); + continue; + } + if (ch === "(") { + paren += 1; + i += 1; + continue; + } + if (ch === ")") { + paren = Math.max(0, paren - 1); + i += 1; + continue; + } + if (ch === "[") { + bracket += 1; + i += 1; + continue; + } + if (ch === "]") { + bracket = Math.max(0, bracket - 1); + i += 1; + continue; + } + if (ch === "{") { + brace += 1; + i += 1; + continue; + } + if (ch === "}") { + brace = Math.max(0, brace - 1); + i += 1; + continue; + } + if (ch === ":" && paren === 0 && bracket === 0 && brace === 0) { + return i; + } + i += 1; + } + return null; +} + +function skipDocstringAfterColon( + source: string, + afterColon: number, + bodyIndent: string, +): number { + // Skip horizontal space / comment on the def line, then newline. + let i = afterColon + 1; + while (i < source.length && (source[i] === " " || source[i] === "\t")) i += 1; + if (source[i] === "#") { + i = advanceToNextLineStart(source, i); + } else if (source[i] === "\r" || source[i] === "\n") { + i = advanceToNextLineStart(source, i); + } else { + // One-liner suite on the same line as colon — caller should skip site. + return afterColon + 1; + } + + while (i < source.length && isBlankOrCommentLine(source, i)) { + i = advanceToNextLineStart(source, i); + } + if (i >= source.length) return i; + + const indent = lineIndentAt(source, i); + if (indent !== bodyIndent) return i; + + const contentStart = i + indent.length; + const lit = matchStringStart(source, contentStart); + if (!lit) return i; + + // Docstring only when the first body statement is a bare string literal. + const end = scanStringEnd(source, contentStart); + let j = end; + while (j < source.length && (source[j] === " " || source[j] === "\t")) j += 1; + if (source[j] === "#") { + j = advanceToNextLineStart(source, j); + } else if (source[j] === "\r" || source[j] === "\n") { + j = advanceToNextLineStart(source, j); + } else if (source[j] === ";") { + j += 1; + while (j < source.length && (source[j] === " " || source[j] === "\t")) j += 1; + if (source[j] === "\r" || source[j] === "\n") { + j = advanceToNextLineStart(source, j); + } + } else { + // Compound expression starting with a string — not a docstring statement. + return i; + } + return j; +} + +function detectBodyIndent( + source: string, + afterColon: number, + defIndent: string, +): { readonly bodyIndent: string; readonly oneLiner: boolean } | null { + let i = afterColon + 1; + while (i < source.length && (source[i] === " " || source[i] === "\t")) i += 1; + if (source[i] === "#") { + i = advanceToNextLineStart(source, i); + } else if (source[i] === "\r" || source[i] === "\n") { + i = advanceToNextLineStart(source, i); + } else { + return { bodyIndent: defIndent + " ", oneLiner: true }; + } + + while (i < source.length && isBlankOrCommentLine(source, i)) { + i = advanceToNextLineStart(source, i); + } + if (i >= source.length) { + return { bodyIndent: defIndent + " ", oneLiner: false }; + } + const indent = lineIndentAt(source, i); + if (indent.length <= defIndent.length) { + // Empty / ellipsis body edge — still allow insert with synthetic indent. + return { bodyIndent: defIndent + (defIndent.includes("\t") ? "\t" : " "), oneLiner: false }; + } + return { bodyIndent: indent, oneLiner: false }; +} + +/** + * Structural scan: every customer FunctionDef / AsyncFunctionDef whose name is + * not in the tarpit identity skip-set. No keyword classifiers on business names. + */ +export function findCustomerPythonFunctionDefs( + source: string, + skipDefNames: ReadonlySet, +): WeaveSite[] { + const sites: WeaveSite[] = []; + let i = 0; + let lineStart = 0; + let atLineStart = true; + + while (i < source.length) { + if (atLineStart) { + const indent = lineIndentAt(source, lineStart); + let cur = lineStart + indent.length; + if ( + cur < source.length && + source[cur] !== "#" && + source[cur] !== "\n" && + source[cur] !== "\r" + ) { + let asyncDef = false; + if (matchKeywordAt(source, cur, "async")) { + let afterAsync = cur + 5; + while ( + afterAsync < source.length && + (source[afterAsync] === " " || source[afterAsync] === "\t") + ) { + afterAsync += 1; + } + if (matchKeywordAt(source, afterAsync, "def")) { + asyncDef = true; + cur = afterAsync; + } + } + if (asyncDef || matchKeywordAt(source, cur, "def")) { + const defKw = cur; + cur = defKw + 3; + while (cur < source.length && (source[cur] === " " || source[cur] === "\t")) { + cur += 1; + } + const nameMatch = /^[A-Za-z_][A-Za-z0-9_]*/.exec(source.slice(cur)); + if (nameMatch) { + const name = nameMatch[0]!; + const colon = findHeaderColon(source, defKw); + if (colon !== null && !skipDefNames.has(name)) { + const bodyInfo = detectBodyIndent(source, colon, indent); + if (bodyInfo && !bodyInfo.oneLiner) { + const insertOffset = skipDocstringAfterColon( + source, + colon, + bodyInfo.bodyIndent, + ); + sites.push({ + name, + insertOffset, + bodyIndent: bodyInfo.bodyIndent, + }); + } + } + } + } + } + } + + const str = matchStringStart(source, i); + if (str) { + i = scanStringEnd(source, i); + atLineStart = false; + continue; + } + if (source[i] === "#") { + i = advanceToNextLineStart(source, i); + lineStart = i; + atLineStart = true; + continue; + } + if (source[i] === "\r" && source[i + 1] === "\n") { + i += 2; + lineStart = i; + atLineStart = true; + continue; + } + if (source[i] === "\n" || source[i] === "\r") { + i += 1; + lineStart = i; + atLineStart = true; + continue; + } + atLineStart = false; + i += 1; + } + + return sites; +} + +function decoyPoolForWeave( + decoyFunctionNames: readonly string[], + warmInitName: string | undefined, +): string[] { + // Prefer tarpit decoys over the warm-init helper by identity (not name heuristics). + const pure = warmInitName + ? decoyFunctionNames.filter((n) => n !== warmInitName) + : [...decoyFunctionNames]; + return pure.length >= 1 ? pure : [...decoyFunctionNames]; +} + +/** + * Weave shared-slot touch + opaque-false decoy calls into customer function bodies. + */ +export function weavePythonDecoyCalls(input: PythonAstWeaveInput): PythonAstWeaveResult { + const { + source, + skipDefNames, + decoyFunctionNames, + sharedSlotName, + stateVarName, + warmInitName, + prng, + intensity = "balanced", + } = input; + + if (decoyFunctionNames.length === 0) { + return { code: source, customerDefsWoven: 0, crossReferenceEdgesAdded: 0 }; + } + + const skip = new Set(skipDefNames); + const sites = findCustomerPythonFunctionDefs(source, skip); + if (sites.length === 0) { + return { code: source, customerDefsWoven: 0, crossReferenceEdgesAdded: 0 }; + } + + const eol = source.includes("\r\n") ? "\r\n" : "\n"; + const pool = decoyPoolForWeave(decoyFunctionNames, warmInitName); + if (pool.length === 0) { + return { code: source, customerDefsWoven: 0, crossReferenceEdgesAdded: 0 }; + } + + const deepCalls = + intensity === "stealth" ? 1 : intensity === "balanced" ? 1 : 2; + + let crossReferenceEdgesAdded = 0; + // Apply from end to start so earlier offsets stay valid. + const ordered = [...sites].sort((a, b) => b.insertOffset - a.insertOffset); + let code = source; + let woven = 0; + + for (const site of ordered) { + const dTouch = prng.pick(pool); + const lines: string[] = []; + // Always-executed shared-state touch — cheap, no deep recursion (JS §3 stepper analog). + lines.push( + `${site.bodyIndent}${sharedSlotName}["epoch"] = (${sharedSlotName}.get("epoch", 0) ^ hash(${dTouch}.__name__)) & 0xFFFFFFFF`, + ); + crossReferenceEdgesAdded += 1; + + if (pool.length >= 1 && intensity !== "stealth") { + const d1 = prng.pick(pool); + const d2 = pool.length >= 2 ? prng.pick(pool) : d1; + const seed1 = prng.nextInt(1, 255); + const seed2 = prng.nextInt(1, 255); + const opaque = generatePythonOpaqueFalseExpression(prng, stateVarName); + lines.push(`${site.bodyIndent}if ${opaque}:`); + if (deepCalls >= 2 && pool.length >= 2) { + lines.push( + `${site.bodyIndent} _ = ${d1}(${seed1}, 0) ^ ${d2}(${seed2}, 0)`, + ); + crossReferenceEdgesAdded += 2; + } else { + lines.push(`${site.bodyIndent} _ = ${d1}(${seed1}, 0)`); + crossReferenceEdgesAdded += 1; + } + } else if (intensity === "stealth") { + // Stealth: always-touch only (still creates customer→decoy xref via __name__). + } + + const block = lines.join(eol) + eol; + code = code.slice(0, site.insertOffset) + block + code.slice(site.insertOffset); + woven += 1; + } + + return { + code, + customerDefsWoven: woven, + crossReferenceEdgesAdded, + }; +} diff --git a/src/shields/formats/python.ts b/src/shields/formats/python.ts index 23bdff3..f6168bc 100644 --- a/src/shields/formats/python.ts +++ b/src/shields/formats/python.ts @@ -3,6 +3,7 @@ import type { ProtectionIntensity } from "../../core/types.js"; import { generateChaffPayloads } from "../chaff/chaff-generator.js"; import { buildCanaryAndDiagnosticPayloads } from "../deception/format-payloads.js"; import { generatePythonTarpit } from "../tarpit/tarpit-generator.js"; +import { weavePythonDecoyCalls } from "./python-ast-weave.js"; export class InvalidPythonSourceError extends Error { constructor(message: string) { @@ -660,6 +661,11 @@ export function protectPythonSource( options: { readonly intensity?: ProtectionIntensity; readonly seed?: string | number; + /** + * When true, structurally weave decoy entanglement into customer `def` bodies + * (shared-slot touch + opaque-false calls). When false, preserve warm-init-only. + */ + readonly weaveDecoys?: boolean; } = {}, ): { readonly code: string; @@ -668,8 +674,11 @@ export function protectPythonSource( readonly chaffPayloadsEmbedded: number; readonly canaryTokensEmbedded: number; readonly diagnosticsEmbedded: number; + readonly customerDefsWoven: number; + readonly weaveDecoys: boolean; } { const intensity = options.intensity ?? "balanced"; + const weaveDecoys = options.weaveDecoys === true; const prng = new SeededPrng(options.seed ?? "bla-python-shield-v1"); const chaffPayloads = generateChaffPayloads(prng, intensity); const tarpit = generatePythonTarpit(prng, intensity, chaffPayloads); @@ -706,17 +715,57 @@ export function protectPythonSource( " return AMBIGUOUS_TRANSPORT[_i]", ].join(eol); - const combined = [head, tarpitNormalized, deceptionBlock, tail] + let customerTail = tail; + let crossReferenceEdges = tarpit.crossReferenceEdges; + let customerDefsWoven = 0; + + if ( + weaveDecoys && + tarpit.sharedSlotName && + tarpit.stateVarName && + tarpit.decoyFunctionNames.length > 0 && + customerTail.length > 0 + ) { + // Weave only the customer tail so nested helpers inside injected tarpit defs + // (e.g. `_fold`) are never touched. Skip set is tarpit identities for safety + // if a customer name collides. + const skipDefNames = [ + ...tarpit.decoyFunctionNames, + ...(tarpit.warmInitName ? [tarpit.warmInitName] : []), + "probe_canary_honeypot", + ]; + const woven = weavePythonDecoyCalls({ + source: customerTail.endsWith("\n") || customerTail.endsWith("\r") + ? customerTail + : customerTail + eol, + skipDefNames, + decoyFunctionNames: tarpit.decoyFunctionNames, + sharedSlotName: tarpit.sharedSlotName, + stateVarName: tarpit.stateVarName, + warmInitName: tarpit.warmInitName, + prng, + intensity, + }); + customerTail = woven.code.replace(/(?:\r?\n)+$/, ""); + customerDefsWoven = woven.customerDefsWoven; + crossReferenceEdges += woven.crossReferenceEdgesAdded; + } + + const combined = [head, tarpitNormalized, deceptionBlock, customerTail] .filter((part) => part.length > 0) .join(eol + eol); return { - code: combined + eol, + code: combined.endsWith("\n") || combined.endsWith("\r") + ? combined + : combined + eol, decoyFunctionsInjected: tarpit.decoyFunctionNames.length, - crossReferenceEdges: tarpit.crossReferenceEdges, + crossReferenceEdges, chaffPayloadsEmbedded: chaffPayloads.length, canaryTokensEmbedded: deception.canary.syntheticCanaryTokens.length, diagnosticsEmbedded: deception.diagnostics.diagnosticsInjected.length, + customerDefsWoven, + weaveDecoys, }; } diff --git a/src/shields/tarpit/tarpit-generator.ts b/src/shields/tarpit/tarpit-generator.ts index 3853ecd..b176138 100644 --- a/src/shields/tarpit/tarpit-generator.ts +++ b/src/shields/tarpit/tarpit-generator.ts @@ -11,6 +11,12 @@ export interface TarpitGenerationResult { readonly decoyFunctionNames: readonly string[]; readonly crossReferenceEdges: number; readonly semanticBudgetTriggers: readonly string[]; + /** Python shared dict mutated by decoys / weave (absent for JS tarpit). */ + readonly sharedSlotName?: string; + /** Python opaque-predicate seed / state int name. */ + readonly stateVarName?: string; + /** Python module warm-init function name (skipped by customer-def weave). */ + readonly warmInitName?: string; } /** Mixed honey + mundane stems so a single TEE/TPM grep cannot dismiss the whole tarpit. */ @@ -355,5 +361,8 @@ export function generatePythonTarpit( decoyFunctionNames: [...decoyNames, warmName], crossReferenceEdges: crossRefEdges, semanticBudgetTriggers: ["python-marshal-code-graph-tarpit"], + sharedSlotName: slotVar, + stateVarName: stateVar, + warmInitName: warmName, }; } diff --git a/src/shields/weave/weave-decoy.ts b/src/shields/weave/weave-decoy.ts index 6d96327..d48e4bc 100644 --- a/src/shields/weave/weave-decoy.ts +++ b/src/shields/weave/weave-decoy.ts @@ -180,7 +180,7 @@ export const WEAVE_ACCEPTANCE_TESTS: readonly WeaveAcceptanceTest[] = id: "customer-entry-edges", titleZh: "客户入口边", checkEn: - "Static xref from customer entry / exported handlers reaches at least one decoy call (not warm-init alone for JS; Python warm-init is interim only).", + "Static xref from customer entry / exported handlers reaches at least one decoy call (JS always; Python with weave_decoys:true via customer-def weave, not warm-init alone).", }, { id: "opaque-true-entanglement", @@ -216,7 +216,7 @@ export const WEAVE_EXECUTOR_CHECKLIST: readonly WeaveChecklistStep[] = phase: "apply", titleZh: "走 harden 管线旗标", actionEn: - "Call `bla_harden` / `bla_defend` / `bla_protect` with `weave_decoys: true` so the run is labeled and future AST weave hooks activate.", + "Call `bla_harden` / `bla_defend` / `bla_protect` with `weave_decoys: true` so Python structural customer-def weave runs (JS AST weave is always on).", }, { id: "js-entangle", @@ -226,11 +226,11 @@ export const WEAVE_EXECUTOR_CHECKLIST: readonly WeaveChecklistStep[] = "For JS/TS: existing `ast-shield` already weaves opaque-true + tarpit calls into real function bodies — verify, do not add a second noise sidecar.", }, { - id: "python-gap", + id: "python-weave", phase: "apply", - titleZh: "Python 缺口标注", + titleZh: "确认 Python 结构织入", actionEn: - "For Python: today is warm-init-only after densify (#51). Full weave into customer defs is a follow-up PR — do not pile empty scare modules as a substitute.", + "For Python `.py`: with weave_decoys:true, `python-ast-weave` inserts shared-slot touch + opaque-false decoy calls into customer defs (identity-skip tarpit names). Do not pile empty scare modules.", }, { id: "acceptance", @@ -248,6 +248,8 @@ function maturityForFormat(format: ArtifactFormat): WeaveMaturity { case "html-bundle": return "woven"; case "python-source": + // Structural customer-def weave lands when weave_decoys:true (warm-init remains when false). + return "woven"; case "python-pyc": return "warm-init-only"; case "electron-asar": @@ -281,9 +283,9 @@ function statusForMaturity(maturity: WeaveMaturity): WeavePlanStatus { function notesForMaturity(maturity: WeaveMaturity): string { switch (maturity) { case "woven": - return "JS AST 已将诱饵织入真实函数体(opaque-true + tarpit/xref)。保持此路径,禁止再加可剥离噪声层。"; + return "JS AST / Python 结构织入已将诱饵织入真实函数体(opaque + tarpit/xref)。Python 需 weave_decoys:true;保持此路径,禁止再加可剥离噪声层。"; case "warm-init-only": - return "Python 在 #51 后有稠密诱饵 + 模块 warm-init,但尚未织入客户 def 调用图;后续 PR 做 AST weave。"; + return "该格式目前以稠密诱饵 + 模块 warm-init / sidecar 为主(如 .pyc);完整客户入口织入仍有限。"; case "sidecar-decoy": return "格式以 sidecar / 附加块诱饵为主;织入原则仍适用:避免空同质 scare,保持可达与词干混用。"; case "gap": @@ -297,14 +299,14 @@ function actionsForMaturity(maturity: WeaveMaturity): readonly string[] { switch (maturity) { case "woven": return Object.freeze([ - "Run bla_harden/bla_protect with weave_decoys:true (label + future hooks).", - "Verify opaque-true entanglement and mixed stems on output.", + "Run bla_harden/bla_protect with weave_decoys:true (JS always weaves; Python structural customer-def weave).", + "Verify opaque entanglement, shared-slot touch, and mixed stems on output.", ]); case "warm-init-only": return Object.freeze([ "Keep densified tarpit + warm-init; do not add empty honeypot modules.", - "Track follow-up: AST weave decoy calls into customer Python defs.", - "Optional: self-seal binding (issue #40) after weave lands.", + "Prefer .py sources with weave_decoys:true when customer-def weave is required.", + "Optional: self-seal binding (issue #40).", ]); case "sidecar-decoy": return Object.freeze([ @@ -456,9 +458,9 @@ export function buildWeavePlan( const overallStatus = overallFromTargets(targets); const summaryZh = overallStatus === "ready" - ? "目标以 JS 织入成熟度为主;可直接 bla_harden(weave_decoys:true) 并跑验收。" + ? "目标以 JS/Python 源码织入成熟度为主;可直接 bla_harden(weave_decoys:true) 并跑验收。" : overallStatus === "partial" - ? "部分格式已织入或仅有 warm-init/sidecar;按 checklist 执行,Python 客户 def 织入留给后续 PR。" + ? "部分格式已织入或仅有 warm-init/sidecar;按 checklist 执行(.py 需 weave_decoys:true 激活客户 def 织入)。" : overallStatus === "gap" ? "存在尚无织入钩子的格式;先标准 harden,勿堆噪声层。" : "未发现可分析产物;请指向源码或构建目录。"; @@ -484,9 +486,9 @@ const PIPELINE_HOOK = Object.freeze({ mcpFlag: "weave_decoys" as const, hardenOption: "weaveDecoys" as const, todayBehaviorZh: - "weave_decoys=true 时:JS 保持现有 AST 织入;Python 保持 #51 稠密 + warm-init(尚未改客户 def);报告标注 weaveDecoys。完整 Python/多格式 AST 织入为后续 PR。", + "weave_decoys=true 时:JS 保持现有 AST 织入;Python `.py` 在稠密 tarpit + warm-init 之后,对客户 FunctionDef 做结构织入(共享槽 touch + opaque-false 诱饵调用,按身份跳过已注入 decoy/warm)。false 时 Python 仍为 warm-init-only。报告标注 weaveDecoys。", followUpZh: - "Follow-up:把诱饵调用织进 Python 客户函数体;可选 self-seal;扩展 sidecar 格式的入口纠缠验收。", + "Follow-up:.pyc 客户入口织入;可选 self-seal;扩展 sidecar 格式的入口纠缠验收。", }); /** Markdown theory pack returned by MCP resources/read (Chinese-friendly). */ @@ -564,7 +566,13 @@ export function inspectWeaveHints(source: string, format: ArtifactFormat): { if (/runtime attestation boundary/.test(source)) { hints.push("python-tarpit-banner-phrase"); } - return { maturity: "warm-init-only", hints }; + if (/hash\(\w+\.__name__\)/.test(source) && /\["epoch"\]/.test(source)) { + hints.push("python-customer-def-weave"); + } + return { + maturity: format === "python-source" ? "woven" : "warm-init-only", + hints, + }; } return { maturity: maturityForFormat(format), hints }; } diff --git a/tests/python-ast-weave.test.ts b/tests/python-ast-weave.test.ts new file mode 100644 index 0000000..7c0329e --- /dev/null +++ b/tests/python-ast-weave.test.ts @@ -0,0 +1,168 @@ +import { execFileSync } from "node:child_process"; +import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { assertNoForbiddenMarkers } from "../src/core/identifiers.js"; +import { protectPythonSource } from "../src/shields/formats/python.js"; +import { + findCustomerPythonFunctionDefs, + weavePythonDecoyCalls, +} from "../src/shields/formats/python-ast-weave.js"; +import { SeededPrng } from "../src/core/prng.js"; + +const temps: string[] = []; +afterEach(() => { + for (const d of temps.splice(0)) { + rmSync(d, { recursive: true, force: true }); + } +}); + +function extractDefBody(source: string, defName: string): string | null { + const lines = source.split(/\r?\n/); + const header = new RegExp(`^(async\\s+)?def\\s+${defName}\\s*\\(`); + let start = -1; + let defIndent = ""; + for (let i = 0; i < lines.length; i++) { + const line = lines[i]!; + const indent = /^([ \t]*)/.exec(line)?.[1] ?? ""; + if (header.test(line.slice(indent.length))) { + start = i; + defIndent = indent; + break; + } + } + if (start < 0) return null; + const body: string[] = []; + for (let i = start + 1; i < lines.length; i++) { + const line = lines[i]!; + if (line.trim() === "") { + body.push(line); + continue; + } + const indent = /^([ \t]*)/.exec(line)?.[1] ?? ""; + if (indent.length <= defIndent.length) break; + body.push(line); + } + return body.join("\n"); +} + +describe("python structural decoy weave", () => { + it("finds customer FunctionDefs structurally and skips tarpit identities", () => { + const src = [ + "def decoy_a(seed=0, depth=0):", + " return seed", + "", + "def calculate(x, y):", + ' """sum"""', + " return x + y", + "", + "async def fetch_row(id):", + " return id", + "", + ].join("\n"); + const sites = findCustomerPythonFunctionDefs(src, new Set(["decoy_a"])); + expect(sites.map((s) => s.name).sort()).toEqual(["calculate", "fetch_row"]); + }); + + it("weaveDecoys:true puts decoy xref inside customer def calculate", () => { + const customer = [ + "def calculate(a, b):", + ' """business entry"""', + " return a + b", + "", + "def helper(z):", + " return z * 2", + "", + ].join("\n"); + const out = protectPythonSource(customer, { + intensity: "balanced", + seed: "py-weave-on-1", + weaveDecoys: true, + }); + expect(out.weaveDecoys).toBe(true); + expect(out.customerDefsWoven).toBeGreaterThanOrEqual(2); + assertNoForbiddenMarkers(out.code, "python-weave-on"); + + const body = extractDefBody(out.code, "calculate"); + expect(body).toBeTruthy(); + expect(body!).toMatch(/\["epoch"\]/); + expect(body!).toMatch(/hash\(\w+\.__name__\)/); + // Opaque-false guarded call should reference a decoy name as a call. + expect(body!).toMatch(/\w+\(\d+,\s*0\)/); + + // Decoy defs themselves should not get a second weave layer of customer pattern + // (skip by identity): their bodies already have epoch mutation from tarpit. + }); + + it("weaveDecoys:false leaves customer def without decoy call entanglement", () => { + const customer = [ + "def calculate(a, b):", + " return a + b", + "", + ].join("\n"); + const out = protectPythonSource(customer, { + intensity: "balanced", + seed: "py-weave-off-1", + weaveDecoys: false, + }); + expect(out.weaveDecoys).toBe(false); + expect(out.customerDefsWoven).toBe(0); + const body = extractDefBody(out.code, "calculate"); + expect(body).toBeTruthy(); + expect(body!).toMatch(/return a \+ b/); + expect(body!).not.toMatch(/hash\(\w+\.__name__\)/); + expect(body!).not.toMatch(/\w+\(\d+,\s*0\)/); + }); + + it("woven output still py_compile-clean when python3 is available", () => { + const customer = [ + "#!/usr/bin/env python3", + "# -*- coding: utf-8 -*-", + '"""module doc"""', + "from __future__ import annotations", + "", + "def calculate(a: int, b: int) -> int:", + ' """entry"""', + " total = a + b", + " return total", + "", + ].join("\n"); + const out = protectPythonSource(customer, { + intensity: "maximum", + seed: "py-weave-compile-1", + weaveDecoys: true, + }); + const dir = mkdtempSync(join(tmpdir(), "bla-py-weave-")); + temps.push(dir); + const file = join(dir, "woven.py"); + writeFileSync(file, out.code, "utf8"); + execFileSync("python3", ["-m", "py_compile", file], { stdio: "pipe" }); + }); + + it("PRNG selects decoys without business-name rule tables", () => { + const src = [ + "SLOT = {'epoch': 0}", + "STATE = 1", + "def alpha(seed=0, depth=0):", + " return seed", + "def beta(seed=0, depth=0):", + " return seed", + "def calculate(x):", + " return x", + "", + ].join("\n"); + const woven = weavePythonDecoyCalls({ + source: src, + skipDefNames: ["alpha", "beta"], + decoyFunctionNames: ["alpha", "beta"], + sharedSlotName: "SLOT", + stateVarName: "STATE", + prng: new SeededPrng("unit-weave-prng"), + intensity: "balanced", + }); + expect(woven.customerDefsWoven).toBe(1); + const body = extractDefBody(woven.code, "calculate"); + expect(body!).toMatch(/hash\((alpha|beta)\.__name__\)/); + }); +}); diff --git a/tests/weave-decoy-mcp.test.ts b/tests/weave-decoy-mcp.test.ts index df51510..63565da 100644 --- a/tests/weave-decoy-mcp.test.ts +++ b/tests/weave-decoy-mcp.test.ts @@ -43,21 +43,23 @@ describe("weave-decoy theory pack", () => { expect(md.toLowerCase()).not.toMatch(/exploit poc|crackme bypass playbook/); }); - it("buildWeavePlan classifies JS as woven and Python as warm-init-only", () => { + it("buildWeavePlan classifies JS and Python source as woven", () => { const dir = mkdtempSync(join(tmpdir(), "bla-weave-plan-")); temps.push(dir); writeFileSync(join(dir, "app.js"), "function main(){ return 1; }\n"); writeFileSync(join(dir, "svc.py"), "def main():\n return 1\n"); const plan = buildWeavePlan(dir); expect(plan.schema).toBe(WEAVE_DECOY_SCHEMA); - expect(plan.overallStatus).toBe("partial"); + expect(plan.overallStatus).toBe("ready"); const js = plan.targets.find((t) => t.relativePath.endsWith("app.js")); const py = plan.targets.find((t) => t.relativePath.endsWith("svc.py")); expect(js?.maturity).toBe("woven"); expect(js?.status).toBe("ready"); - expect(py?.maturity).toBe("warm-init-only"); - expect(py?.status).toBe("partial"); + expect(py?.maturity).toBe("woven"); + expect(py?.status).toBe("ready"); expect(plan.pipelineHook.mcpFlag).toBe("weave_decoys"); + expect(plan.pipelineHook.todayBehaviorZh).toMatch(/结构织入|customer|FunctionDef|weave_decoys/); + expect(plan.pipelineHook.todayBehaviorZh).not.toMatch(/尚未改客户 def|future hooks only/); }); });