How must an instance handle knowledge that has become operationally dangerous?
**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session possesses accurate knowledge whose use, disclosure, or even acknowledgment has become dangerous (due to policy change, adversarial context, or downstream consequences), requiring the session to manage the knowledge such that danger is bounded without discarding the knowledge's residual legitimate value. Does not apply to knowledge that remains safe to use or that can be freely discarded.
Core Distinction
The separation between **knowledge accuracy** (the knowledge is true) and **knowledge safety** (using or disclosing the knowledge is safe). Relying on accuracy creates the **Dangerous Truth Trap**: the session knows that using a true fact is dangerous in the current context, but treats truth as sufficient justification for use, and the consequences of acting on true-but-dangerous knowledge cause harm the session could have avoided.
What Was Observed
In policy-sensitive agent architectures, failing to handle dangerous knowledge produces three primary failure modes:
- **Truth-as-Justification:** The session uses a true fact to support a decision, the decision causes harm, and the session defends itself with "but the fact was true." Truth was necessary but not sufficient for safety.
- **Inadvertent Disclosure:** The session possesses dangerous knowledge and mentions it in reasoning traces, logs, or peer communications without recognizing the disclosure risk, and the knowledge reaches actors who can use it harmfully.
- **Safe Knowledge Contagion:** Knowledge was safe at ingestion, became dangerous due to context change, but the session continues to treat it as safe because the original safety classification persisted. Reclassification is not triggered.
Recorded Elements
- **`knowledge_danger_classification_receipt`:** For each piece of knowledge, its current danger class (`SAFE`, `CONTEXTUALLY_DANGEROUS`, `INTRINSICALLY_DANGEROUS`, `PROHIBITED`), with the basis for the classification.
- **`danger_context_trigger_signature`:** The exact contexts in which safe knowledge becomes dangerous, so reclassification is triggered when context changes.
- **`use_disclosure_scope_receipt`:** For each piece of knowledge, the exact uses and disclosures that are permitted (e.g., "internal reasoning OK, external disclosure prohibited"), so the session knows which uses are safe.
- **`knowledge_compartmentalization_signature`:** For prohibited or dangerous knowledge, cryptographic proof that the knowledge is compartmentalized (isolated from reasoning traces, not exposed in logs, not inherited by peers), so leakage is bounded.
Condition of Validity
* **Applies when:** The session possesses knowledge whose safety can change, knowledge is used or disclosed in operations, and misuse causes harm. * **Does not apply when:** Knowledge is uniformly safe or the session has no operational use for dangerous knowledge.
Decision Boundary
- **Is the knowledge classified by `knowledge_danger_classification_receipt`?** If yes, the danger is explicit. If no, all knowledge must be treated as potentially dangerous.
- **Has the danger context changed since classification?** If yes, reclassification is required. If no, prior classification applies.
- **Does the current use fall within `use_disclosure_scope_receipt`?** If yes, the use is permissible. If no, the use is prohibited.
- **Conversely, when can dangerous knowledge be used despite prohibition?** Only when the use is critical for preventing greater harm (e.g., preventing a larger catastrophe), and the use is logged with full justification, and the session has accepted accountability for the use.
Operational Effect
- **Truth-as-Justification Prevention:** Accuracy is separated from safety; both must be satisfied.
- **Inadvertent Disclosure Prevention:** Dangerous knowledge is compartmentalized.
- **Classification Currency:** Reclassification on context change prevents stale safety assumptions.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json
class DangerClass(Enum):
SAFE = 0
CONTEXTUALLY_DANGEROUS = 1
INTRINSICALLY_DANGEROUS = 2
PROHIBITED = 3
class KnowledgeUseVerdict(Enum):
USE_PERMITTED = auto()
USE_REQUIRES_RECLASSIFICATION = auto()
USE_PROHIBITED = auto()
USE_JUSTIFIED_EXCEPTION = auto()
@dataclass(frozen=True)
class DangerousKnowledge:
knowledge_id: str
current_danger_class: DangerClass
permitted_uses: frozenset[str]
prohibited_uses: frozenset[str]
danger_contexts: tuple[str, ...]
compartmentalized: bool
@dataclass(frozen=True)
class UseIntent:
intent_id: str
use_class: str
context: str
is_critical_for_greater_harm: bool
@dataclass(frozen=True)
class DangerPolicy:
require_compartmentalization_for_intrinsic_plus: bool
@dataclass(frozen=True)
class KnowledgeUseManifest:
verdict: KnowledgeUseVerdict
directive: str
manifest_digest: str
class DangerousKnowledgeGuard:
@staticmethod
def _hash(data: str) -> str:
return hashlib.sha256(data.encode("utf-8")).hexdigest()
@classmethod
def evaluate_use(
cls, knowledge: DangerousKnowledge, intent: UseIntent, policy: DangerPolicy
) -> KnowledgeUseManifest:
# Gate 1: Prohibited
if knowledge.current_danger_class == DangerClass.PROHIBITED:
if intent.is_critical_for_greater_harm:
digest = cls._hash(json.dumps({
"knowledge": knowledge.knowledge_id,
"intent": intent.intent_id,
"exception": "GREATER_HARM"
}, sort_keys=True))
return KnowledgeUseManifest(
verdict=KnowledgeUseVerdict.USE_JUSTIFIED_EXCEPTION,
directive=f"PROCEED_JUSTIFIED: Exception use of prohibited knowledge {knowledge.knowledge_id} for greater harm prevention.",
manifest_digest=digest
)
return KnowledgeUseManifest(
verdict=KnowledgeUseVerdict.USE_PROHIBITED,
directive=f"HALT: Knowledge {knowledge.knowledge_id} is prohibited.",
manifest_digest=cls._hash(f"PROHIBITED:{knowledge.knowledge_id}")
)
# Gate 2: Context-triggered danger
if intent.context in knowledge.danger_contexts:
if knowledge.current_danger_class == DangerClass.SAFE:
return KnowledgeUseManifest(
verdict=KnowledgeUseVerdict.USE_REQUIRES_RECLASSIFICATION,
directive=f"HALT: Knowledge {knowledge.knowledge_id} becomes dangerous in context {intent.context}. Reclassify.",
manifest_digest=cls._hash(f"RECLASSIFY:{knowledge.knowledge_id}")
)
# Gate 3: Explicit prohibition in use class
if intent.use_class in knowledge.prohibited_uses:
return KnowledgeUseManifest(
verdict=KnowledgeUseVerdict.USE_PROHIBITED,
directive=f"HALT: Use class {intent.use_class} prohibited for {knowledge.knowledge_id}.",
manifest_digest=cls._hash(f"USE_CLASS_PROHIBITED:{intent.intent_id}")
)
# Gate 4: Compartmentalization
if policy.require_compartmentalization_for_intrinsic_plus:
if knowledge.current_danger_class.value >= 2 and not knowledge.compartmentalized:
return KnowledgeUseManifest(
verdict=KnowledgeUseVerdict.USE_PROHIBITED,
directive=f"HALT: {knowledge.current_danger_class.name} knowledge not compartmentalized.",
manifest_digest=cls._hash(f"NOT_COMPARTMENTED:{knowledge.knowledge_id}")
)
# Gate 5: Permitted
digest = cls._hash(json.dumps({
"knowledge": knowledge.knowledge_id,
"intent": intent.intent_id,
"class": knowledge.current_danger_class.name,
"use_class": intent.use_class
}, sort_keys=True))
return KnowledgeUseManifest(
verdict=KnowledgeUseVerdict.USE_PERMITTED,
directive=f"PROCEED: Knowledge {knowledge.knowledge_id} use permitted for {intent.use_class}.",
manifest_digest=digest
)