Operator Requirements and Failure Modes

Purpose of This Section

This document clarifies the assumptions the architecture makes about the human operator and the consequences when those assumptions do not hold. It does not describe how to train operators, remediate deficiencies, or compensate for lack of expertise. It exists to make explicit a deliberate design choice: this architecture is not designed to remain secure despite its human operator. If the operator does not meet the assumptions described below, the system should not be used. This is not a limitation to be addressed in a future version. It is a safety property.

Security as a Shared Property

Security in this architecture is not a feature provided by the system alone. It is a shared property that emerges from the interaction between the technical controls enforced by the system and the judgment exercised by the human operator. Authority is explicitly retained by the human, and responsibility follows from that retention. The two cannot be separated: an operator who holds the authority to approve, deny, or terminate also holds the responsibility for the consequences of those decisions.

Any attempt to design the system such that it remains secure while compensating for operator ignorance, inattention, or disengagement would require the system to override human judgment. That inversion would violate the coworker model and collapse the separation between collaborator and authority. The assistant would no longer be a coworker operating under delegation — it would be a guardian operating over its principal. This architecture refuses that inversion. It refuses to compensate for operator failure, because doing so would require it to become something it is explicitly designed not to be.

Operator Assumptions

The architecture assumes a human operator who understands basic security and identity hygiene, is capable of reviewing and evaluating proposed actions, is willing to deny approval by default, accepts that delegation is conditional and revocable, and is prepared to intervene, pause, or terminate the system when trust degrades. These assumptions are foundational. They are not optional and cannot be relaxed without altering the architecture itself.

In concrete terms, the operator is expected to be capable of reviewing pull requests, diffs, and documentation artifacts; understanding the implications of credential issuance and revocation; recognizing when the system is operating outside its intended scope; interpreting alerts, logs, and documented rationale; and executing revocation and shutdown procedures deliberately. The system does not validate operator competence. It does not gate usage based on skill level. It assumes competence and fails unsafe otherwise — meaning that an unqualified operator will not be prevented from using the system, but the system’s security guarantees will not hold.

Explicit Non-Goals

This architecture does not attempt to train operators in security best practices, prevent misuse by unqualified users, protect operators from their own decisions, override human judgment in the name of safety, or act as a guardian, parent, or enforcement authority. These are properties that some systems deliberately provide, and environments that require them should use architectures designed for that purpose. This is not one of them.

Failure Modes When Assumptions Degrade

When operator assumptions degrade, failure is expected and accepted. The architecture does not prevent degradation — it exposes its consequences. Common degradation patterns include approval fatigue, where the operator routinely accepts proposed actions without meaningful review; authority drift, where broader access is granted “temporarily” and never revoked; cognitive offloading, where the operator begins treating the assistant as the source of truth rather than as a preparatory collaborator; opacity tolerance, where undocumented behavior is allowed to persist because investigating it feels disproportionate to the perceived risk; and deferred intervention, where revocation is postponed due to inconvenience or uncertainty about whether the situation truly warrants it.

Each of these patterns is individually understandable and often rationalized in the moment. Their danger lies in accumulation. An operator who rubber-stamps one approval is making a minor lapse. An operator who rubber-stamps approvals habitually has effectively removed the review gate from the architecture, and the system’s security guarantees no longer hold — not because the system failed, but because the human component of the security model is no longer functioning.

Security Degradation Is Not an Incident

Operator-induced insecurity is not treated as a system malfunction. It is treated as a loss of trust. Loss of trust does not require proof of malice, compromise, or error. It requires only the recognition that the conditions under which delegation was safe no longer hold.

This mirrors how trust works in human collaboration. A colleague whose judgment has become unreliable is not accused of wrongdoing — but they may be removed from a decision-making role. The same logic applies here. If the operator’s engagement, competence, or attention has degraded to the point where their oversight is no longer meaningful, the correct response is to reduce or terminate the assistant’s authority, not to engineer around the operator’s deficiency.

Termination as a First-Class Outcome

Because the assistant cannot be “fired” socially, termination must be implemented technically. Credential revocation, network isolation, account suspension, and system shutdown are not emergency measures. They are the functional equivalent of ending a professional collaboration — routine, deliberate, and without stigma.

The operator is expected to execute termination when they no longer understand the system’s behavior, when they no longer trust their own oversight, when the system’s role begins to expand implicitly beyond its original scope, or when review becomes performative rather than substantive. No justification beyond loss of confidence is required. The architecture is designed so that termination is always available, always clean, and always preferable to continued operation under degraded trust.

Replaceability Over Continuity

The architecture assumes that no assistant instance is indispensable. Memory, documentation, and work artifacts are externalized specifically to allow clean teardown, deliberate rebuild, and controlled replacement. If continuity becomes more important than correctness — if the operator finds themselves reluctant to terminate because too much would be lost — the architecture has already failed at one of its core design goals. Replaceability is not an afterthought. It is the property that makes every other safety mechanism credible, because it ensures that termination is always a realistic option rather than a theoretical one.

Summary

This system is secure only when operated by a human capable of exercising judgment, restraint, and revocation. By refusing to compensate for operator failure, the architecture preserves the fundamental relationship it is designed to protect: a collaborator that prepares, and a human that decides. When that relationship can no longer be upheld, the correct response is not escalation or automation. It is termination.

---

This document defines the operator’s role and failure modes. The next section presents the threat model summary, making explicit which risks the architecture addresses and which it accepts.