A Chatbot Is Not the Surface: Testing the Path From Prompt to Consequence

An AI feature can look reassuringly small. There is a chat box, a system prompt and a short list of example questions. It is tempting to test that visible conversation, find a handful of unwelcome answers, and call the job done.

But the chat box is usually only the first surface. A useful review follows the path from the person asking, through instructions and retrieved context, to the model, its tools, any connected systems and the action that follows. That is where a helpful answer can become an unauthorised search, a misleading recommendation or a consequential change.

Test the path, not only the prompt

Prompt tests are valuable. They establish whether a model follows instructions under a known set of inputs. They do not prove that a real application preserves identity, limits access, handles retrieved material safely, or controls what a tool can do.

For a practical AI surface, the questions are broader:

• Who is allowed to begin this task, and which permissions travel with them?
• Which documents, APIs or memories can shape the answer?
• Can untrusted text in those sources redirect the model’s behaviour?
• Which tools can the model call, with what arguments and approval points?
• What happens after the answer: a draft, a recommendation, a search, a message, or a system change?

The aim is not to make a model refuse everything unfamiliar. It is to make the permitted path explicit, observable and proportionate to the consequence.

A harness makes the boundary testable

A security harness lets a team exercise an AI feature repeatably without pretending that a single prompt is representative. It can hold a known actor, a controlled set of instructions and retrieved material, a defined tool contract, and an expected result. That makes it possible to compare intended behaviour with what actually happened.

This is particularly useful for chat. A polite conversation can conceal an unsafe transition: a retrieved note changes the model’s priorities, an apparently harmless request causes an over-broad search, or a suggested action is treated as an instruction. A harness makes those transitions visible enough to examine.

It also gives teams a place to state the limit of each check. A passing harness run shows that the specified path behaved as expected in the tested conditions. It does not prove that every future model version, integration, user role or source document will behave safely. That distinction keeps assurance honest and turns testing into an ongoing practice rather than a one-off demonstration.

MCP expands the surface area

Model Context Protocol (MCP) can make useful capabilities available to an AI application: a model may discover information, call a service or prepare work through a tool. That convenience changes the question from “what did the model say?” to “what could the application do?”

Each MCP connection needs the same care as any other integration. Its identity, available tools, scopes, input validation, output handling and audit trail should be understood before it is placed in a production path. A tool description is not an authorisation boundary by itself. Nor is an agent’s intent a substitute for the permissions enforced by the system receiving a request.

In practice, this means testing both directions. Can a malicious or malformed source influence the model into selecting an inappropriate tool? Can a tool return content that the model mistakenly treats as trusted instruction? And if a request is permitted, can the team explain who initiated it, what evidence was used and what happened next?

Where Injectionator fits

Injectionator is Viewyonder’s working platform for examining these paths. It supports controlled exercises of AI features so teams can gather evidence about instructions, retrieved material, tool use and outcomes. The point is not a theatrical “break-in”; it is a disciplined way to learn which claims the feature can support and where a control needs strengthening.

That work is most useful when it begins with a real decision. A team might bring a customer-support assistant that can search internal knowledge, a drafting workflow that can call approved services, or an MCP-enabled tool that is about to move beyond a prototype. We identify the specific surface, the actors and consequences that matter, then choose evidence that can help a product, security and delivery team decide what to do next.

A professional-services pathway

The first engagement should be small enough to be useful and concrete enough to change a decision.

1. Bring one surface. Describe the feature, its intended users and the decision or action it supports. Do not send credentials, customer data, contracts or confidential architecture details through a public form.
2. Map the intended path. We frame the relevant identity, context, tool and consequence boundaries, and agree what evidence would be useful.
3. Exercise the agreed cases. Using Injectionator and an appropriate harness, we test authorised scenarios and record what was observed.
4. Turn evidence into next steps. The output is a focused set of findings, limits and priorities—not a generic score or a promise that the system is now safe forever.

That pathway can lead to a targeted assessment, a build-and-test engagement, or a recurring practice as the surface evolves. The right next step depends on the consequence of the feature and the evidence a team needs to make its next decision.

If you are building an AI feature, explore Surfaces. If you already have one decision in mind, start an assessment.