The AI Surface Convergence Continuum

I’ve been taking deep dives into agent platforms, frameworks and implementations, helped along by fabulous resources like the OWASP GenAI Security Project, Promptfoo and the FinBot CTF, all as part of my research for Injection8r.

The only consistent thing I’ve found is inconsistency.

When I say AI surface, I mean the total composition: logic, code, an LLM, supporting internal systems and connected external systems, including the main application. An agent is one flavour of that composition.

Some agent descriptions concentrate on the LLM and its accompanying code: the harness, the loop, or whatever we’re calling it this week. I want to include the systems around them too: authentication, authorisation and accounting (AAA), policy, observability, and everything else the surface depends on.

What did the implementation actually adopt?

People implement abstract frameworks to suit their situation. Anyone who’s worked with ITIL will recognise the pattern. And it’s never the framework’s fault when it goes wrong, is it?

An implementation might adopt only part of a framework. That can be a deliberate, sensible decision. The trouble starts when you presume it adopted a control that it didn’t, or when a feature claim leaves that distinction unclear.

For example, a framework might recommend a policy engine such as Cedar and provide a reference implementation. But does the AI surface use it?

What about user authentication? Does the surface use an identity service like Clerk, or its own authentication code backed by a local database? Either way, where are those identities used? Does the surface carry the user’s identity and permissions through to its tools, or do those tools act through a privileged service account?

Maybe the surface provider built their own version of a component that’s recommended in the framework. Or maybe they deliberately left it out. Or maybe it’s on the roadmap.

How do you know? Well, you could wrap it all in a Somebody Else’s Problem field.

Or you could do what I do and stick your sticky beak in. Here’s what I check.

Four different claims

For any feature, control or important configuration, I check four distinct claims:

1. Claimed: what the product or platform says it does.
2. Available: what the code and features make possible.
3. Enabled: what this deployment has configured and switched on.
4. Used in the hot path: what actually runs when a request reaches a tool, some data or a consequential action.

Claimed ≠ available ≠ enabled ≠ used in the hot path.

A policy engine can exist, be configured correctly, and still sit outside the path that performs the action. Even when it runs, we need evidence that it enforces the intended boundary.

That distinction matters more than another tick in a feature comparison.

This reminds me of a Vblock

Looking through these complicated and often incomplete agent reference architectures took me back about 15 years, to converged infrastructure. We’d tackled a similar problem before.

I once counted the features and controls across the components in a Vblock. There were hundreds. Each control had its own settings. The number of possible combinations was enormous.

The point of the Vblock approach was to down-select those possibilities into a more limited, proven composition. Selected components. Supported configurations. A combination we could build, test and deliver repeatedly.

My recollection is that we were using something like 10–20% of the combined component feature set. Customers weren’t asking for every possible feature and setting. They wanted a system that did its job predictably.

Restricting the configuration to something we understood deeply gave us a basis for making promises about performance, security and delivery. The trade-off was that you, the customer, couldn’t frig about with every configuration setting and expect those promises to survive. There may have been a very heavy-duty Excel spreadsheet.

Fast forward to today, and I’m seeing AI surfaces assembled with much more freedom, lots of frigging about, and then patched with sticking plasters and Blu Tack as problems turn up.

There has to be a better way. At the risk of resurrecting the much-abused term “shift left”, I want to bring that same discipline into how we build AI surfaces: choose the composition, constrain the configuration and build quality into the process from the start.

A convergence continuum for AI surfaces

Here’s the spectrum I’m proposing. It moves from a white box you assemble and configure towards a black box you operate through a smaller, defined interface.

DIY. Choose the parts, wire the controls and take responsibility for validating the whole.

Reference-architected. Follow a blueprint, while choosing how to implement it. You still have to prove that your version fits together.

Framework-integrated. Adopt shared plumbing, with choices about which controls to use. Framework adoption alone doesn’t tell us which controls the application actually invokes.

Converged. Combine selected components into a larger integrated system, with constrained configurations and validation of the whole composition, including the paths through its controls.

Hyper-converged. Package runtime and controls into a compact, repeatable unit with few knobs and dials. Scale out by adding units, such as more agent instances, with validation of each unit and how the units work together.

These positions can overlap: a reference architecture can use a framework. The progression is about how much of the composition is settled for you, how much you can change, and what you operate as a unit.

Much of what I’ve examined sits towards the DIY and reference-architected end, including implementations that use parts of a framework. I’ll come back to the individual examples in future posts.

Building quality in earlier means moving right on this continuum: accepting fewer configuration choices in exchange for a more integrated composition. Reducing the combinations gives us a more manageable system to validate. We still have to validate it.

And a smaller operating interface still needs to expose evidence of what happened inside. A black box doesn’t remove the need to prove which controls ran and what they did.

Where Injection8r is heading

This is the direction I’m taking with Injection8r, within the broader Surfaces work.

I want to explore how far we can select the parts, constrain the configuration and validate the meaningful paths together. Then keep that evidence useful as models, tools, permissions and integrations change.

Testing remains part of that. So does knowing what happens when a control is unavailable, when a request should be denied, or when an update changes the composition we previously tested.

The question I want to bring into the build is simple: which controls does this surface depend on, and what proves they are on the path that matters?

We learned something useful from converged infrastructure. I think it’s time to apply that thinking to AI surfaces.

For the related testing perspective, read A Chatbot Is Not the Surface.