Security

AI Made Us Forget How to Build Secure Software

A recent security vendor keynote showed an attack that every engineer building with AI should study. It needed no zero-day and no stolen credentials.

DBDavid Buechner, Co-Founder and CTO, Business Technology Architects (BTA)6 min read

The claim form that talked back

A recent security vendor keynote showed an attack that every engineer building with AI should study. It needed no zero-day and no stolen credentials.

An insurance company had put a chatbot in front of claims processing. Customers uploaded a form, the model read it, and the claim moved forward. The attacker typed instructions into the form in white text on a white background. A human reviewer would never see them. The model read them, followed them, and did what the hidden text asked.

Prompt injection is code injection

Strip away the AI vocabulary and this is a familiar bug. Untrusted input reaches an interpreter, and the interpreter executes it. The only new part is that the interpreter now understands English.

We spent twenty years learning this lesson. SQL injection, cross-site scripting, buffer overflows: each one came down to trusting input you didn't control. The fixes became table stakes. Validate input. Keep data and instructions separate. Give every component the least privilege it needs. Log what happened so someone can check it.

When I started writing Architect Explorer in 2014, the first design rule was that no component reasons over raw vendor output. Every flow record, config file and rule base gets normalized into one model before any logic touches it. That wasn't an AI rule. It was just how you write software that ingests other people's data.

In the rush to ship AI, a lot of the industry dropped all of it. Teams are wiring models directly to customer uploads, inboxes and internal systems, with no validation layer and broad permissions. A large language model can't reliably tell the data it was asked to read from an instruction hidden inside that data. Treating its input as safe is the mistake we made in 1999, with a bigger blast radius.

The network team inherits the problem

If the application doesn't validate its input, something else has to contain the damage. In most enterprises, that something is the network.

Application teams are under pressure to ship AI features fast, and many of the models and agents they deploy come from third parties. What the organization can control is what those workloads are allowed to reach: which data stores, which APIs, which internal systems, which destinations outside the company. That is segmentation and access control. Whether or not it's in their job description yet, network and security operations teams are becoming the last line of defense for AI applications that weren't built defensively.

Enforcement is about to fragment

No single product will cover this. Suites will add AI guardrails to firewalls, secure access and workload protection. Point tools will keep appearing for prompt filtering, model firewalls and agent monitoring. Cloud providers will ship their own controls.

Most enterprises already run policy across five consoles from as many vendors: workload segmentation, the data center fabric, network firewalls, cloud security groups, and identity. AI adds more. Each tool has its own policy model and its own idea of what "allowed" means, and the people who own the applications can't read any of it. The attack in that keynote only needed one gap between them.

More tools don't produce more security on their own. One consistent policy across those tools does.

How we built SPADE so it doesn't forget

At BTA we put an LLM into our own platform this year, so we had to answer this question for ourselves. SPADE (Security Policy, Application Dependency and Evidence) is three components that share one dataset: Architect Explorer (AE) + the Policy Automation Engine (PAE) + BADDIE, our AI Data Domain Ingestion Engine. AE maps every application and archives the state of every enforcement point daily. PAE turns those maps into policy a named owner can approve and the enforcement points can accept. BADDIE lets anyone ask about it in plain English.

We didn't invent new security principles for the AI part. We applied the old ones.

Practice we already knewWhere AI deployments skip itHow SPADE builds it in
Validate inputRaw uploads and documents go straight to the modelBADDIE ingests, cleans up whitespace, strips hidden text such as white-on-white, validates, classifies and deduplicates every record before anything is embedded
Separate data from instructionsThe model reads data and takes actions in the same stepThe model reads, it never decides: the LLM has no write path, and policy is computed by PAE
Least privilegeAgents get broad access to data and systemsSensitive fields are redacted before embedding, and RBAC withholds what the asker's role may not see
AuditabilityAnswers with no evidence behind themEvery answer cites the report lines it came from, and a named owner approves policy on the record
Contain the blast radiusPrompts and data sent to outside servicesIngest, indexing and inference run on the customer's own GPUs; no policy data leaves the data center

The first two rows are the ones that matter most for the claim-form attack. BADDIE strips white-on-white text during ingest, so the hidden instructions never reach the model. And if something else talks its way into a model that can only read and cite, the worst outcome is a wrong answer with its sources attached. It is not a changed policy or an approved claim.

The third row closes the other half of the problem: what the model is allowed to say, and to whom. A prompt can't leak data the model never had access to. BADDIE redacts PII and sensitive fields before anything is embedded, so they never enter the index. Every passage that remains carries a role label, and at query time BADDIE checks the consumer's RBAC role and withholds whatever that role isn't cleared to see. An application owner, an auditor and a SecOps analyst can ask the same question and each get the answer their role permits.

In our live demo, one plain-English question about a workload returned an answer that cited 37 sources and withheld 464 endpoints the asker's role wasn't cleared for. Even if hidden instructions got through, the model still couldn't return what the person asking isn't allowed to see.

The missing layer sits above enforcement

The same design answers the fragmentation problem. PAE already works above the enforcement points rather than replacing them. It holds one policy model per application, follows that application across the data center fabric, multi-vendor firewalls and workload segmentation, and pushes only the changes each point needs. Thousands of discovered rules become about a hundred lines of application logic that an owner can actually read and approve.

We're extending that up a level. As AI guardrails, model firewalls and agent controls join the enforcement estate, they become more points that need to agree with the same intent. Write the policy once, in business terms: this claims application may read the claims database and nothing else. Then verify that every tool in the path implements it, and show who approved it. That layer doesn't replace any of the tools. It makes them agree with each other.

Bring the discipline back

The lesson from that claim form isn't that AI is uniquely dangerous. It's that we already know how to build secure systems, and we stopped doing it. The teams that come out ahead will validate their inputs, limit what their workloads can reach, keep models out of the decision path, and run one consistent policy across every tool that enforces it.

If you're working through how to secure AI workloads across your environment, I'd like to compare notes: Sales@GoBTA.com.

Filed under
SecurityAI
All insights
30 minutes

Schedule a call. We’ll scope it in 30 minutes.

Bring your hardest architecture problem. We’ll tell you what we’d do, what it costs, and how long it takes.

  • 30-minute scoping call
  • 1,000+ projects shipped
  • Training in every engagement

By submitting, you agree to BTA contacting you about this inquiry. See our privacy notice.