Defending with AI, and testing your AI

Catherine Ives-YimPublished 27 September 2026 · 7 min read

This is the last article in the security series, and it draws together the three series so far on AI in engineering companies. The first described five levels of AI value, from finding information to closing the loop from field data. The second covered agents, the means by which the upper levels get built. This one has been about security.

It ends with two questions every engineering company using AI needs to answer. How can AI strengthen our defences? And how do we know the AI systems we depend on are themselves secure?

Part one: defending with AI

AI is already useful for security work in engineering companies, provided it is used as an aid to expert judgement rather than a substitute for it.

Code and firmware review. AI can read large amounts of code and point out common weaknesses: unchecked inputs, unsafe defaults, credentials in the wrong place, security controls applied inconsistently. It is particularly useful on older code that nobody has reviewed for years. It produces false alarms and misses things, so its findings are a starting point for an engineer, not a verdict.

Vulnerability triage. When a vulnerability is announced in a component, working out whether and how your products are affected is exactly the kind of structured investigation the triage agent in Agents at work in engineering is designed for. With a good software inventory, it turns days of work into minutes.

Threat modelling. AI is a useful partner in thinking through how a system could be attacked. Given a description of a design, it can suggest attack routes and questions a reviewer might not have considered. It does not replace an experienced security architect, but it makes a review more thorough.

Testing inputs. Generating large numbers of unusual and malformed inputs to test how software and devices handle them is laborious by hand. AI makes it faster, which matters because, as discussed in Connected products versus AI-assisted attackers, attackers are using it for the same purpose.

Watching logs and field data. The digests and analysis described in Closing the loop can look for signs of attack as well as signs of faults: unusual patterns of access, failed authentication, devices behaving in ways that do not match normal use.

Documentation. Security and compliance documentation is extensive and often repetitive. AI drafting, with engineers reviewing and approving as described in AI drafts, engineers decide, can reduce the burden considerably.

Two cautions apply to all of these. Security work involves an organisation's most sensitive material, including source code, firmware and vulnerability details, so where the AI runs matters. And AI-assisted security work supplements independent expert testing. It does not replace it.

Part two: testing your AI

Every AI system an organisation depends on should be tested for security as seriously as any other system. In practice, this is where most organisations have the biggest gap.

Red-team it. Deliberately try to make your AI systems misbehave: to reveal data they should not, to take actions they should not, to follow instructions hidden in content, as described in When the data gives the orders. Do it before deployment, and again whenever the system changes. Do it with the real model. As I noted in Running agents in production, tests that use pre-written model responses tell you about the surrounding code, not about how the model actually behaves under hostile input.

Test the permissions, not just the model. The most important question is not whether the model can be persuaded to try something harmful. It is whether the system would let it succeed. Test what an AI system can actually reach and do, using the identity it runs with.

Test the data boundaries. Check that search systems respect document permissions, that assistants act with the user's rights rather than their own, and that logs and indexes are protected, as described in How data leaks out of AI systems.

Keep testing. AI systems change when models are updated, prompts are revised, tools are added and data grows. Security testing needs to be part of every change, not a one-off exercise at launch.

Include AI in independent testing. When commissioning penetration testing of a product or platform that includes AI, make sure the AI components, their tools and their data access are in scope.

Questions to ask suppliers

Many engineering companies will buy AI capability rather than build it. These questions separate suppliers who have thought about security from those who have not:

Where is our data processed and stored, including indexes, logs and backups, and who can be legally compelled to produce it?
Is our data used to train or improve models, and can we prevent that?
What can your system access in our environment, with what identity, and how is that limited?
How do you defend against instructions hidden in content your system reads?
Which actions can your system take without human approval?
What is logged, who can see the logs, and how long are they kept?
What happens when the underlying model changes, and how are we told?
When was the system last independently security tested, and was the AI in scope?

A supplier who cannot answer these clearly is asking you to trust rather than verify.

The regulatory picture

For engineering companies, several frameworks now bear on AI and security, and they are converging on the same expectations.

The EU Cyber Resilience Act requires manufacturers of connected products to handle and report vulnerabilities, with full obligations from December 2027. The EU AI Act brings obligations for AI systems depending on their use and risk. Data protection law applies to any personal data processed by AI. In the UK, Cyber Essentials is increasingly expected in supply chains, and the government has published a code of practice for the cyber security of AI systems.

The common thread is that organisations are expected to know what they have, control what it can do, record what it did, and respond when something goes wrong. That is not a new idea in engineering. It is simply being extended to AI.

Bringing the three series together

Across these three series, I have tried to make one argument.

AI's value to an engineering company rises through five levels: find, structure, connect, act and learn. Each level depends on the one below, and the most valuable work happens at the top, where AI helps the organisation learn from its products in the field.

Agents are how the upper levels get built. They are most useful where the path of the work cannot be known in advance, and they should earn their autonomy one task at a time, with guardrails, records and people at the points of consequence.

And every step up the ladder, and every agent, creates something that needs securing. The principles are the ones good engineers have always applied: least privilege, defence in depth, knowing what is in your systems, testing honestly, and keeping a record. What has changed is where they need to be applied.

The models will keep improving and keep becoming cheaper. The advantage will lie with organisations that build the foundations well, choose deliberately where AI runs, and can stand behind what their systems do.

Four things worth taking seriously

For engineering leaders: use AI to strengthen your security, and test your AI as seriously as anything else you depend on.

For security teams: bring AI systems into scope for red-teaming, penetration testing and permission reviews, starting now.

For anyone buying AI: ask the eight questions above, and expect clear answers.

For everyone: AI is an engineering system. It deserves the same rigour, and rewards it.

The next series turns to coding agents specifically: how to exploit them, what they cost, how to keep code private, and why they need experienced engineers. It begins with Vibe coding is just another level of abstraction.

I would be interested to hear which of these articles was most relevant to where your organisation is now, and what you would like to see explored next.