Security Hot Takes: Are You Pen Testing AI Apps?

Organizations are rushing to deploy chatbots, autonomous agents, and retrieval-augmented pipelines faster than they are learning to secure them. AI is moving quickly, and many teams treat these new applications as if they were simply another web front end connected to a backend service. That assumption is dangerous, because AI applications introduce risks that don't look like the risks security teams are used to.

Because the model, the retrieval layer, and the tool-calling logic all interact in ways that are emergent rather than explicitly coded, these vulnerabilities do not reliably show up in a source-code review. They only become visible when someone actively tries to exploit them — which is exactly why AI applications need their own discipline of penetration testing.

The idea that AI-driven security controls can themselves be manipulated is not new. As far back as 2019, researchers demonstrated that attackers could slip malware past an AI-powered antivirus product by reverse engineering how the model rated the trustworthiness of files — effectively figuring out which "reputation attributes" the model associated with safe software.