What If AI Goes Wrong Inside Your Business?
What if your AI-built app sends confidential company records to someone who should never receive them?
If the app uses a language model to read documents, a hidden instruction could redirect its behavior. The app could use credentials and connections your team approved to carry out an action you never authorized.
Your business would face the consequences: exposed records, unauthorized transactions, or customers relying on incorrect information. AI safety includes controls over what the app can access, what it produces, and which actions it can take. Adding it after launch leaves people using the app while those boundaries remain unresolved.
Why AI security needs additional controls
A valid login cannot establish whether every record an app retrieves belongs in that user’s hands. A firewall cannot determine whether a permitted API call serves the intended task.
Traditional software has established methods for separating instructions from data. Parameterized database queries, for example, prevent supplied values from rewriting the query’s structure. OWASP: SQL injection
Language models interpret operating instructions, user requests, and retrieved information within the same context. Instruction roles help distinguish them, but untrusted content can still influence behavior. OWASP: prompt injection
| Security question | Traditional controls | Additional AI considerations |
|---|---|---|
| What determines the next step? | Programmed conditions govern execution. | Models interpret context and propose responses or actions. |
| How are instructions separated from data? | Compilers, parsers, and parameterized queries establish boundaries. | Untrusted content can influence the same model context. |
| Which vulnerabilities need review? | Buffer overflows, SQL injection, cross-site scripting, and broken access controls. | Prompt injection, system prompt leakage, and excessive agency. |
| How are inputs checked? | Schemas, pattern matching, and cryptographic signatures. | Content checks for manipulation and sensitive information. |
| How can actions be traced? | Stack traces and defined API contracts. | Records of model inputs and outputs, tool calls, and generated parameters or schemas. |
| Which protections apply? | Static analysis, dynamic testing, and firewalls. | Runtime checks, evaluation models, isolation, and programmed permission checks. |
Editorial synthesis. Sources: OWASP SQL Injection Prevention Cheat Sheet · OWASP LLM01: Prompt Injection · OWASP LLM06: Excessive Agency · OWASP LLM07: System Prompt Leakage · OWASP LLM05: Improper Output Handling · OWASP Top 10 for LLM Applications
These additional controls belong alongside your existing application and infrastructure security.
How prompt injection can redirect your app
An app can encounter malicious instructions while completing a legitimate request. Direct prompt injection comes through the user’s request. Indirect injection arrives in material the app reads. Retrieval-augmented generation, or RAG, brings external information into the model’s context, where embedded instructions can influence its behavior. OWASP: prompt injection
The damage depends on what the application can do. OWASP calls unnecessary functionality, permissions, or autonomy “excessive agency.” An AI agent can make individually valid API calls that collectively exceed the task you authorized. Least privilege limits it to the records and actions its task requires. OWASP: excessive agency
Low-code tools make these connections accessible to business teams. A working demonstration can still leave access rules, output checks, data loss prevention, and rate limits unresolved. Threat modeling, identity checks, and code review need to cover how the connected services operate together.
Shadow AI: what employees use beyond your visibility
Shadow AI means using AI without organizational approval or oversight. Personal accounts and unreviewed workflows can move company information beyond your visibility. These studies measure different populations and activities.
| Finding | Figure | Study context |
|---|---|---|
| Shadow AI involvement in security incidents | 43% in 2026; 20% in 2025 | IBM 2026: 602 organizations that experienced breaches. IBM, 2026 |
| Average cost of shadow-AI incidents | $5.39 million | IBM 2026: employees’ unapproved AI use. IBM, 2026 |
| Average cost of malicious breaches involving attackers using AI | $6 million | IBM 2026: separate from shadow AI. Overall global breach average: $4.99 million. IBM, 2026 |
| AI governance policies absent or developing | 63% | IBM 2025: 600 organizations that experienced breaches. IBM, 2025 |
| AI usage through unmanaged personal accounts | 67% | LayerX 2025: enterprise-customer browser activity. LayerX, 2025 |
| AI users bringing their own tools | 78% | Microsoft/LinkedIn 2024: 31,000 knowledge workers, 31 markets. Microsoft/LinkedIn, 2024 |
| Employees bypassing blocked applications | 41% | UpGuard 2025: 1,020 employed people, US and UK. UpGuard, 2025 |
| Submitted corporate data that was sensitive | 34.8% | Cyberhaven 2025: seven million workers’ AI usage patterns. Cyberhaven, 2025 |
| Source code within submitted sensitive data | 18.7% | Cyberhaven 2025: largest named category; R&D followed at 17.1%. Cyberhaven, 2025 |
| Respondents reporting a GenAI-related breach in the previous year | 89.5% | AvePoint/Osterman 2026: broad definition, including inaccurate content and harmful advice. AvePoint/Osterman, 2026 |
| Respondents reporting an AI-agent-related breach in the previous year | 88.4% | Same survey: 750 information management, data security, or AI leads. AvePoint/Osterman, 2026 |
IBM’s shadow-AI costs concern employees’ unapproved use. Its $6 million figure concerns breaches involving attackers who used AI. Cyberhaven measured submitted data; submission alone does not establish a breach. AvePoint’s self-reported incidents include inaccurate content and harmful advice. Its sample covered financial services and insurance, healthcare, and government organizations.
For your business, a ban leaves an oversight problem when employees work around it. Reviewing approved tools covers only part of the activity. You also need visibility into the information moving through personal accounts.
Your business may answer for an incorrect response
An app can create consequences through information alone. A customer may act on its answer before anyone checks it.
Air Canada’s chatbot gave incorrect fare information
In Moffatt v. Air Canada, a passenger relied on the airline’s chatbot when buying a ticket. It said he could claim a bereavement discount within 90 days. The airline later refused because its policy prohibited retroactive claims.
The tribunal found Air Canada liable for negligent misrepresentation. It rejected the suggestion that the chatbot was a separate entity responsible for its statements. The decision concerned the facts of that dispute. ABA: Air Canada decision
New York City’s chatbot gave incorrect legal guidance
Testing of MyCity found incorrect advice about employment, housing, and waste-disposal rules. City leaders initially kept it available while promising improvements. THE CITY: MyCity testing AP: MyCity testing
A December 2025 audit found inconsistent answers and insufficient detail in the supplied testing summary. The technology office said it conducted testing. The city decommissioned the chatbot in February 2026. NYC Comptroller, 2025 The Markup, 2026
Testing needs to account for the decisions people may make from an answer. A promised improvement can arrive after someone has relied on incorrect guidance.
The OWASP Top 10 risks for AI applications
The 2025 OWASP Top 10 for Large Language Model Applications organizes risks across inputs, data, outputs, permissions, and resource use. OWASP, 2025
| ID | Risk | What can go wrong |
|---|---|---|
| LLM01 | Prompt injection | Instructions in requests or external content redirect behavior.OWASP Top 10, 2025 · OWASP LLM01 |
| LLM02 | Sensitive information disclosure | Responses expose confidential or personal information.OWASP Top 10, 2025 |
| LLM03 | Supply chain | Compromised models, datasets, libraries, or dependencies introduce vulnerabilities.OWASP Top 10, 2025 |
| LLM04 | Data and model poisoning | Altered data causes incorrect or malicious behavior.OWASP Top 10, 2025 |
| LLM05 | Improper output handling | Another system uses generated content without adequate checks.OWASP Top 10, 2025 · OWASP LLM05 |
| LLM06 | Excessive agency | Functionality, permissions, or autonomy exceed the task’s needs.OWASP Top 10, 2025 · OWASP LLM06 |
| LLM07 | System prompt leakage | Internal instructions or sensitive information within them become exposed.OWASP Top 10, 2025 · OWASP LLM07 |
| LLM08 | Vector and embedding weaknesses | Storage or retrieval weaknesses enable manipulation or unauthorized access.OWASP Top 10, 2025 · OWASP LLM08 |
| LLM09 | Misinformation | The model produces false or misleading information.OWASP Top 10, 2025 |
| LLM10 | Unbounded consumption | Uncontrolled processing disrupts service or increases costs.OWASP Top 10, 2025 |
Several risks can combine when an application retrieves information and acts on it. Retrieval systems use numerical representations called embeddings to find related material. They also need to filter results by the user’s permissions before those results enter the model’s context.
Attribute-based access control uses information about users and resources to enforce those boundaries. System prompts can leak, so their instructions cannot be relied on to protect secrets. OWASP: retrieval risks OWASP: prompt leakage
Generated content can also enter a browser, database, or command interpreter. Improper handling can enable cross-site scripting, server-side request forgery, or remote code execution. The receiving system needs checks suited to its destination. OWASP: output handling
Where your app needs AI safety checks
AI guardrails are checks that inspect information and restrict what an application can return or do. They need to follow information through the workflow. A proxy can inspect requests before passing them between components. Organizing checks around inputs, context, and outputs helps identify gaps; the implementation depends on the application.
1. Check information before it reaches the model
Inspect requests and external content for manipulation, sensitive information, and invalid structure. Data loss prevention can identify personal information, payment details, API credentials, and health records for redaction or anonymization. Confirm identity and permissions before retrieving company records.
2. Control the context the model receives
Separate operating instructions from external content. Enforce access controls outside the prompt. Document metadata can support filtering against authenticated users’ permissions. Isolation must also limit information crossing between users, sessions, and workloads.
3. Check responses and actions before releasing them
Check whether answers have support in the supplied sources. Inspect generated code using structural and static analysis, with separate checks for shell commands. Validate tool permissions, parameters, transaction limits, and required approvals before execution.
Managing AI risk after launch
Someone needs responsibility for investigating failures and changing the controls. NIST’s voluntary AI Risk Management Framework organizes this work into Govern, Map, Measure, and Manage. These functions establish responsibilities, identify context and risks, assess performance, and address what needs to change. NIST AI RMF NIST AI RMF Core
Maintain an inventory of AI components and connections. Monitor behavior and provide a way to isolate a compromised system. Keep records of access decisions, blocked requests, token usage, and tool activity. Protect retained inputs and outputs with access and retention controls. These records support investigation and audits; they do not establish compliance by themselves.
Five AI safety requirements for the build
- Include runtime input and output checks in the architecture and launch review.
- Give internal builders reviewed API connections, wrappers, and low-code components with the required controls.
- Restrict permissions and define human approval for consequential actions, including database writes, financial transactions, and external communications.
- Inspect information leaving through company and personal accounts, including source code and personal data.
- Map controls against relevant NIST and OWASP guidance. Use records of model behavior, guardrail decisions, and tool activity to update them as the application changes.