When an AI Failure Is Not Fraud but Still Creates Regulatory Exposure

Not every serious AI failure is fraud

An AI system rejects a legitimate customer.

A chatbot invents a product feature.

A recruitment tool disadvantages candidates with a protected characteristic.

A fraud-detection model freezes innocent customers while missing a new fraud pattern.

An automated summary omits a critical warning.

A generative tool produces confident but false advice which an employee sends to a client.

The consequences may be serious. A person may lose money, employment, access to a service or an important opportunity. A regulated firm may make the wrong decision. A board may discover that a system was deployed without a reliable owner, adequate testing or any practical way to explain what it did.

None of that, without more, proves fraud.

Fraud is not a synonym for error, harm, opacity or poor governance. The principal Fraud Act 2006 offences require dishonesty and an intention to make a gain or cause, or expose another to the risk of, loss. A model cannot itself form the legally relevant dishonest state of mind. A human may use an AI system dishonestly, but an unreliable output does not become fraudulent merely because it was generated by AI or caused loss.

That distinction protects the integrity of the criminal law. It should not be mistaken for an absence of exposure.

The same facts may instead engage:

• data-protection law;
• consumer-protection law;
• equality and employment law;
• Financial Conduct Authority rules and principles;
• operational-resilience and outsourcing requirements;
• online-safety duties;
• product-safety or sector-specific regulation;
• professional duties;
• contract, negligence or misrepresentation;
• directors’ and senior-management responsibilities; and
• notification, redress and remediation obligations.

The central principle is:

The absence of dishonesty may remove fraud. It does not remove accountability.

The correct response begins by identifying what actually failed, who was responsible for the decision, which legal regime protects the affected person or interest, and what the organisation knew before and after deployment.

Start with the legal elements, not the label

Organisations often move too quickly between two extremes.

At one extreme, every harmful AI incident is described as fraud, deception or criminality. That can distort the investigation, contaminate communications and cause an organisation to make allegations before the evidence supports them.

At the other, the incident is dismissed as a technical error because no one intended the result. That can overlook strict, negligence-based, outcomes-focused and regulatory duties which do not depend on dishonesty.

A disciplined assessment asks separate questions.

Was there a dishonest human act?

Potential fraud requires evidence capable of establishing the relevant human conduct and state of mind.

Questions include:

• Did anyone knowingly make a false representation?
• Did anyone dishonestly fail to disclose information they were under a legal duty to disclose?
• Did anyone dishonestly abuse a position in which they were expected to safeguard another’s financial interests?
• Was there an intention to make a gain or cause, or expose another to the risk of, loss?
• Was AI used as an instrument to create, scale, disguise or deliver that conduct?
• Is there evidence of manipulation, concealment, fabricated records or a knowingly false explanation after the event?

An employee who knowingly instructs a model to create false invoices may be using AI as an instrument of fraud. An employee who reasonably relies on a flawed system may have made an error. Between those positions lie recklessness, wilful blindness, policy breaches and negligent deployment. They require careful legal characterisation rather than an assumed criminal label.

What duty existed even without dishonesty?

The next question is whether another legal duty governed:

• the data used;
• the design or procurement process;
• the decision made;
• the information communicated;
• the treatment of affected people;
• the safety or resilience of the service;
• the supervision of staff or suppliers;
• the accuracy of records;
• the handling of complaints; or
• the organisation’s response once harm became apparent.

Many such duties are concerned with outcomes, reasonable care, fairness, transparency, governance or risk management. Deliberate wrongdoing may aggravate the position, but it is not always an element of liability.

What changed after the warning?

An accidental initial failure can become a much more serious governance event if the organisation then:

• suppresses or minimises the incident;
• continues using a system known to be unsafe;
• fails to preserve evidence;
• gives an inaccurate account to customers or a regulator;
• disables or bypasses complaint routes;
• retrospectively alters logs or decision records;
• treats affected individuals inconsistently;
• fails to reconsider similar decisions; or
• makes unsupported claims that the problem has been fixed.

The legal analysis must therefore distinguish the original AI failure from the human response to it.

The AI failure exposure map

No single label captures the range of AI failures. A practical assessment should map the incident across seven connected routes.

The routes are not alternatives. One incident may engage several.

A biased credit model, for example, may involve personal-data fairness, equality law, the FCA’s Consumer Duty, complaint handling, model governance, third-party oversight and civil redress. The fact that no employee intended to discriminate does not answer any of those questions.

Data protection: the decision is part of the processing

AI systems frequently depend on personal data at several stages:

• training or fine-tuning;
• retrieval and prompt construction;
• identity matching;
• profiling and risk scoring;
• generating recommendations;
• monitoring performance;
• recording user interactions; and
• using outputs to make decisions about people.

The Data Protection Act 2018 and UK GDPR can therefore apply well before the final output.

The Information Commissioner’s Office explains in its guidance on AI and data protection that organisations must consider lawfulness, fairness, transparency, data minimisation, accuracy, security and accountability. The guidance is under review following the Data (Use and Access) Act 2025, so organisations must check the current statutory position and updated ICO material rather than rely on an old compliance template.

Lawfulness and purpose

An organisation needs a lawful basis for processing personal data. If special-category data is involved, an additional condition will usually be required. Data collected for one purpose cannot simply be repurposed for model development or evaluation without considering compatibility, transparency and the relevant legal basis.

The fact that data is publicly accessible does not automatically make every use fair or lawful.

Organisations should be able to explain:

• what data the system uses;
• where it came from;
• why its use is necessary;
• whether people were told about that use;
• how long the data and prompts are retained;
• whether a provider can use them for its own purposes;
• whether data leaves the UK;
• how individual rights can be exercised; and
• what happens when data or an inference is wrong.

Fairness, accuracy and statistical performance

AI complicates the concept of accuracy. An input may be factually correct while the inference drawn from it is unreliable. A model may be accurate in aggregate but materially less reliable for a subgroup. A generated answer may be fluent but false.

The organisation should not rely only on a headline accuracy percentage. It should understand:

• false-positive and false-negative rates;
• performance across relevant groups;
• the cost of each type of error;
• the conditions under which performance degrades;
• whether the system is being used outside its tested purpose;
• how uncertainty is communicated; and
• how errors can be corrected before they produce an adverse outcome.

Automated decisions and meaningful human review

The legal treatment of solely automated decisions has changed under the Data (Use and Access) Act 2025 and associated measures. The current regime should be checked for the particular decision, data and safeguards involved.

Whatever the precise statutory route, a nominal human in the process is not necessarily meaningful oversight.

Human review is weak if the reviewer:

• sees only the model’s recommendation;
• lacks authority to depart from it;
• does not understand its limitations;
• is measured against agreement with the model;
• has too little time or information to reconsider the result; or
• routinely approves outputs without independent analysis.

A defensible process records what the human considered, what information was available, whether the output was challenged and who owned the final decision.

Data protection impact assessments

A data protection impact assessment is required where processing is likely to result in a high risk to individuals’ rights and freedoms. AI profiling, large-scale sensitive-data processing, innovative technology and decisions with significant effects may make that requirement particularly important.

A DPIA should be a living risk assessment. It should not end at procurement. It should change when:

• the model or provider changes;
• a new dataset is introduced;
• the system is deployed for a new purpose;
• an incident reveals an untested failure mode;
• outcome data identifies a disparity; or
• the system moves from recommendation to autonomous action.

Consumer protection: a machine can still mislead

A consumer does not lose protection because the statement was generated dynamically rather than written by a marketing team.

AI can affect:

• product descriptions;
• eligibility decisions;
• pricing and personalisation;
• sales recommendations;
• customer-service explanations;
• vulnerability identification;
• complaint handling;
• renewal or cancellation journeys; and
• claims about what a product, service or AI system can do.

The Digital Markets, Competition and Consumers Act 2024 strengthened the consumer-protection enforcement regime. The Competition and Markets Authority can directly enforce consumer law and, for serious infringements, impose substantial financial penalties. Existing duties concerning misleading actions, misleading omissions, aggressive practices and professional diligence apply to AI-enabled customer journeys just as they apply to conventional ones.

The relevant question is not whether the model “knew” a statement was false. It is whether the trader’s commercial practice, viewed in its proper context, breached the applicable standard.

Hallucination is not a defence

If a business deploys a chatbot to answer customer questions, it must decide:

• which questions the tool is authorised to answer;
• which claims are grounded in approved information;
• when the tool must abstain or transfer to a person;
• how changing terms and prices are updated;
• whether material limitations are disclosed;
• how transcripts are retained;
• how recurring errors are detected; and
• how affected customers will receive correction and redress.

Calling an invented statement a hallucination describes a technical phenomenon. It does not resolve the legal consequence of communicating it to a consumer.

Claims about AI itself

Risk can also arise from overstating an AI system’s abilities.

Statements such as “bias-free”, “fully accurate”, “human-level review”, “secure”, “compliant” or “completely automated” require an evidential basis and careful qualification. A business should preserve the testing, assumptions and limitations supporting any public or contractual claim.

Equality and employment: intention is not the only route

AI may reproduce historic disadvantage, use proxy variables for protected characteristics or create a criterion which is harder for one group to satisfy.

The Equality Act 2010 can apply to employment, services, public functions, education and other protected fields. Direct discrimination, indirect discrimination, discrimination arising from disability, failures to make reasonable adjustments, harassment and victimisation have distinct legal tests. Not all require an intention to discriminate.

Examples include:

• a recruitment model trained on historic hiring patterns;
• speech or facial analysis performing less reliably for particular groups;
• automated attendance or productivity scoring disadvantaging disabled workers;
• a customer-risk model using postcode or behavioural data as an indirect proxy;
• an online process which cannot accommodate a reasonable adjustment;
• an AI-generated performance summary repeating untested allegations; and
• automated scheduling which systematically disadvantages workers with particular religious or caring commitments.

For public authorities, the Public Sector Equality Duty requires due regard to specified equality needs. The Equality and Human Rights Commission has published guidance and case studies on AI and equality, emphasising that equality must be considered when commissioning and using AI, not only after harm occurs.

Test the decision process, not merely the model

Bias testing should include:

• the data selected;
• the target variable;
• the definition of success;
• missing or low-quality data;
• proxy variables;
• thresholds and confidence levels;
• the population on which the system was tested;
• the human process around the output;
• routes for adjustment and challenge; and
• actual outcomes after deployment.

Removing protected characteristics from a dataset is not, by itself, an equality control. It may make disparities harder to detect while leaving proxies untouched.

Employment investigations and discipline

An AI summary should not become a substitute for evidence in a disciplinary process. Employers should preserve the underlying material, test accuracy, allow the employee to understand and challenge the case, and identify what role the AI output actually played.

If the system shaped selection, promotion, performance or dismissal, the organisation should be able to reconstruct the decision. “The algorithm decided” is neither a fair explanation nor a transfer of legal responsibility.

Financial services: existing obligations travel with the technology

The FCA has generally pursued a technology-neutral, outcomes-focused approach. It has not treated the absence of an AI-specific rule as a regulatory vacuum.

Its July 2026 review of AI in retail financial services confirmed that the FCA’s approach continues to rely on existing frameworks, including the Consumer Duty and Senior Managers Regime. The review also identified the potential for AI to amplify fraud, cyber, consumer-harm and market-concentration risks.

Depending on the firm and use case, relevant expectations may include:

• acting to deliver good outcomes for retail customers;
• treating customers fairly;
• communicating in a way which is fair, clear and not misleading;
• identifying and responding to vulnerability;
• maintaining adequate systems and controls;
• managing operational risk and resilience;
• overseeing outsourced and third-party services;
• keeping appropriate records;
• handling complaints properly;
• controlling financial promotions;
• protecting market integrity; and
• allocating clear senior-management responsibility.

The firm remains responsible

Buying a model, API or managed service does not outsource regulatory responsibility.

The firm should know:

• who owns the use case;
• what decision the system supports or makes;
• what customer harm can follow from error;
• how the provider has tested it;
• what independent validation the firm performed;
• what data the provider receives;
• how model changes are notified;
• what audit, access and exit rights exist;
• what monitoring identifies drift or unequal outcomes;
• when the system must be suspended; and
• how customers can obtain human reconsideration and redress.

The contract matters, but a contractual warranty cannot replace the firm’s own governance.

Consumer Duty evidence

An organisation using AI in a retail customer journey should be able to evidence outcomes, not merely process completion.

Useful measures may include:

• error and override rates;
• outcome differences between customer groups;
• complaints and repeat contacts;
• abandonment rates;
• false fraud alerts;
• time to correction;
• customer understanding;
• vulnerability outcomes;
• redress paid; and
• incidents in which human review changed the result.

A model that is efficient for the firm but persistently harmful or incomprehensible to customers creates an obvious supervisory risk.

Online services, information and safety

AI may also sit within services regulated by Ofcom under the Online Safety Act 2023. Whether duties apply depends on the service, functionality, users and statutory scope. The legal assessment should not assume that every generative-AI service is in scope, or that AI-generated content sits outside the regime.

Potential issues include:

• illegal-content risk assessments;
• systems and processes for content moderation;
• risk of recommender systems amplifying harmful material;
• fraudulent advertising duties for relevant services;
• child-access and age-assurance questions;
• complaints and reporting routes;
• transparency and record keeping; and
• provider accountability for safety systems.

Ofcom’s strategic approach to AI for 2026/27 recognises AI both as a source of risk within regulated sectors and as a tool used by the regulator.

The operational lesson is that content generation, recommendation and moderation must be assessed together. A service may have a capable moderation model but a recommender system that repeatedly increases exposure to the same risk.

Safety, professional duties and civil liability

The consequences of an AI error depend heavily on context.

An inaccurate restaurant recommendation is not equivalent to an inaccurate clinical recommendation, safety instruction, legal analysis, credit decision or control-room alert. The greater the potential harm, the stronger the case for validation, restricted use, effective human control and reliable fallback arrangements.

Safety-critical and regulated contexts

Depending on the product and sector, relevant regimes may include:

• health and safety legislation;
• medical-device regulation;
• product-safety and product-liability law;
• transport or infrastructure regulation;
• professional standards;
• safeguarding duties; and
• statutory duties governing public decisions.

The organisation should not begin with the generic capability of the model. It should begin with the consequence of failure in the intended use.

Contract and negligence

Civil exposure may arise where:

• an organisation promises functionality the system does not provide;
• a professional relies on an unverified output;
• confidential information is disclosed to a provider;
• an output contains another person’s protected material;
• service levels or accuracy standards are not met;
• a customer reasonably relies on an inaccurate statement;
• known defects are not disclosed;
• an organisation fails to exercise reasonable skill and care; or
• a supplier’s limitation clause does not allocate the loss as expected.

Liability between supplier and deployer will depend on the contract and facts. Liability to the affected person may follow a different route. The organisation should therefore separate:

• who caused the defect;
• who selected the system;
• who defined the use;
• who made or adopted the decision;
• who communicated the result; and
• who owed the relevant duty.

Professional responsibility

Professionals remain responsible for work presented as their own. AI may assist research, drafting, classification or analysis, but it does not hold the retainer, owe the professional duty or answer to the regulator.

A defensible professional workflow should address:

• confidentiality;
• authority to use client data;
• verification of facts and authorities;
• competence to understand the tool’s limitations;
• supervision;
• record keeping;
• disclosure of AI use where required;
• conflicts and independence; and
• responsibility for the final advice or decision.

The supply chain does not dissolve accountability

AI systems are often assembled from several layers:

• a foundation model;
• fine-tuning or retrieval data;
• a platform or interface;
• integrations with business systems;
• prompts and guardrails;
• a third-party implementation partner;
• internal policies; and
• human users and decision-makers.

When something fails, each party may point to another layer.

The model provider may say the system was used outside scope. The integrator may say the customer supplied the data. The organisation may say it relied on the supplier’s assurance. The user may say they followed the workflow.

Those positions may matter to contribution or contractual allocation. They do not answer the regulator’s first question: which regulated or responsible person owed the duty that was breached?

Procurement evidence

Before deployment, preserve:

• the business case and intended use;
• risk classification;
• supplier due diligence;
• model and version information;
• evaluation criteria;
• test results and limitations;
• security and privacy assessments;
• equality or customer-impact assessments;
• contractual assurances;
• audit, notification and access rights;
• change-control arrangements;
• exit and continuity plans; and
• the approval decision.

Operational evidence

After deployment, preserve:

• model and prompt changes;
• monitoring data;
• overrides;
• complaints;
• incidents and near misses;
• supplier notices;
• identified drift;
• outcome disparities;
• remediation;
• retraining and revalidation; and
• decisions to continue, restrict or suspend use.

Governance cannot be proved by a policy which was never connected to the actual system.

When does a regulatory failure move towards criminal exposure?

The initial classification should remain open to revision.

Indicators that an apparent AI failure may involve criminal conduct include:

• a person deliberately altered data or thresholds to produce a desired result;
• false testing or assurance records were created;
• known limitations were concealed from customers, auditors or regulators;
• an employee used AI to fabricate documents, identities or transactions;
• access controls were bypassed for an improper purpose;
• evidence was deleted or altered after an investigation became foreseeable;
• a false statement was knowingly made during remediation;
• a person benefited financially from the failure; or
• repeated warnings were suppressed while misleading claims continued.

Even then, the precise offence must be proved. Suspicion should trigger preservation and legal analysis, not public accusation.

Other criminal regimes may also be relevant depending on the conduct, including computer misuse, data-protection offences, false accounting, money laundering, perverting the course of justice or offences arising in a safety-critical context.

The response team should therefore maintain two propositions at once:

1. the incident may create serious regulatory exposure without fraud; and
2. evidence of dishonest manipulation or concealment may change the legal characterisation.

A seven-phase response protocol

Phase 1: stabilise the outcome

• Stop continuing harm.
• Identify whether the system should be suspended, restricted or placed into a safe mode.
• Protect affected people and critical services.
• Introduce a reliable manual or alternative process.
• Prevent automated repetition of the same decision.
• Record who authorised the intervention and why.

Suspension is not an admission of liability. It may be the only responsible step while the facts are tested.

Phase 2: preserve the decision trail

Preserve the material needed to reconstruct:

• the business event;
• the person affected;
• the input data;
• the prompt or instruction;
• the model and configuration;
• the output;
• the human review;
• the final decision;
• the communication to the affected person;
• the monitoring alert or complaint; and
• the response.

Follow the fuller preservation framework in Preserving Evidence in an AI-Enabled Fraud Investigation. A screenshot of the output is rarely enough.

Phase 3: define the legal routes

Create an exposure map covering:

• possible criminal conduct;
• personal data;
• consumers;
• equality and employment;
• sector regulation;
• safety and resilience;
• professional obligations;
• contract and civil claims;
• notifications; and
• cross-border consequences.

Identify which questions are legal, technical, factual and governance questions. Do not ask a technical investigator to decide whether conduct was dishonest, or a lawyer to infer model behaviour without reliable technical evidence.

Phase 4: establish ownership and conflicts

Identify:

• the legal entity responsible;
• the system owner;
• the business decision-maker;
• the accountable senior manager;
• the data controller and any processors;
• relevant suppliers;
• individuals whose conduct may be criticised;
• who instructs the lawyers; and
• whether separate representation is required.

Avoid allowing the person who approved the deployment to control the investigation without independent oversight.

Phase 5: test cause, scale and impact

Determine:

• whether the output was wrong;
• why it was wrong;
• whether the human process should have caught it;
• how many decisions may be affected;
• whether particular groups suffered different outcomes;
• whether the same failure exists in another system;
• when the organisation first knew or should have known;
• what harm occurred;
• what remediation is possible; and
• whether the problem remains live.

Do not limit the review to the first complainant if the failure may be systemic.

Phase 6: decide notification and engagement

Consider:

• statutory reporting deadlines;
• regulator notification duties;
• personal-data breach requirements;
• market or contractual disclosure;
• insurers;
• customers and affected individuals;
• law enforcement;
• overseas authorities; and
• preserving privilege while communicating verified facts.

One report does not necessarily satisfy another regime. Coordinate the factual account so that speed does not produce inconsistency.

Phase 7: remediate and prove

Remediation may require:

• correcting individual decisions;
• identifying the affected population;
• redress;
• retraining or replacing the model;
• changing thresholds or use restrictions;
• improving human review;
• revising notices and consent or lawful-basis analysis;
• supplier action;
• governance changes;
• disciplinary action;
• retesting;
• independent assurance; and
• board monitoring.

The organisation should be able to show not only that a fix was announced, but that it worked.

What regulators and claimants are likely to ask

The precise questions will depend on the legal regime, but scrutiny will often converge around the same evidence.

Before deployment

• What problem was the system intended to solve?
• Was AI necessary and proportionate?
• Who approved the use?
• What were the foreseeable harms?
• Which people or groups could be affected?
• What testing was performed?
• What limitations were known?
• Was the system used within its tested scope?
• What human oversight and fallback existed?

During operation

• What performance and outcome data were monitored?
• Were complaints treated as possible system evidence?
• Were disparities or drift identified?
• Were supplier changes assessed?
• Could staff challenge the system?
• Were people able to contest adverse decisions?
• Did management receive meaningful information?

After the incident

• When did the organisation first know?
• What was preserved?
• What immediate harm was stopped?
• Was the problem systemic?
• Were earlier decisions reconsidered?
• Which regulators and individuals were notified?
• Was the account accurate and consistent?
• What redress was offered?
• How was the fix validated?

An organisation that cannot answer these questions may face greater difficulty than one whose system failed despite a documented and genuinely operating control environment.

Board questions

Boards and senior leaders should ask:

1. Which AI systems can make or materially influence decisions about people, money, safety or legal rights?
2. Which legal entity owns each system and which senior person is accountable?
3. What would failure look like for the affected person, not only for the organisation?
4. Which systems use personal or special-category data?
5. Where could bias, misleading content or unequal outcomes arise?
6. Which decisions require meaningful human review?
7. Can the organisation reconstruct a decision after the event?
8. What evidence supports claims made about the system’s accuracy, fairness and safety?
9. How are model changes, drift, complaints and near misses monitored?
10. Can the organisation suspend the system without losing the service entirely?
11. Do supplier contracts provide the information, access and cooperation needed for an investigation?
12. Which incidents require notification, to whom and within what period?
13. How will affected decisions be identified, corrected and redressed?
14. What evidence would prove that remediation operated in practice?

The board does not need to become a model-development team. It does need enough reliable information to govern the consequences of using the system.

Common mistakes

“No one intended it, so there is no legal issue”

Many regulatory and civil duties do not require dishonesty or intention.

“The supplier is responsible”

The supplier may have contractual or legal responsibility. The deploying organisation may still owe duties to customers, employees, data subjects or a regulator.

“There was a human in the loop”

The phrase proves nothing without evidence of authority, competence, information, time and actual challenge.

“The overall accuracy was high”

Aggregate performance may conceal severe harm, unequal outcomes or an unacceptable error type.

“We removed protected characteristics”

Proxy variables and historic patterns may remain, while removing the data needed to test disparities.

“It was only a hallucination”

That does not answer who deployed the system, what it was authorised to communicate, why the error was not caught or what legal consequence followed.

“The policy required verification”

The relevant question is whether verification occurred and whether the operating environment made it realistic.

“We fixed the model”

That does not address past decisions, affected people, notifications, redress or proof that the fix is effective.

“AI regulation has not arrived”

Existing law already regulates data, decisions, communications, services, safety and professional conduct. The technology does not displace those duties.

Frequently asked questions

Conclusion

The most dangerous classification error is not a model error. It is the organisation’s error in treating “not fraud” as “not serious”.

Fraud requires proof of dishonesty and the other elements of the offence. That boundary matters. It prevents criminal language from replacing analysis.

But an AI system can still expose an organisation to regulatory action, redress, civil claims, professional discipline and profound reputational damage. The responsibility usually rests not in the machine as an abstract actor, but in the human and organisational decisions that selected it, supplied its data, defined its authority, relied on its output and responded to its failure.

The defensible organisation can show:

• what the system was permitted to do;
• who owned the decision;
• which risks were anticipated;
• how performance and outcomes were tested;
• how people could challenge the result;
• what evidence was preserved;
• how harm was stopped;
• which legal routes were assessed; and
• whether remediation worked.

The final principle is:

AI changes the mechanism of failure. It does not erase the duties attached to the decision.