Home > Security and Compliance Blog > Cybersecurity Risk Management > Why Regulated Industries Can’t Bolt Privacy Onto Their AI Projects

Why Regulated Industries Can’t Bolt Privacy Onto Their AI Projects

by Uri Kedem updated August 21, 2026 Cybersecurity Risk Management

Reading Time: 11 minutes

A compliance officer at a mid-sized regional bank watches an AI chatbot pilot demo, and the first question out of her mouth isn’t “will it work?” It’s “where’s our customer data going?” That instinct, described in a recent SC World perspective piece by David Balaban, is the correct one, and it’s also the instinct too many financial services, healthcare, and insurance organizations still skip past on their way to a pilot launch.

The argument Balaban makes is simple and, for regulated industries, uncomfortable: privacy cannot be a phase-two concern for AI projects. A retailer that leaks customer PII can patch the bug in next sprint and move on. A hospital running the same playbook is looking at a HIPAA violation, breach notification obligations, and a board-level conversation about how this happened. The stakes are not symmetric, and treating an AI pilot like a normal software rollout ignores that asymmetry.

Table of Contents

There is no breach, no incident, and no named enforcement action behind this piece, it is thought leadership, not a case study. What makes it worth Kiteworks’ attention is topical fit: the article walks through data flow mapping, retention policy, access records, and the NIST AI Risk Management Framework, using GDPR and HIPAA to frame the financial and regulatory stakes. Those are exactly the two frameworks Kiteworks is certified against, and the prescriptions in the piece map closely onto how a governed data layer is supposed to work in practice.

This post walks through Balaban’s argument, verifies the one hard figure in it, and connects the practices he recommends, mapping data before training, screening for PII, keeping humans in regulated decisions, and adversarial testing, to what a compliance officer or CISO should expect from a Kiteworks secure data exchange deployment supporting AI initiatives in a regulated environment.

Key Takeaways

1. Privacy has to be designed in before the pilot, not patched in after.

Financial services, healthcare, and insurance organizations face breach notification duties, regulator scrutiny, and board escalation that a typical software bug never triggers, so AI data handling decisions need to be made before the first prompt is written, not after a complaint arrives.

2. LLMs don’t distinguish protected data from ordinary text.

Training logs, chat transcripts, support tickets, and unscrubbed spreadsheets all look like tokens to a model, which means an organization’s own data hygiene, not the model’s judgment, is the only thing standing between a training set and a compliance violation.

3. Auditors want three specific artifacts: a data flow diagram, a documented retention policy, and access records.

Vague assurances about “responsible AI” don’t satisfy an assessor. Producing those three items on demand is the actual bar, and the NIST AI Risk Management Framework gives teams shared vocabulary for building them.

4. Regulatory accountability does not transfer to the AI vendor.

GDPR, the EU AI Act, and FTC enforcement history all treat the deploying organization, not the model provider, as responsible for what the data does, a subscription to a third-party AI tool does not move that liability.

5. Governance has to cover human and agent access under one policy, not two.

Any AI initiative that touches regulated data needs the same access controls, retention rules, and audit logging applied consistently whether a person or an AI agent is the one making the query.

The Compliance Officer’s First Question: Where Is Our Customer Data Going?

Balaban’s opening scene is worth sitting with. A bank’s compliance officer watches a chatbot demo and asks about data destination before asking about functionality. That ordering is not paranoia; it’s the job. In regulated industries, “does it work” is a product question. “Where does the data go” is a legal and reputational question, and it has to be answered before the product question even gets asked, because the answer determines whether the pilot is allowed to exist.

This is the persona Kiteworks writes for by default: the CISO or Deputy CISO who owns AI risk regardless of who deployed the tool, and the Chief Compliance Officer or Head of GRC who owns the evidence package when a regulator asks for one. Neither of those roles is served by a vendor’s confidence that “the model is well-behaved.” They’re served by access controls, a documented data governance structure, and an audit trail that can be handed to an examiner without a scramble.

You Trust Your Organization is Secure. But Can You Verify It?

Read Now

Why Financial Services, Healthcare, and Insurance Can’t Patch Their Way Out of an AI Privacy Failure

The contrast Balaban draws is the clearest part of the piece: a retailer that exposes customer PII patches the flaw and ships a fix next sprint. A hospital that runs the same playbook is staring down a HIPAA violation, mandatory breach notifications, and a board that wants to know why nobody caught this before launch. Insurance carriers and financial institutions sit in the same position, GLBA, state insurance regulators, and functional banking regulators all treat customer financial and health data with a different level of seriousness than a retail loyalty program’s purchase history.

That asymmetry is the entire argument for building privacy in from day one. It’s not that regulated organizations move slower out of caution; it’s that the cost of a privacy failure in these sectors is categorically different, and no amount of post-launch cleanup buys back the notification clock, the regulator’s attention, or the customer trust. Kiteworks’ compliance heritage, FedRAMP Moderate authorization, FIPS 140-3 validated encryption, and single-tenant deployment options, exists precisely because this class of buyer needs evidence, not assurances, before data ever reaches a model.

LLMs Are Hungry: How Training Data Ingestion Creates Unmanaged PHI and PII Exposure

Balaban’s description of large language models as “hungry” is a useful shorthand for a real technical problem. LLMs pull in training logs, chat transcripts, support tickets, and unscrubbed spreadsheets, and none of that data arrives labeled. As he puts it, none of it knows it’s protected health information or a customer’s Social Security number, it’s just tokens to the model. The model has no innate concept of PHI or PII classification, that judgment has to happen before ingestion, not during it.

This is why data classification and access governance sit upstream of any model, not inside it. If a support ticket containing a customer’s account number and diagnosis code gets swept into a fine-tuning set because nobody mapped where that ticket lived or who could query it, the model has already learned something it shouldn’t have, and there’s no reliable way to make it un-learn it. Data classification and data minimization aren’t nice-to-haves in an AI project; they’re the only mechanism that prevents this category of exposure before it happens.

Mapping Data Flow Before You Write a Prompt

Balaban’s practical advice is to map data flow before writing a single prompt: where does the data live, who can query it, and does the vendor retain conversation logs for training, and for how long. These are basic questions, and the fact that they still catch teams off guard says something about how AI pilots typically get started, with a proof-of-concept mentality that skips the due diligence a production data project would require by default.

A data flow map isn’t a diagram for its own sake. It’s the artifact that lets a compliance team answer, specifically, which systems a given AI feature touches, which of those systems hold regulated data, and what happens to a query or a document once it leaves the organization’s control. Without that map, “we don’t think the vendor retains our data” is a hope, not a control. Zero trust data exchange principles, verify every access request, log every transaction, never assume trust based on network location, give teams a concrete way to build that map instead of guessing at it.

What Auditors Actually Want: Data Flow Diagrams, Retention Policies, and Access Records

Balaban is specific about what auditors expect to see: a data flow diagram, a documented retention policy, and access records. That’s a short list, but each item on it has to be produced on demand, in a form an outside assessor can read and verify, not reconstructed after the audit notice arrives.

Access records in particular are where a lot of AI pilots fall down. If an organization can’t produce a clean answer to “who queried this AI system, when, and with what data,” it doesn’t matter how good the model’s guardrails are, the evidence gap itself is the finding. This is the Chief Compliance Officer’s most distinctive problem: an audit trail that technically exists somewhere in a log file is not the same thing as one that’s evidence-quality and can be produced inside the regulator’s timeline rather than weeks after the request lands.

The NIST AI Risk Management Framework as Shared Vocabulary

Balaban points to the NIST CSF family’s AI counterpart, the NIST AI Risk Management Framework, as the tool that gives legal, security, data science, and procurement teams a shared vocabulary for talking about AI risk. That framing checks out independently: the AI RMF’s Govern, Map, Measure, and Manage functions are built specifically to give cross-functional teams, and the auditors and regulators they eventually talk to, a common structure for describing what controls exist and why.

It’s worth being precise about what the framework is and isn’t. NIST’s AI RMF is voluntary and not something an organization gets certified against; nobody audits a company “to” the AI RMF the way a C3PAO assesses CMMC compliance. Its value here is exactly what Balaban describes, a common language for the data flow diagram and the risk register, not a compliance checkbox in itself.

Four Practices That Separate Organizations That Pass AI Audits

Balaban lists four practices that distinguish the organizations that pass AI audits from the ones that don’t:

Map every data source before training begins. Know which systems feed the model and which of those systems contain regulated data before the first training run, not after a question comes up in review.
Screen training sets for PII with dedicated tooling. Balaban names Microsoft Presidio and AWS Macie as examples of tools built specifically to catch personal data hiding inside a training corpus.
Keep humans in the loop for regulated decisions. Credit decisions, claims determinations, and diagnostic support all belong in a workflow where a person reviews and can override the model’s output, not one where the model’s output is the final word.
Test models with adversarial prompts. Actively try to get the model to leak data it shouldn’t have, before an outside researcher or a curious customer does it for you.

None of these four practices is exotic. What’s notable is how consistently skipping them correlates with the organizations that end up explaining a gap to a regulator after the fact rather than showing a control that was already in place.

Accountability Doesn’t Transfer With a Subscription: GDPR, the EU AI Act, and the FTC

Balaban’s piece leans on two named regulatory regimes to make the accountability point concrete, and both hold up under scrutiny. Under GDPR and the EU AI Act, regulators have made explicit what Balaban states directly: accountability doesn’t transfer just because an organization bought a subscription. Buying an AI tool from a vendor doesn’t make the vendor responsible for how the deploying organization’s data gets used, the deploying organization still is.

He also points to the U.S. Federal Trade Commission, noting it has already gone after companies for quietly repurposing customer data to train models nobody signed off on. That claim matches the broader enforcement pattern independent of this specific piece: the FTC has pursued multiple companies over the past several years for training AI or building profiles on user data collected under different terms than the ones later used to train models, and has required data and model deletion as a remedy in more than one case.

The one hard number in Balaban’s piece is the GDPR statutory ceiling: fines under GDPR can run up to 4% of global annual revenue. That figure is directionally correct, with one wording nuance worth flagging for precision, the GDPR text itself sets the upper tier at up to €20 million or 4% of a company’s global annual turnover for the preceding financial year, whichever is higher, not “revenue.” The two terms are often used interchangeably in commentary and the substance of the point stands regardless: for a multinational bank, health system, or insurer, 4% of global turnover dwarfs a fixed-euro cap and is the number that gets a board’s attention.

Governing Data Access for Humans and AI Agents Under One Control Plane

Every one of Balaban’s four practices assumes something regulated industries increasingly have to state outright: the same access rules, retention limits, and logging requirements apply whether the entity querying regulated data is a person or an AI agent. HIPAA doesn’t care whether a human analyst or an AI agent reads a patient record without authorization, the violation is the same either way. GDPR’s accountability principle doesn’t soften because an agent, rather than an employee, initiated the data access.

That’s the reasoning behind a Kiteworks Control Plane approach to AI governance: one policy layer governing data access, use, and exchange for humans and AI agents together, not a human-access policy with an AI carve-out bolted on afterward. Framing this as governance being “extended” to agents implies agents were previously operating outside any policy at all, which is the wrong mental model and the wrong control design. The correct model is a single set of rules, who can see what data, under what conditions, logged how, that both human users and AI agents operate inside from the start.

How Kiteworks Closes the Gap Between AI Ambition and Regulatory Reality

Balaban’s prescriptions, map data before training, screen for PII, keep humans in regulated decisions, test adversarially, and produce a data flow diagram, retention policy, and access records on demand, describe a set of outcomes. They don’t describe how an organization actually builds the plumbing to deliver those outcomes at scale across dozens of AI initiatives and business units.

That’s where two specific Kiteworks capabilities map onto the article’s argument. Kiteworks Compliant AI governs which content an AI system is permitted to retrieve and applies that policy at the point of retrieval, rather than depending on the model itself, or a downstream scrubbing pass, to catch protected data after the fact. That’s the architectural answer to Balaban’s ingestion problem: if the AI system can’t retrieve unscrubbed PHI or PII in the first place, there’s nothing to un-train later.

On the audit side, the Kiteworks Control Plane produces the unified log and access records that map directly to what Balaban says auditors want: a record of who accessed what, when, and under what policy, consolidated across email, file sharing, managed file transfer, and AI access rather than scattered across a dozen disconnected system logs. Kiteworks’ compliance program addresses HIPAA compliance and GDPR compliance requirements, the same two regimes Balaban uses to frame the stakes in his piece, and includes FedRAMP Authorization, SOC 2 Type II attestation, and ISO 27001-aligned controls that extend to NIS2 and DORA obligations. For a compliance officer whose first question about any AI pilot is “where’s our data going,” that’s the answer that needs to exist before the pilot starts, not after an examiner asks for it.

To learn more about mapping AI data flows, producing evidence-ready access records, and governing AI agent access alongside human access under HIPAA and GDPR, schedule a custom demo today.

Frequently Asked Questions

It depends entirely on what the chatbot can access, not on its stated purpose. If the system can query or retrieve any record containing protected health information, including a support ticket that references a diagnosis or a patient account number, it is handling PHI regardless of the pilot’s original scope. The safer approach is to govern retrieval at the data layer with Kiteworks Compliant AI so the chatbot physically cannot pull PHI it wasn’t authorized to see, rather than relying on the pilot’s intended use case to keep it out of scope. See Kiteworks’ HIPAA compliance page for how the Security Rule applies to AI-assisted workflows.

Auditors in this space typically want three specific artifacts: a data flow diagram showing where data moves, a documented retention policy stating how long data and logs are kept, and access records showing who queried what and when. A policy statement describing good intentions doesn’t substitute for producing those three items on demand. A consolidated audit trail across every system touching the AI workflow is what turns “we have a policy” into evidence an assessor can verify.

No. The NIST AI RMF is voluntary guidance, not a certification scheme, there’s no equivalent to a CMMC assessor evaluating an organization against it. Its value is providing a shared structure (Govern, Map, Measure, Manage) that legal, security, data science, and compliance teams can use to describe AI risk consistently to each other and to regulators. Organizations still need named frameworks like HIPAA and GDPR, along with audited attestations like SOC 2, to demonstrate actual compliance.

Not under GDPR, the EU AI Act, or FTC enforcement history, all of which treat the deploying organization as accountable for what happens to its data, regardless of which vendor’s model processed it. A vendor’s terms of service may limit the vendor’s own liability, but they don’t transfer the deploying organization’s regulatory accountability. That’s why access governance and audit logs need to sit under the deploying organization’s control rather than solely inside a vendor’s black box.

The deploying organization is accountable either way, HIPAA, GDPR, and GLBA regulate the data, not the identity of whoever or whatever touched it. In practice, this means access controls, retention rules, and logging need to apply identically to human users and AI agents under one policy layer, which is the reasoning behind a Kiteworks Control Plane approach: one set of governed rules for both, rather than a human policy with an agent exception. Organizations without a named owner for agent behavior specifically are the ones most likely to discover this gap during an audit rather than before one.

Additional Resources