Checking data access in an AI application
Map an AI application’s data, permissions, external inputs and outputs before letting it handle business information.

An AI feature can expose information through more than a prompt box. Connected files, application permissions, logs and exported answers all belong in the review.
Before you put an AI application into use, map what it can read and where its results go. A promise that prompts aren’t used for training still leaves questions about storage, access and connected services.
Map the information and copies
Record the input sources, connected repositories, conversation history, logs and destinations for generated results. Identify the provider and any other services involved.
Ask how long each copy remains, who can access it and what happens when the account or service is closed. Check the exact product, plan and configuration being proposed.
Canada’s privacy authorities’ generative AI principles address legal authority and safeguards for personal information throughout the system’s lifecycle. Check which privacy law applies to you (PIPEDA or a provincial act) before the application handles personal information.
Check permission enforcement outside the model
Test the information an ordinary user can retrieve and the information they should be denied. Include a user with fewer permissions and one whose access has been removed.
Ask the application owner which system checks each user’s permissions. Telling the model to keep something confidential won’t protect it; the systems holding the data have to check access.
Review service accounts and connector permissions separately from the user’s sign-in. A broad connector can create access that the intended user wouldn’t have through the original application.
Treat retrieved content as untrusted input
Documents, websites and messages can contain instructions that attempt to redirect an AI application’s behaviour. This is known as prompt injection. The application has to be built to handle it, especially when it reads outside material or can take actions.
OWASP’s sensitive-information guidance describes disclosure risks and the limits of prompt restrictions. Ask the developer to explain the controls and test them with harmless, fictional examples.
Don’t let a generated instruction be enough to send a confidential file or change a business record. A suggestion from the model should still go through the same permission checks and sign-off as any other action.
Inspect outputs and operational records
Check whether answers reveal information from another user’s context or include sensitive content that the task doesn’t need. Test generated attachments and exports as well as the visible chat response.
Review diagnostic logs and alert emails. Troubleshooting shouldn’t silently create a widely accessible copy of the original business records.
Understanding why a model produced an answer can help investigate it, but it won’t tell you whether the data stayed confidential or whether the user should have seen it.
Prepare a response to unexpected disclosure
Define who can suspend the feature, remove a connector or revoke access. Keep a route for staff to report an unexpected answer or upload promptly.
Record the affected information and available evidence using your incident process. Under PIPEDA, a breach of personal information has to be reported to the Privacy Commissioner when it creates a real risk of significant harm, so assess each case against the facts.
Start with a narrow use and retest changes
Begin with sample data and a narrow task. Record the accepted permissions, limitations and checks before expanding access.
Recheck after changing providers, connected data or action permissions. The public-AI data handling guide covers what staff may enter; this review covers the application around that input.


