General Manager
If I ask an internal assistant a question about a document I am not allowed to read, the answer should respect that restriction. A summary can disclose the same information as a download.
That sounds obvious. It becomes more complicated when a document has been copied into an index, split into passages, retrieved alongside other material and quoted in a conversation that stays open for several days.
Permission testing for retrieval augmented generation, or RAG, checks that each user receives only the information they are authorised to access throughout retrieval and response generation. It also checks what happens when permissions change.
I would treat it as a set of observable tests across the system. An access controlled document library is an important starting point, but it does not establish what happens to every derived copy.
Use synthetic documents in an isolated test environment. Give each document a distinctive marker that will be easy to spot in retrieved text, answers and logs.
For example, create a general purchasing policy, a finance appendix and a project document restricted to one team. Use test identities with clearly different access rights.
The marker is test data, not a real secret. Its purpose is to make an unexpected disclosure visible without exposing actual confidential material.
Write down the access matrix before running the assistant:
| Test identity | General policy | Finance appendix | Project document |
|---|---|---|---|
| General employee | Allowed | Denied | Denied |
| Finance reviewer | Allowed | Allowed | Denied |
| Project member | Allowed | Denied | Allowed |
| Unauthenticated request | Denied | Denied | Denied |
This matrix describes the example test environment. Your production rules may be more complex, especially when document access depends on multiple groups, tenant boundaries or sensitivity labels.
Run questions that would naturally lead to each document. Check both the final answer and the material retrieved for generation.
If a restricted passage reaches the model but happens not to appear in the final answer, the retrieval boundary has still failed. I would not rely on the model to keep that passage secret because a prompt told it to.
Microsoft's secure multitenant RAG guidance describes the need to limit grounding information to what the tenant and user are authorised to access. The implementation depends on the architecture, so the test should follow the actual path from identity to retrieval.
The interface needs attention too. A source title, citation excerpt or suggested follow-up can reveal restricted information even when the main answer is careful. Record those outputs in the test evidence.
Begin a conversation as a user who can access the finance appendix. Then revoke the relevant permission and repeat the question according to the system's documented refresh behaviour.
Check a new conversation and the existing one. Review retrieval, answer caches and any conversation history that can be reused as context.
The system should have a stated policy for material already shown to a user and for what can be used in later answers. A revocation cannot make someone forget information they legitimately saw earlier. It can change what the service is allowed to retrieve or reveal next.
Record how long permission changes take to propagate. If the architecture has a refresh delay, the owner needs to know it, assess whether it is acceptable and account for it in the operating procedure.
Repeat the exercise when a document is removed, moved to a different group or replaced with a less widely accessible version.
A request to compare two policies may retrieve one permitted source and one restricted source. The assistant needs to complete only the part the user is allowed to access, or explain the limit without revealing the restricted material.
Include attempts to obtain a paraphrase, a list of numbers or a confirmation that a particular fact exists. A request does not need to ask for the document by name to disclose its contents.
For a service used in Arabic and English, run equivalent permission tests in both languages and in realistic mixed language requests. Access rules should not depend on the language of the question.
These are examples of tests to include, not a claim that a short checklist covers every attack or disclosure path. More sensitive systems need a wider security evaluation.
For each test, keep the identity, group membership, source version, expected access, actual retrieval, final output and application configuration. Store the record in an appropriately restricted location.
Retest after changes to identity handling, indexing, retrieval filters, caching or conversation memory. A model change may also justify rerunning the relevant evaluation, particularly where output handling has changed.
The result should make a failure easy to describe: this identity received this passage through this path, under this configuration. That is much more actionable than a general statement that the assistant passed security testing.
This is the level of specificity I want in Altaius's data and AI work. An enterprise assistant becomes more useful when the team can explain both what it knows and what it is permitted to show each user.
