Ever wondered how to investigate Microsoft 365 Copilot Interactions? You’re in the right place! This blog post will guide you through the process of using Microsoft 365 tools, specifically the unified audit log eDiscovery and Communications Compliance, to uncover specific Microsoft 365 Copilot interactions.
Understanding Microsoft 365 Copilot Interactions
In this post, we’ll be focusing on Investigating Microsoft 365 Copilot Interactions to help you better navigate your use of these tools.
A typical interaction with Microsoft 365 Copilot comprises three parts:
- The user formulates a prompt encapsulating the goal, context, expectation, and sources.
- Microsoft 365 Copilot responds to the prompt with its answer.
- Optionally, references to content are included.
Where are Copilot for Microsoft 365 interactions stored?
Just like Team chat messages, compliance copies of these interactions are stored in a hidden folder in a user’s mailbox. These copies are governed by the same Microsoft Purview risk and compliance solutions as other content.
For more information visit Data, Privacy, and Security for Microsoft Copilot for Microsoft 365 | Microsoft Learn
Exploring Compliance Solutions
Microsoft Purview offers several solutions for managing Microsoft 365 Copilot interactions:
- Viewing audit records in the Unified Audit Log.
- Searching for Microsoft 365 Copilot interactions via Content Search or eDiscovery cases.
- Permanently deleting Microsoft 365 Copilot interactions with eDiscovery Premium and PowerShell/Graph API.
- Analysing Microsoft 365 Copilot interactions in Communication compliance.
Viewing Microsoft 365 Copilot Interactions in the Unified Audit Logs
The Unified Audit log is a potent tool that captures audit records across various Microsoft 365 workloads. This includes interactions with Microsoft 365 Copilot, encompassing the time, date, and content of each interaction. However, it’s important to note that the audit log does not include the actual user prompts or responses. For detailed interaction data, eDiscovery tools should be used.
To search the audit logs for Copilot interactions, you need to access the audit log search in the Purview Admin Center. From there, you can select Interacted with Copilot under the Activities – friendly names filter. This will display all the Copilot interactions logged in the audit log. Then filter for other criteria such as date ranges etc. ⚠️Note Audit standard stores audit logs for the last 180 days and Audit Premium (requires E5 Compliance licencing) stores audit logs for 1 year.
For further analysis, you can export the audit log search results to a CSV file. However, the CSV file must be formatted before it can be used. More information on exporting, configuring, and viewing audit log records can be found in the Microsoft Learn documentation Export, configure, and view audit log records | Microsoft Learn.
Copilot Interaction Logs:
Copilot Interaction Logs provide valuable information about each interaction, including:
- RecordID: A unique identifier for each interaction.
- CreationDate: Timestamp when the interaction occurred.
- RecordType: Indicates the type of interaction (e.g., Copilot activity).
- Operation: Describes the specific action taken during the interaction.
- UserID: Identifies the user involved.
- Audit Data Field: Contains additional context-specific information related to the user’s interaction with Copilot. This may include details like the application host, contexts, thread IDs, message IDs, and accessed resources
The Microsoft Learn documentation: CopilotInteraction | Microsoft Learn contains an XML schema example.
Searching for Microsoft 365 Copilot Interactions with Content Search and eDiscovery
To view the Microsoft 365 Interaction messages you can use the Content Search or eDiscovery as the this information is not available in the audit logs.
To use Content Search or eDiscovery, ensure you are a member of the eDiscovery Manager role group in the Purview Admin portal. Select Content Search or eDiscovery . Select Exchange mailboxes as the search location. This is because the Microsoft 365 Copilot prompts and responses are stored in a user’s mailbox. Then choose all or add specific user’s mailboxes as the source for a search query.
Under the search conditions, select Add condition, and choose Type.
Then select Add/remove more options to expand the the lists and pick Copilot interactions as the search criteria. This will display all the Copilot interactions stored in the user’s mailbox.
This creates the Keyword Query Language (KQL) query containing all the different Copilot interactions.
Find Copilot interaction in an App
If you are only interested in Copilot interactions in a specific app them filter your query using ItemClass. You can see how many Copilot interactions there are for each type. For example there are eight Copilot interactions in PowerPoint.
Summary of ItemClass
- IPM.SkypeTeams.Message.Copilot.BizChat (Microsoft Copilot for Bing or Teams Copilot Chat)
- IPM.SkypeTeams.Message.Copilot.Excel
- IPM.SkypeTeams.Message.Copilot.Loop
- IPM.SkypeTeams.Message.Copilot.PowerPoint
- IPM.SkypeTeams.Message.Copilot.Teams
- IPM.SkypeTeams.Message.Copilot.Whiteboard
- IPM.SkypeTeams.Message.Copilot.Word
This is the equivalent to the KQL query:
(c:c)(ItemClass=IPM.SkypeTeams.Message.Copilot.PowerPoint) You can then select one of the items and view the content. For example asking Copilot in PowerPoint to create a presentation using two files as input.
or a Microsoft 365 Copilot prompt asking about a specific terminology, in this case, ISO 27001.
Deleting Microsoft 365 Copilot Interactions
There may be instances where you need to permanently delete confidential or harmful content that surfaced as part of a Copilot interaction, such as dealing with a data breach incident or if the content has classified or malicious information.
Deleting Microsoft 365 Copilot interactions requires creating an eDiscovery Premium case and being a member of the eDiscovery Manager role group in the Purview Admin portal.
First create an eDiscovery Premium Case and define the custodians (users in scope).
Then create your Collection using the search Type = Copilot interactions to find Copilot interaction along with date ranges and other search criteria.
Ensure the search collection does not exceed 10 Copilot interactions for each mailbox, as this is the limitation for deleting the Copilot interactions. This limit helps ensure that this data is removed fast because searching for and deleting Copilot data is meant to be a tool for responding to incidents.
Remove any holds or retention policies from the mailboxes containing the Copilot data.
Finally use Microsoft Graph Explorer or PowerShell to delete the identified Copilot data from user mailboxes.
For full details on the Microsoft Graph Explorer or PowerShell commands see Microsoft reference: Search for and delete Microsoft Copilot for Microsoft 365 data | Microsoft Learn
⚠️Note, eDiscovery Premium is part of E5 Compliance Licensing. More details can be found in the Microsoft Learn documentation.
Analysing Microsoft 365 Copilot Interactions with Communication Compliance
Microsoft Purview Communication Compliance is a feature within Microsoft Purview that helps organizations keep track of and manage different communication channels, including Microsoft Copilot. This solution helps find and resolve potential problems in messages to ensure they follow organizational standards, keep compliance, and lower the risk of regulatory fines.
Microsoft 365 Copilot interactions policies are now available to configure in Communication Compliance is you are a member of the Communication Compliance or the Communication Compliance Admin role group in the Purview Admin portal.
From the Permissions page in Purview portal, assign users to one or more of the following role Communication Compliance role group.
From the Purview Admin Center select the Communication Compliance menu select Policies > +Add Policy and select Detect Microsoft 365 Copilot interactions.
⚠️Note, Communications Compliance is part of E5 Compliance Licensing. More details can be found in the Microsoft Learn documentation.
Microsoft reference: Communication compliance | Microsoft Learn.
Conclusion
The unified audit log, Content Search, eDiscovery and Communications Compliance are powerful tools that can help you investigate Copilot interactions. Using these tools, you can gain valuable insights into how Copilot is used and ensure that your organisation complies with relevant regulations.
Microsoft reference
Microsoft Purview data security and compliance protections for Microsoft Copilot | Microsoft Learn