Dev
Managing Copilot Credit Consumption
Dev.toUnited States · NORTH AMERICA
With Copilot Credits (aka Copilot Studio Credits) becoming more and more key to Microsoft's expanding AI offering we are fast approaching the time when we will need full FinOps for them. Currently the...
With Copilot Credits (aka Copilot Studio Credits) becoming more and more key to Microsoft's expanding AI offering we are fast approaching the time when we will need full FinOps for them. Currently they are used by:
- Copilot Studio users*
- Copilot Studio GitHub Harness developers
- Agent Flow runs
- Workflow runs
- AI Builder actions
- CoWork
- Copilot Code
- Microsoft Managed app users*
*free to users with specific licenses
And this list is only going to grow. Microsoft has promised improved monitoring of these credits, but how is it now? Well I wouldn't say its the best, but we don have some reports, and that's what I want to cover today
- PPAC Reports
- Environment Reports
- Agent/Flow Reports
Note as I said, this is planned to change any day, so keep an eye out for new ones, I've also not covered anything in MAC (M365 Admin Center).
1. PPAC Reports
The Power Platform Admin Center is the place you would expect all of the reports, and there are a few.
Call out here, all of these reports require Global Admin, so you will need to activate your PIMS. Also this is all Copilot Credit consumption in the Power Platform, so includes Agents, Agent Flows, Workflows, and AI Builder.
The first place is the most obvious, Copilot Studio under licenses. Here you have the 3 main reports to download.
All of the reports require you to run a 'create' request, which takes a minute of two, and then you can download the report. That's right no live data here, you have run the report up to the previous day and get the data as a csv. There are 3 date ranges, 30, 60, 90 and 180. 30 is quick, 60 takes a while and everything else takes a long while (and sometimes fails).
First in the report is totals by environment:
Environment Consumption Summary
This rolls up all usage by environment and also includes allocate amount to the environment. There is simply one row per environment, no date data, so the number is for the date range you request (so make sure you remember it). It does show pre-paid vs pay-go as well, which can be useful.
For me this is a pretty useless report, as allowance is by calendar month, so I cant see my utilisation for the current month.
Agent-Level Credit Consumption
This allows you to see agent consumption, with a couple of particularly useful breakdowns.
Channel - where your uses are interacting with your agent
Feature - what generated the charge, that could be generative action, agent flow, graph grounding, etc
Although useful from an analytics and understanding of the agents usage, it again has little help for FinOps, unless you look at doing some sort of post usage billing, which I personally wouldn't recommend.
User-Level Credit Consumption
This report shows all of the usage by user, and will show you what agents they used, AI Builder actions, and so on. This report shows if they were billable or consumption (i.e. free with M365 Copilot) and if they have a M365 Copilot license.
This is a great report if you have a pool of M365 Copilot licenses that you want to allocate to the right users, as this will allow you to see who will get the most benefit from them (within Power Platform).
But again not the best for FinOps, unless you want to post usage charge to the users department.
The good news is we have a couple more reports that are a little more useful for FinOps.
To access these we need to go to the Agent section from the Copilot Studio page in licensing.
The first thing you see here is all of your agents and their live consumption for the month.
This is my go to screen for controlling my agents (not Flows/AI Builder), as I can see in one screen who's using credits.
Even better if you click the 3 dots next to an agent you can set per agent limits. Unfortunately you cant currently set a default for new agents, but you can at least set them a hard limit (well hard as in they can still overspend up to the end of the day they hit the limit)
As this report is useful I wanted to see the API that created it. Microsoft has been working hard on the licensing.powerplatform.microsoft.com api's, and this uses one. The headless approach to licensing management is definitely the future, allowing us to build out our own solutions with complimentary data.
The api is:
https://licensing.powerplatform.microsoft.com/v2.0/tenants/2242945a-4ab9-4132-840e-cce1c66e31bb/entitlements/MCSMessages/resources?fromDate=yyyy-MM-dd&toDate=yyyy-MM-dd&pageNumber=1&pageSize=100
It allows you to see per agent usage by date range, which is exactly what we want. But there is one small problem. The fromDateparameter doesn't work, it only ever sends you from the first day of a month. If you put an older date it just ignores it. This means that if you want to automate this (and got around Global Admin PIM's which is needed) you would lose some or all of the last day of the month, as if you ran it the day after it has reset to the current month.
Luckily this screen also has a download report.
Capacity Consumption Tenant Details
To get to this report you also need to run a 'create' request. You follow the same pattern.
And then export as a csv for Excel.
The good news is this report is not only by agent/flow, but by date. So you can filter out days outside of the calendar month, allowing you to see how much is being used by who in this month. With this data you can do some basic forecasting and track to see if you have enough credits for the month, and who is overspending them.
Environment Level
There is also some data at the environment level, but this is hit and miss.
In Copilot Studio you can see the Operate tab that shows all consumption for your agents (System admin sees all).
Though as you can see, for me it often shows 0 when in fact there was some.
There is also a new cool view for Workflows (https://copilotstudio.**preview**.microsoft.com/) that shows each workflows usage in last 30 days. Sadly it does not cover Agent Flows, only the new Workflows.
So that covers Copilot Studio, but what about AI Builder (which from November has to use Copilot Credits not AI Builder credits as well). Well for that we have the Automation Center (an under utilised feature in Power Automate). It lists every AI builder action by flow, which is ideal for developers to use to track how much and what specific action is spending.
Under the hood the Automation Center is just pulling from the AI Events table, so you can use Dataverse connectors/API's to pull this data almost live.
Note this is only for actions that use credits, so you won't see any Agent Flow or Workflow run costs, only specific actions they use that are chargeable.
Its a shame that all cost transaction are not in this table, as its easy to monitor/integrate with.
Agent/Flow Reports
Finally we have the per Agent/Flow reports. Copilot Studio has the agent monitor screen that shows usage.
The only challenge I would say with this is you cant share access to this with groups, so if you wanted to give a group of uses access to track performance/consumption, they have to be individually shared 🤦♂️.
For Workflows and Power Automate the environment level reporting is Role based, so you see data for the flows you own through the preview agent list screen and the Automation Center.
As you can see the reporting strategy is still disjointed, and trying to manage consumption is not difficult, I can imagine so unexpected bills hitting some organisations (especially as there is currently no way to limit pay-go consumption). The good news is Microsoft has heard everyone, and has promised a big update soon with things like reports, role access, and most excitingly API's