Measuring the impact of Microsoft 365 Copilot and AI at Microsoft

It’s clear that we can do incredible things with Microsoft 365 Copilot and AI in general. This technology boosts creativity, streamlines productivity, and it makes data-driven insights more accessible than ever before.

For many organizations, what remains unclear is exactly how to measure AI’s impact. Without that information, businesses are at a loss for ways to articulate value and drive improvement.

At Microsoft, we’re pioneers in every part of AI transformation, from developing the technology itself to implementation and adoption and ensuring its responsible use. Now, we’re leading the way on measuring the returns AI provides—and helping organizations understand exactly how these new tools can help them accomplish their goals.

Although we’re at the beginning of our impact-tracking journey, our framework provides a foundation you can use to measure AI’s value for your organization.

New technologies, new challenges

Any new technology presents challenges for identifying its impact. In the case of AI, which is reshaping the nature of work as we know it and accelerating at an unprecedented pace, those challenges are even more difficult.

As a result, organizations have struggled to identify AI’s impact and apply it to business strategy. As leaders in the AI space, our team at Microsoft Digital, the company’s IT organization, knew we needed to address these questions.

“We started by doing both internal and external research, and what we found was that companies are really struggling with how and where to invest in AI,” says David Laves, director of business programs in Microsoft Digital. “The barrier is that the benefits are impressive, but they don’t always immediately show up on a profit-and-loss sheet.”

Articulating AI’s impact came with several challenges. First, we had to land on a consistent taxonomy for different measurement areas. Then, we needed to make the relevant data accessible, ensure its quality, and confirm that it made sense.

From there, we determined how to tell that story meaningfully through reporting. And finally, we considered ways to translate that story of impact into a path toward greater value and continuous improvement.

“The last thing you want to do is know what you want to measure but not understand how to measure it,” Laves says. “That’s like flying a plane without instruments.”

Measuring progress and unlocking value

The Microsoft Digital AI Value Framework is our flexible, modular tool for measuring the impact of our AI initiatives. With tools for measurement firmly in place, we can effectively demonstrate value and guide further decision-making.

The framework breaks impact assessment into six different areas.

As we developed the framework, we knew that flexibility would be an essential part of measuring our efforts. Not every AI initiative targets every measurement area, so we only apply relevant tracking categories and metrics to any single initiative.

“We’ve seen tremendous business value from this framework, because now we can clearly articulate what we’re driving as a result of our top AI scenarios at the organizational level,” says Don Campbell, senior director of Employee Experience Success in Microsoft Digital. “This gives us real data so we can have deep and insightful dialogue that leads to strategic pathways forward.”

It’s also important to note that not all measurement areas are inherently financial. Some are more about quality, experience, risk, and other factors.

Different measurement areas will also be more relevant for different roles with different behaviors. For example, productivity and efficiency metrics might be more relevant for AI solutions tailored to information workers who need to sift through and synthesize information quickly. On the other hand, improving the quality and consistency of outputs may be more important for technical roles.

“The more targeted, the better,” says Stephan Kerametlian, director of Employee Experience in Microsoft Digital. “The more invested you are in uncovering the hero scenarios at a team, role, or individual level, the more impactful the results will be.”

A foundation for continuous improvement

A framework for measuring impact doesn’t lead to value without incorporating what we learn into a cycle of progress.

At Microsoft, we follow a structured methodology for continuous improvement that includes rapid learning cycles to deliver tangible improvements to our technology and processes. It’s grounded in a relentless focus on creating value by identifying and solving problems across four stages: plan, do, check, and adjust.

The Microsoft Digital AI Value Framework fits neatly into this model. When applying this framework to a specific AI initiative, we follow these four steps:

1. Define: Define the scenarios your AI initiative impacts, articulate the specific measures and targets, and establish baselines. At this stage, we look for a clear understanding of the value we want to track and form a hypothesis.
2. Implement: Here, we implement the AI initiative itself along with our apparatus for tracking value. For us, measuring impact isn’t an afterthought—it’s a key part of the design and implementation of an AI product. The key outcome is an AI Investment that’s deployed and ready for measurement.
3. Measure: At this phase, we capture impact tracking and perception data and compare it against targets and baselines to validate our hypotheses, then determine insights and make recommendations for improvement.
4. Action: Finally, we implement recommendations to continuously improve our work. That includes building a publishing rhythm, tracking trends, articulating learnings, and creating action plans. It might also result in suggestions for further AI initiatives.

The continuous and iterative nature of the framework unfolds through regular reporting and consultation. At Microsoft Digital, we track metrics on an ongoing basis during monthly operating reviews conducted by our team of pillar leads and sponsors. This group of leaders owns horizontal accountability across three pillars: transforming and securing our network and infrastructure, revolutionizing user services, and accelerating growth for our corporate functions.

To maintain visibility over our target metrics, we use bowler scorecards that clearly demonstrate whether we’re meeting our targets over time. Meaningful reviews depend on honest assessments of impact, so we’re not afraid to “embrace the red”—accepting negative results as valuable data to fuel our efforts.

This approach empowers both strategic and tactical decision-making. At the macro level, the framework enables better high-level AI investment decisions. At a lower altitude, it gives implementation teams the guidance they need to design, deploy, and track their initiatives.

Monthly operating meetings give us a forum for interpreting results, validating hypotheses, and forging a path forward. This kind of ongoing, flexible measurement is crucial for a technology that’s evolving as fast as we can implement it.

“What success looks like today in terms of the technology at hand is going to look vastly different in the months and years ahead,” Kerametlian says. “As we unlock new capabilities, as subject matter expertise around AI continues to deepen, as people’s proclivity to use Microsoft 365 Copilot increases and matures, these iterations will be crucial for our learning curve.”

Unlocking insights from our ongoing AI initiatives

The Microsoft Digital AI Value Framework is a relatively new development, but we’re already successfully applying it to initiatives across the company. We’ve only just begun tracking the success of these projects, so results are preliminary, but we now have a secure foundation for assessing their impact over time.

Among others, we’re currently measuring the following projects:

• For Global and Technical Support agents, we’re validating productivity and efficiency by tracking the amount of time employees save when they seek help and the number of tickets they log using the assistant.
• For a GitHub agent that supports our engineers, the primary tracking areas are the quality and security of its code outputs. Naturally, those have knock-on effects on productivity and the employee experience by reducing workloads and building trust.
• Our Global Workplace Services team has an AI initiative that aims to deliver greater energy efficiency at Microsoft facilities. In this case, we can track direct cost savings associated with reduced power use.
• The Device, Network, and Infrastructure team is undertaking a number of sub-initiatives under the umbrella of improving network performance and issue remediation. We’re tracking their projects across several key areas, including productivity and efficiency in terms of time savings, as well as cost savings and security and risk management.

The way we’re tracking these initiatives demonstrates the framework’s flexibility and the need to apply measurements selectively to capture the most useful insights. As you look for ways to track the success of your own AI projects, consider the specific metrics that apply, then commit to measuring them over time.

“The impetus is measuring through the lens of value and improving on it to align with our culture of continuous improvement,” Laves says. “We align on our AI placements regularly through our monthly operating reviews and bowler scorecard tracking, and that gives us the visibility and data we need to drive decisions for which investments to pursue and how to improve them.”

It’s been a lengthy journey to create this structured framework. Like any capability, there’s a maturity curve to measuring AI’s impact. It might make sense for your organization to start with simple metrics using built-in tools like the Copilot adoption report. This out-of-the-box dashboard for Viva Insights provides quick and easy access to data around adoption, impact, and learning, including the number of Copilot actions taken, Copilot-assisted hours, and Copilot-assisted value.

The sooner you take a thorough and thoughtful approach to tracking your AI initiatives, the more easily you’ll slip into a virtuous cycle of continuous improvement.

“We strive to be the first and best in terms of testing, deploying, adopting, and measuring the value of our technology,” Kerametlian says. “We have to learn first and provide leadership on AI tools themselves and the methodologies we use to uncover the value they have to offer.”

The first step is prioritizing measurement as a core implementation discipline. By following Microsoft’s lead and learning lessons from our journey, you’ll discover which AI initiatives can help you capture the most value and accelerate your business goals.