Skip to main content

Stress Testing Microsoft Copilot vs Claude vs ChatGPT

AI Bakoff with MS CoPilot

This weekend, I did a real world Microsoft Copilot vs Claude vs ChatGPT bakeoff while wrapping up a lead magnet calculator. In preparation for a Microsoft call to discuss an AI Copilot rollout, I wanted some hands on experience.

The Bakeoff Workflow

  1. Take a detailed calculator requirements doc (AI generated from source code).
  2. Recreate a simplified version in Excel via prompt.
  3. Document the structure.
  4. Translate the workflow into an executive ready PowerPoint story.
  5. Use the output as preparation for a Copilot rollout conversation.

This would be a day of work for multiple people. The project was complete in less than an hour.


Phase One: Translating the App into Excel

The spreadsheet needed:

  • Clear input structure supplied by a 400 line markdown file.
  • Clean calculation logic
  • Organized output summary
  • Executive ready formatting for review and sign off
Microsoft CopilotClaudeChatGPT
Copilot fragmented the logic across multiple tabs. Inputs and outputs were not logically grouped. Structural coherence was inconsistent. If an AI tool creates cleanup work, the productivity gain erodes immediately.Claude generated a tight, single page spreadsheet. Inputs were grouped cleanly. Calculations were centralized. Outputs were summarized clearly. It felt intentional and the result was the best of the group.ChatGPT produced a multi tab structure with clear separation between inputs, logic, and results. It was operationally sound and logically organized.
It required slightly more navigation than Claude’s single page approach, but the structure held.
Microsoft CoPolot ExcelOpen AI Excel

Phase Two: Explaining the Build in PowerPoint

I have never been a fan of PowerPoint. It is a corporate time and knowledge sinkhole. My hope is one day data / knowledge management tools paired with LLMs will force PowerPoint to evolve or go away.

PowerPoint exists as a corporate knowledge artifact that memorializes a point in time. In concept that would be a great thing if the real story and context wasn’t lost in meetings and presentations where PowerPoints are delivered. Microsoft has all of the pieces to the puzzle, so I am blown away they haven’t put it all together.

Clearly this stress test wasn’t going to be transformational to my way of working… At minimum, I wanted to produce a single slide that would explain my app design workflow, to highlight how I was using AI.

  • How the idea evolved
  • How AI accelerated development
  • Where structure improved
  • Where friction was eliminated
Microsoft CopilotClaudeChatGPT
Copilot generated an image instead of an editable diagram. My issue is when shapes cannot be modified, it becomes static decoration. Even the text was encoded as text which is annoying.Claude produced a comprehensive diagram with strong narrative flow. It mapped the journey clearly and felt cohesive. Text was editable.ChatGPT generated a simpler diagram, fully editable in PowerPoint. Less polished, more modular.
Microsoft CoPilot PowerPointClaude PowerPointOpenAI PowerPoint

My findings with Microsoft Copilot so far..

Copilot’s core advantage is integration within Microsoft 365. Outlook was not part of this evaluation but I am praying when I get to the proof of value, it is the star of the show. The Excel and PowerPoint experience was underwhelming for creation. However, I did use co-pilot to evaluate and edit my Claude produced Excel. It did a great job with that task.

Adoption fails when cognitive load remains unchanged. Frustration happens when more time and cognitive load are required than the previous solution.. Without a major payoff in the form of pain reduction or value creation, its tough to recover.

Bottom line: Claude felt magical, and Copilot felt like something I experienced 18 months ago in ChatGPT. Enterprise platform alignment with Office and Azure, security, and wider distribution are real value drivers. That feeling of being behind could make this an acceptable solution.

My Strategic Criteria for Evaluating Copilot

Productivity and Communications Compression Across Microsoft 365

Copilot’s primary strategic function is to compress knowledge work inside the Microsoft ecosystem. Copilot is not designed to replace core application functions; it is designed to accelerate them.

When it comes to communication (Email and Teams), my hope is Copilot will clearly increase the velocity of information consumption and delivery. If not, upcoming proof of value exercise could be short lived.

My primary objectives as I evaluate CoPilot…:

  • Streamline Email search (Gemini in GMail has been a game changer).
  • Speed to response via email
  • Shorten drafting cycles.
  • Consolidation of meeting summary tools into 1 repository.
  • Speed spreadsheet modeling
  • Automate presentation generation

Provide a secure, standardized AI layer across the organization

Security is a major concern for every operator and executive when it comes to these AI models. Copilot provides at least one controlled AI entry point with potential access to confidential data.

My Biggest Concerns as I Continue exploring

  • Training focused on value creation – Understanding the span of capabilities is important but connecting business challenges to tech is where we will create value.
  • Clear use case alignment- The gap between expectations of possibilities, and real feature availability is a concern I want to remove early.
  • Adoption Management – If users do not adopt, it is a failure. If Co-Pilot fails, we are going to do it fast and move on to the next alternative.

Without high value use cases, adoption, and education AI becomes just another data tool that blames bad data or process rather than becoming an enabler that reduces operational drag.


Final Take

AI productivity is not about who generates prettier demos. Real AI success requires distribution of knowledge and experience across a team. Data alignment and influence are about getting a group of people rowing at the same speed and in the same direction. AI is the same data activation and knowledge delivery exercise as analytics, so I feel well equipped to take it on!

There are many other bright spots for Microsoft and AI including the work I have done in Azure and recently with Power BI MCP Server at BIChart.

In this test, Claude shined the brightest. I am still excited to do a proper CoPilot proof of value and see how it goes!

March Madness: The Road to AI

Mach Madness Bracket AI

March madness is our favorite time of year where the top college basketball programs face off on their road to the Final Four. March Madness earned it’s name from intense competition and exciting buzzer beater finishes!

In the spirt of March Madness, we have our own road to AI where we are looking at 4 important factors that will directly influence near term AI adoption and success. Our team reviewed a list of 15 topics and narrowed it down to our own final four for 2024!

1 AI Ethics and Privacy: AI ethics and privacy tackle the moral principles and data protection measures critical to maintaining user trust and upholding human rights in the digital age.

2 Large Language Models: Large language models, like GPT, have transformed natural language understanding and generation, enabling more sophisticated and nuanced human-AI interactions.

3 Data Governance: Data governance ensures the proper management, quality, and security of data assets, serving as the backbone for trustworthy AI systems.

4 AI Chatbots and Co-Pilots: AI chatbots and co-pilots are enhancing work productivity and knowledge experience through large language models.

In March, we going to deep dive into these topics and let them face off head-to-head. We are set for an exhilarating journey of discovery and debate while enjoying a few weeks of exciting basketball at the same time! Join our linked in newsletter for updates!

AI Powered Picks for the 2024 March Madness

We created a GPT March Madness bracket bot available in OpenAI GPT Store to help anyone wanting to make pics based purely on season stats. The beauty of march madness is that the stats don’t matter as teams face off. We intend our stats driven bracket to be busted by the end of the first weekend!

View GPT Powered Picks on ESPN

New Microsoft Copilot Studio Custom Actions

Azure CoPilot Studio

New Azure Copilot Studio custom actions have opened the door for us to connect live, connected Salesforce metric and data dictionaries into the MS Copilot experience. Over the weekend I jumped into Azure and setup a functioning Azure Copilot, trained on our website data, that is available for you to try out below. A little bit of work and reading landed us in the same place we found ourselves a few weeks ago while testing OpenAI’s GPT actions for the first time. In a similar process, I embedded our DataTools Pro app as an action, in the same time it took to finish a cup of tea.

Unlike OpenAI, Azure OpenAI and now Azure Copilot are designed with enterprise in mind with the full suite of Azure services behind it.

DataTools as a GPT

Microsoft Copilot Bot Live Demo

This weekend, I dug in and with only a few clicks, built a co-pilot built a co-pilot based on the DataTools Pro website. With a little more work, we were security connecting in real time to DataTools Pro API and surfacing Salesforce metrics as context to Copilot our own business. We will continue to update this live demo below with our DataTools API demo account connected to Salesforce Essential Metrics.

Ask Questions about DataTools

Azure OpenAI and Copilot won’t fix your data

We are in unprecedented times with the speed that these AI advancements are rolling out and evolving. The real benefit of a Copilot is:

  1. Increasing speed and ease for consuming large bodies of information
  2. Improving the level and depth of understanding (for people who are inquisitive)
  3. Translating and communicating information (text and visual).

While the innovation and art of the possible is very exciting, a sobering reality is you still need to double down on the same data and metadata management and governance.

The path is clear with Azure AI services.

Microsoft has done an incredible job weaving AI into the existing suite of data services and tools..

As we officially roll out new and novel solutions with our DataTools App for Salesforce, we will continue to integrate our APIs throughout the Azure Open AI and Copilot stack. Schedule a call to learn how we can help bring fully trained co-pilots to your organization!

Check out Microsoft Marketing on Copilot Studio


Learn about our DataTools Pro API