Recent AI advancements with large language models have broken through and forever changed how we think about information access and retrieval. Metrics and AI is at the top of my mind as AI agents today provide universal translation and curation of information. Here, in our data tools niche where we wrangle and transforming data into information, we are seeing incredible results using AI to write code and deliver the same results that traditionally required an analyst. We don’t believe AI will replace analysts, but we know already that AI augmented retrieval for researching large bodies of information a job better suited for machines. Enterprises need to re-frame documentation as context data generation for people and AI. You will likely see a rise in “knowledge graphs” as a hot topic. Unstructured data has always been deemed “untapped” gold, so now the race down the yellow brick road is on!
Behind the Curtain: Unveiling the Reality of Modern Bots
Chatbots have propelled large language models into the forefront. The benefit of these AI chatbots to individuals is the way an LLM can breaking down knowledge and experience into personalized bite sized information chunks. We are not too far from this being a shared experience in a collaborative setting. This is where we can see early adopters super charge team productivity. The big opportunity is how how enterprises will use AI to curate and deliver information us using the vast collections of empirical knowledge created over time. It goes without saying there are lots of smart people working tirelessly on these problems. Here at DataTools Pro, we are obsessed with this problem as a small scrappy team.
Does self-service analytics help?
Analysts, data scientists, and data professionals have always been required to distill complex business concepts into quantifiable analytics. Business Intelligence (management information systems) and Analytics disciplines still have the same problems today as 15 years ago. AI, LLMs and data platforms will not solve these problems without radical changes how people work.
- Age old “multiple versions of truth” problems still exist. It has moved from spreadsheets to self service reports and dashboards
- Empirical knowledge gained from pulling data together becomes disparate in spreadsheets, documents and PowerPoints, and email.
- Methods to build a live, connected semantic layers to categorize and measure quantitative performance remain siloed and technology oriented.
Reports, dashboards, and data remain the primary delivery mechanism for performance metrics and KPIs. The need for speed to prepare and deliver self-service analytics has shortcut slow moving BI platforms of yesterday. Similarly, modern cloud data platforms have helped democratized working with structured and unstructured data that historically required database administrators, software engineers, expensive technology components. “Deluge” is the best word to describe the state of most enterprises in regard to the number of data and analytics assets flowing thus creating newer “data mesh” and “data fabric” methodologies to help strategize designing systems and process to tackle the deluge problem. We are experimenting with this ourselves with Azure Co-Pilot while our team, data, and metrics library is small.
What about Self Service via Natural Language Requests?
Natural language queries is a feature and not a solution. Many professionals simply do not know what data and metrics are available to start asking questions. This is where AI agents armed with a glossary, semantics and a large body of context data will be transformational. AI co-pilots are still very new, so we are experimenting ourselves what is real vs art of the possible. The keystone is aligning AI and business professionals with a common taxonomy and language and where we are working to build a common thread between business, data, analytics, and soon auto-pilots aiding these teams.
What about data?
Many enterprises do not have enough resources behind data governance and management. I still think this is a massive area of opportunity to somehow democratize and distribute data management in a way that is non-intrusive. Otherwise our point of arrival for AI automation will be autopilots and agents spitting out useless information. Incorrect information leads to mistrust and failed adoption of “co-pilots”.
Data storage is dirt cheap and modern data platforms make it fast and easy to analyze and model data into sophisticated analytics. How do you create universal focus?
All roads to AI metrics and analytics mastery leads back to goals and data governance
Every company has a set of metrics that indicate the health of the business. Your financial metrics within your income statement and balance sheet don’t get a lot of love on social media, but they are the bedrock to your performance (assuming you are a for profit business). Highly sophisticated metrics or amassing hundreds of metrics wont translate to good performance. Universal understanding, consistency, correctness and execution against a finite set of metrics will!
Quick metrics maturity quiz:
- Do you have an inventory of all of the metrics your operational team is using?
- Who are the owners, stakeholders, and oracles (keepers of institutional knowledge)?
- Are you certain those metrics are calculated and deployed consistently across teams and individuals?
- Are your business, technology, data and analytics teams aligned how to implement these metrics into analytics?
Fact Finding Process:
Traditionally, this is the process most consultants utilize to thoughtfully acquire and organize your metrics glossary. Many enterprises already have documents, presentations, or spreadsheets with this information formally gathered. Rarely are they universally understood and up to date.
Getting consensus and universal understanding is slow and cumbersome. It is one of the challenges we wanted to tackle at DataTools Pro
From Metrics to Key Performance Indicators and OKRs
There are a number of different organizing principles and methodologies to translate your organizational goals into metrics. A KPI differs from a metric in that it has a specific target, timeline, and direct impact on your organizational goals and objectives. You may have dozens of metrics without targets, and that is okay. There are a number of widely adopted models to help you formally structure and organize your KPIs:
SMART Goals
SMART created by George Doran that offers a system for organizing and defining and measuring your business goals.
- Specific – target a specific area for improvement.
- Measurable – quantify or at least suggest an indicator of progress.
- Assignable – specify who will do it.
- Realistic – state which results can realistically be achieved, given available resources.
- Time-related – specify when the result(s) can be achieved.
OKR – Object Key Results
- OKR – An objective is a clearly defined, inspirational goal aimed at driving motivation and direction. Key results are specific, measurable outcomes used to track the achievement of the objective.
As you get deeper into the performance management, process improvement, you will discover what works best for your corporate culture.
How are metrics and KPI is evolving with AI
No article is complete in 2024 without a hot take on AI. A lot of the focus in technology and analytics is centered on amassing feeding large volumes of quantitative data into machine learning models to predict outcomes. Now with generative AI, we are vectorizing large bodies of data to train, fine tune or simply retrieve data using natural language requests. Unfortunately, without the right semantics, definitions, governance, and context data, your AI investment won’t feel magical. Our team is racing ahead knowing the path to have meaningful dialogue and results with AI co-pilots requires context. We have taken a novel approach to run along side enterprises on their journey down the yellow brick road to help with our upcoming DataTools Pro metrics analyst!
Join us for a webinar March 13 where we will formally introduce our Metrics Analyst AI.
