How Organizations Can Measure Their Data & AI Literacy
Note: a version of this article was first published for Forbes Business Council on Jan 27, 2025.
By Ben Jones, Co-founder and CEO of Data Literacy
In today’s data-fueled economies, organizations of all types and sizes need to successfully channel data and artificial intelligence (AI) in order to compete and achieve their objectives. A question that naturally arises in the mind of the executive, then, is – how well are we doing?
I’d like to share what I’ve learned over the past five years running the Data Literacy Score™ Organizational Assessment. There’s no single, perfect metric for assessing maturity in data and AI. Instead, there are four key dimensions that constitute a framework for establishing a baseline and tracking progress.
1. Subjective: What Employees Think
The first dimension, and the easiest to assess, reflects the opinions and perspectives of the employees. While this dimension is, by definition, subjective, it’s by no means invalid. An executive can learn a lot by paying attention to this dimension.
How do you find out what your employees think about how data and AI are being used by the organization? You ask them. This can take the form of everyday conversations in which you listen carefully to how people on your team respond to open-ended questions. And it can take the form of more formal surveys, in which you solicit employees to anonymously rate their team and the organization as a whole, and provide their own suggestions for improvement.
Ask them about your systems, and the quality of the data itself, but don’t stop there. Go on to ask them about other factors that relate to their environment and their work. In our assessment, we include questions about the following seven factors:
- Purpose: Are data and AI being used to achieve goals and objectives?
- Ethics: Are they being used in a way that’s helpful and not harmful?
- Data: Is timely, relevant, high quality data available and accessible?
- Technology: Do tools and systems function, and integrate well?
- People: Do team members have sufficient knowledge and skills?
- Process: Do team processes run smoothly thanks to data and AI?
- Culture: Do teams enable and reward successes with data and AI?
Gauging the thoughts of your team members may be a subjective lens, but it’s a highly valuable one through which to look at your organization. You can identify pain points that are blocking your teams, and you can develop strategies to break through these barriers.
2. Objective: What Employees Know
As valuable as opinion surveys can be, it’s important to go beyond them, and to complement subjective insights with an objective understanding of what your employees know about data and AI. At Data Literacy, we craft questions with right and wrong answers about the following topics:
- Foundational Concepts: data types, storage, and formats; AI terms, concepts, and architectures
- Ethics: data privacy and security, AI fairness and bias, transparency and accountability
- Governance: roles and responsibilities, policies and procedures, compliance (e.g. HIPAA, GDPA, etc.)
- Data Quality & Preparation: data collection, accuracy and consistency; data cleaning and transformation
- Analysis & Insights: basic statistics, analytics, metrics and KPIs
- AI Applications & Limitations: natural language processing, computer vision, recommendation systems, industry-specific applications
- Communication of Insights: data documentation, visualization and storytelling
No one likes to be tested, so assuage employee concerns and remove any pressure associated with an objective test by making it anonymous. Communicate upfront that the purpose of the assessment is to establish a baseline of organizational literacy, not to score or rank individual employees. Explain that the results will be used to inform future training initiatives, so that they’ll address the needs of the organization.
3. Progress: What’s Being Implemented
Organizations are implementing Data & AI Literacy Programs to help spread the effective use of data and AI. It’s best to establish a baseline of subjective and objective performance first, and then to design the program to address issues that surface in these early assessments.
Once the program has begun, it’s imperative to track its progress. Establish a balanced scorecard that displays metrics such as:
- Training effectiveness
- Training adoption and completion
- Training performance and satisfaction
- Project tracking
- Data project completion
- AI project completion
- Data and AI utilization
- Report and dashboard utilization
- AI application utilization
In addition to these metrics, you may also want to include metrics related to data and AI quality, data community participation, and others. Only include the metrics that are the most important and relevant to your team’s situation.
4. Impact: What Value is Created
The holy grail of data and AI literacy measurement is ROI – Return On Investment. However, like the actual holy grail, it’s hard to find, and may not even exist. Your finance leaders will want to know how your program has contributed to the bottom line, both in terms of incremental revenue and cost savings.
You can only obtain a rough estimate of the financial value of a Data & AI Literacy Program. To do this, keep track of the following:
- Decisions: What decisions were affected by the program? What was their financial impact?
- Processes: What processes were improved due to utilization of data or AI? What was the financial impact of the improvement in these processes?
- Products: What data or AI products or services were launched into the marketplace, and how can you prove that they were a direct result of capabilities resulting from the program itself?
It’s difficult to tie such financial improvements to any one program. The very nature of a program is that it contributes to a broader set of internal and external forces that affect decisions, processes, and products. That’s why a perfect link is difficult to make. In spite of this, it might be reasonable to estimate the financial impact.
Conclusion
In summary, these four dimensions are like four lenses through which you can look at your organization’s data and AI literacy. Instead of relying on only one of them, it’s best to consider all four. And assessing your maturity level should not be a one-time affair. Instead, you should track these metrics on an ongoing basis, and revisit them over time to make sure you’re moving in the right direction.
