Last month, I delivered a Keynote presentation at the DataCampus meeting in Hamburg. The topic was trends and challenges in data management. While preparing the presentation, I discovered that we not only have trends and challenges in data management but also experience challenges in defining trends.
In Part 1 of this article, I will:
- Discuss challenges associated with defining data management trends
- Demonstrate key challenges in the overall data management capability and several core sub-capabilities: business architecture and data governance
Before diving into the subject, I want to share my understanding of the terms “driver,” “trend,” and “challenge” that I will use in this article.
A data management trend is a general direction in which data management evolves.
A driver is a factor of the external business environment that determines trends.
A challenge is “a new or difficult task that tests somebody’s ability and skill.”
Trends and challenges can impact each other. Challenges can lead to trends. For example, the challenges of high IT costs for onsite IT tool deployments led to the development of cloud technologies.
Trends can pose challenges. For example, digitalization and online communication have brought challenges related to cybersecurity, privacy issues, and digital well-being.
Now, let’s discuss the challenges associated with defining data management trends and trends themselves.
Challenge 1. Different sources have pretty different viewpoints on trends in data management.
I analyzed 18 sources that discussed data management trends in 2022 and 2023. Gartner and Dataversity are examples of these sources. The total number of trends they mentioned reached 114! This number of trends was a starting point to dive deeper and try to make sense of these trends.
Challenge 2. Trends are mixed with business drivers that lead to these trends.
When I went through these trends, I discovered at least four factors or business drivers that cause trends in data management. Some authorities consider these factors as trends. I have different opinions and recognize them as the reasons that cause trends in data management. I will briefly discuss these four drivers:
Economic uncertainty
This factor forces companies to focus on generating more monetary value from data and decreasing IT-related costs. These goals will motivate companies to use cost-effective technology.
Regulations pressure
For all companies worldwide, personal data protection regulation is one the most important regulations that impact data management solutions. Analysts at Gartner have predicted that 65% of the world’s population in 2023 will be covered by laws similar to GDPR. For financial institutions, complying with risk-related regulations is the biggest driver for implementing data management.
Cybercrime
Cybercrime has grown over the last few years. Ransomware is one way for cybercriminals to profit off a company’s data at its expense. To mitigate the risk of cybercrime, companies must invest in cybersecurity.
Development of artificial intelligence (AI)
The latest factor that significantly impacts data management is the development of artificial intelligence. Multiple data management (DM) functionalities provided by DM tools can be enriched using AI. I mean, for example, data mapping and cataloging, metadata management, anomaly detection, data analytics, and data quality.
Challenge 3. An unaligned definition of data management within a data management community leads to challenges in defining the trends.
After carefully analyzing all these 114 trends, I realized that the key reason for this impressive number of trends is different viewpoints on data management.
I shared the results of the analysis in multiple articles and webinars. This article will present one of the data management capability models I use in my practice. This model, demonstrated in Figure 1, describes data management in the following way.
Figure 1: The model of a “Data Management Capability”
(the “Orange” data management framework).
The core value proposition of data management is enabling a data lifecycle and delivering information to all relevant stakeholders.
So, data lifecycle management is the core capability of data management, focusing on value delivery.
Two capabilities, data governance, and business architecture, provide direction for data management development. These capabilities belong to the strategic level of the data management capability map.
Then, we have multiple supporting capabilities. They enable the data lifecycle.
And, of course, to bring these capabilities into operations, we need a set of policies, processes, roles, IT tools, and other resources to deliver the intended artifacts of each capability. Data governance supports their development.
Let’s consider examples of trends that characterize each of these capabilities.
Trend 1. The maturity level of data management worldwide has grown over the last four years.
Figure 2 illustrates the overall data management maturity trends. The results are based on the anonymous DM maturity scan available at the Data Crossroads site, shared in the “Data Management Maturity Assessment Review 2022.”
Figure 2: The trends in data management maturity.
More than 800 companies worldwide have performed this scan for the last four years. I use five levels to demonstrate maturity. Level 1 is the lowest, and Level 5 is the highest. The level of maturity is an aggregated figure based on the answers to 20 questions.
So, in general, the level of maturity has slightly improved over the last four years. You can see that no company had the lowest level of maturity. The number of companies at levels 2 and 3 has decreased. At the same time, the number of companies with maturity levels at the 4th and 5th levels has slightly increased over the last four years.
Business architecture
Business architecture is a company’s ability to provide a complete unified overview of its business model expressed in business capabilities, value chains, data and information domains, and organizational structure and their relations.
Trend 2. Use of business architecture in strategic planning.
Gartner predicts that “by 2027, 50% of extra-large organizations will use business architecture to advance strategic planning[…].”
Data governance
Defining a data governance capability is one of the most significant challenges in the data management community. Various authorities have different views on data governance, as shown in Figure 3.
Figure 3: Different viewpoints on data governance.
DAMA-DMBOK2 considers data governance a set of regulations, processes, and roles defining the day data management operates. Others include multiple other capabilities, initially considered by DAMA-DMBOK2 as data management capabilities.
You can read a series of articles about it or watch a webinar on this subject.
Data governance has the following trends.
Trend 3. Adoption of data governance in data management.
It is different to interpret what people mean by that due to challenges with the definition of data governance.
In the interpretation of Dataversity, this trend means that many companies implement a data management organization or function as the result of implementing the set of policies, regulations, and roles.
Trend 4. Data democratization
I think this complex, ambitious term has a straightforward meaning: Non-technical data users get access to data and insights from it.
Trend 5. New roles
The next one is the creation of new roles. In some of my publications, I’ve already shared the results of my investigations about the number of data management roles mentioned in DAMA-DMBOK2. How many roles do you think DAMA-DMBOK2 describes? It is more than 100. So, the question is: “Do we need more roles?” However, new developments in data architecture, like data mesh, require new functions like “data product owner” and so on.
Trend 6. Partnering with other business functions
Collaboration between data management and financial planning and analysis teams is an example. I wrote multiple articles on this topic. Let me provide a simple example. We, data management professionals, deal with data lineage. It demonstrates the relations between sourcing data elements and the reports and dashboard elements. FP&A people deal with driver-based planning.
The driver-based model is based on the relations between input business drivers and financial outcomes like revenue. These two concepts are closely related.
Trend 7. Data Management and Data Governance IT tools provide similar functionalities.
Figure 4 demonstrates the capabilities provided by DM and DG IT tools. You can hardly see the differences between these types of tools. DM and DG tools deliver functionalities related to various data-, metadata-, and data lifecycle management.
Figure 4: Functionalities provided by data management and governance IT tools.
You can find a detailed analysis of the functionalities provided by various data management-related tools in the series of articles “Choosing Data Management IT Tools.”
Trend 8. The data governance maturity demonstrates conflicting trends.
Figure 5 demonstrates the results based on the Data Management Maturity Review 2022. The results are controversial. You can see that the results in 2022 worsened compared to previous years.
Figure 5: Trends in data governance development worldwide.
Fewer companies reached levels 3,4 and 5 compared to the previous years.
This is the end of the Part 1 of the article. In Part 2, I will demonstrate the trends in developing the following capabilities: data management lifecycle, enterprise architecture, metadata management, data analytics, and IT infrastructure.
