Setting up a Data Management Function Follows the Logic of Data Lineage Documentation
Setting up a Data Management Function Follows the Logic of Data Lineage Documentation
Are you interested to know more about setting up a data management function?
The name of this article is the summary of my 10-year hands-on experience in data management. Implementation of data management frameworks and data lineage are the core of my professional experience. These topics might seem quite different, but my practical experience made me conclude that they are very much related to each other. In this article, I explain how and share my knowledge and techniques. We take a deep look at setting up a data management function.
WHAT are data management and data lineage?
This question might look odd as we, data management professionals, should know the answer. One of the key data management tasks is ensuring aligned and unambiguous definitions. Yet, it is not the case for most data management terms.
Each of my workshops on data management-related topics starts with me asking the participants to formulate in one/two sentences their understanding of data management. So far, I have never heard the same answer from two people. When you try to analyze the definitions of data management from published sources, you face the same challenge. This shows that data management has a variety of meanings that strongly depend on the context.
An IT-related set of data management sub-capabilities enables the functioning of the chain.
In my opinion, data lineage is one of the most abstract and misaligned concepts in data management. As one of my colleagues once said: ‘Everybody needs data lineage, nut no one can explain their understanding of it.’ Usually, metadata data lineage documents the flows of data across the organization. The concept of data lineage intercepts with five other concepts: data chain, data value chain, integration architecture, data flow, and information value chain. Some of these terms are even considered to be synonyms of ‘data lineage.’ There are different viewpoints on the constituent components of data lineage. Based on the analysis of the concept of data lineage and the requirements of several legislative documents, I came up with the following set of components that are required for proper documentation of data lineage:
three levels of data models (conceptual, logical, and physical) with corresponding business rules/ETLs are linked vertically.
business processes and roles
data quality checks and controls catalogs.
(For a more detailed explanation, please check out this article.)
The scheme of the metadata lineage components can be seen in Figure 2. More about data lineage can be found in the set of articles Data Lineage 101, 102, 103, 104, and 105.
Figure 2. The scheme of the metadata lineage components
Now let us discuss how to implement the data management function and demonstrate its relation to data lineage.
HOW are the data management function setup and the documentation of data lineage linked?
The answer to this question lies in similarities between:
the deliverables of data management sub-capabilities and key components of data lineage
the logical steps of implementation of data management and documentation of data lineage.
Figure 3. The data management star by Data Crossroads.
Step 1. Defining needs and requirements.
Data management has different business stakeholders with specific needs concerning data and information. In Step 1, a company will focus on specifying a feasible scope of data management initiative. The deliverables list includes business drivers, stakeholders, and their most urgent information needs. Information is delivered in the form of reports and dashboards. The report catalog is one of the data lineage components. The scope of the data management initiative will limit the scope of data lineage documentation.
The corresponding data management tasks and responsibilities should be defined when the scope is clear.
Step 2. Dividing tasks and responsibilities
The set of tasks and responsibilities belong to the data management framework, which is a set of rules and roles. Rules include but are not limited to data management strategy, policies, standards, processes, procedures, and plans. Roles should be linked to data management processes, tasks, and deliverables.
Data lineage is one of the deliverables of data management. Therefore, a company needs to specify and document its understanding of data lineage, constituting components, and how to document it (descriptive or automated). Accountabilities regarding data lineage documentation should be assigned to the relevant data management-related roles.
Step 3. Building the data management framework.
The implementation of data management will be done in several steps.
Step 3.1. Specify data requirements
To meet the information requirements specified in Step 1, corresponding data should be found, delivered, and processed. Very often, the relationship between raw data and information is not (entirely) known. Data lineage is a means to fill in this gap. Usually, data lineage documentation starts with the specification of existing business processes and mapping them to the data sets.
Step 3.2. Document business processes
Business process documentation is not considered to be a part of any of the data management sub-capabilities. Still, this is a required component of data lineage. Most companies begin their data lineage documentation with the analysis of business processes. The performance of business processes is closely related to the systems and applications used in these processes.
Step 3.3. Document system and application landscape
Data transformation usually takes place in systems and applications. Documentation of applications and data flows a part of information systems architecture. At the same time, these flows are mandatory components of data lineage.
Step 3.4. Develop conceptual, logical, and physical data models and link them.
Data flows can be documented on different levels of data models: conceptual, logical, and physical. A company might prove data flow/lineage on any of these levels. A company can document data lineage on the combination of these levels. These models and the links between them are data modeling and architecture deliverables. These models and links are, at the same time, mandatory components of data lineage. The links between different data model levels are called ‘vertical data lineage’ or ‘linkage.’
Step 3.5. Identify critical data elements
The definition of critical data elements is a state-of-the-art task. You can read more about practical techniques for the specification of critical data elements in this article. The key reason to apply the concept of critical data is the prioritization of data management initiatives, including building data quality checks and controls. The set of critical data elements is the deliverable of the data modeling sub-capability. The mandatory pre-requisite to specify critical data elements is knowing data lineage.
Specification of data quality requirements and corresponding checks and controls belongs to the deliverables of data quality sub-capability. Data checks and controls are considered being a component of data lineage.
Step 3.6. Assemble data lineage
Data lineage can be assembled only when all the steps mentioned above are done. And at this point, a company de facto is ready to implement a data management function.
Step 4. Intermediate assessment and gap analysis
This step is required to compare the desired results specified in Step 1 with the achieved ones. This is also a point where the maturity assessment of data management can be performed.
Step 5. Planning further actions
As soon as the company has achieved the desired results, it might want to extend the scope of its data management initiative, including the scope of data lineage to be documented.
I hope you understand how the initial statement ‘The setup of data management function follows the logic of the documentation of data lineage’ came to be by now.
Irina is a data management practitioner with more than 10 years of experience. The key areas of her professional expertise are the implementation of data management frameworks and data lineage.
Throughout the years, she has worked for global institutions as well as large- and medium-sized organizations in different sectors, including but not limited to financial institutions, professional services, and IT companies.
Strictly Necessary Cookies
Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.
If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.
3rd Party Cookies
This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.
Keeping this cookie enabled helps us to improve our website.
Please enable Strictly Necessary Cookies first so that we can save your preferences!