Metadata and tagging are key to discovering the right data in a data lake

Metadata is what we talk about when capturing data about data, such as the source the data originated from, how data is structured and formatted and when it was captured.

Metadata capture and tagging are closely related and support data exploration and governance within the business.

Tagging drives data discovery in a business. A data lake could potentially have petabytes of data, much of which might not be funneling to production systems yet, because its value has not been identified. Tagging is both an automatic and manual business process and is unique to a business.

Tagging is used to understand the inventory of a data lake through the notion of tags, similar to those attached to social media and blog posts. Consider a data lake in a healthcare business housing data from multiple electronic medical record systems. You might have a tagging strategy of EMR > System Name > Patient Encounters (the dataset). This allows individuals throughout the business to search for EMR system data by tag.

When devising a tagging strategy, the following should be considered:

• When does the data get tagged?
• Who will own and be responsible for tagging the data sets?
• What will the naming convention be so that data sets can be easily discoverable by other business units?

Meta data maturity

Traditional metadata: This includes information about flat files, tables, connection strings, file store location, file name, data types and length, etc. You can leverage this metadata in patterned ETL development and management.

Modern metadata: Utilizing file formats that allow metadata to be stored within a data file. Storing metadata within the file itself assists in solving problems such as evolving schema changes.

Advanced metadata: Automated processing and pre-defined aggregations upon loading into the data lake. If you have worked with Microsoft’s Power BI application, an example would be the Quick Insights feature that shows simple relationships in the data and quick aggregations of the data.