Introduction

Many executives tasked with leading data projects face confusion about common data terms. What’s the difference between big data, a data lake, a data warehouse, and a data mart? When is big data “big enough”? Why does it matter?

In this article, we break down the differences between the different terms so you can sound smarter in any data-related meeting, project brief, or staffing initiative.

First- What is data, anyway?

Data is, at its core, the storage of quantitative and qualitative information. In your business, this could be information about:

•  Your internal employees
•  Your customers
• Your internal processes
• Your financial data
• Helpdesk Ticket Numbers

And so much more that is impossible to list in a single document across dimensions and measures. Your data is spread across many systems, databases, and groups within and without your organization.

Your data is also in various states of relevancy, accuracy, timeliness, and accessibility. Some of it is aligned with other data, and some of it isn’t.

If your organization is like over 95% of companies, then you face an abundance of unstructured, unclean data. That state is what we often call a “data lake”.

What is a Data Lake

A data lake is a place where your data is collected and in its natural state. That might mean that the data isn’t structured, clear, or available for use in pipelines. Data lakes store ALL DATA you have, including decades of historical data, for all time.

The storage for your data lake is usually cheap and mostly unmanaged.

When Does Data Become “Big Data”?

Another term that’s thrown around a lot is big data- and for good reason. Big data is a growing phenomenon thanks to the availability of data storage, the affordability of storage, and the number of systems and devices generating data on a daily basis.

Big data is, according to a common definition, “data that contains greater variety arriving in increasing volumes and with ever-higher velocity. “

Your data is your greatest asset and your biggest risk… yet it’s only useful if you can actually make sense of it.

This is why data marts are so helpful.

What’s a Data Mart?

A data mart is a subject-focused segment of a data warehouse that can answer specific questions, often about specific business areas or key business problems. Even better, a data mart is built to be able to answer questions on demand and have the data be trusted, as opposed to a data lake, which might be unclean and unstructured.

A data warehouse is usually a combination of many different data marts.

What is a Data Warehouse

A data warehouse is, according to Wikipedia, “…central repositories of integrated data from one or more disparate sources. They store current and historical data and are used for creating trending reports for senior management reporting such as annual and quarterly comparisons.”

Unlike a data lake, a data warehouse is a deliberate source of structured data. Even further, it’s a single repository of multiple sources… many of which are themselves data lakes.

Our dashboarding services are built off of developing robust data warehouses and pipelines so your team can easily view, analyze, and visualize your data.

How to Determine Your Next Steps

For the most part, moving from data lakes into a Data Warehouse or Data Mart involves a lot of data engineering activities like data cleanup, ETL processes, and data pipelines.

Before you decide to hire anyone for your next data project, it’s important to understand where you’re at now.

Here are some questions to ask your team:

• How siloed is our data?
• How clean is our data?
• Do we have a large enough data set for the initiative?
• Do we have a clear use case?
• Which parts of the project can our internal team handle now?
• What kind of ROI are we looking for?
• Do we know which area we might need more urgently than others?
• Do we really just need visualizations first before we try ML?

If you’d like some expert helping in figuring out where to start and what you need in terms of data, manpower, tools, and budget, we can help. Many of our projects involve data-related initiatives, especially since we now have a Python Center of Excellence in Mexico City, Mexico.

With over 20 years’ experience, Aptude’s Data team can help you figure out which capabilities you need, develop a project roadmap, and staff your project with experienced team members. Our process starts with a conversation and an NDA, so you can be sure that even if you decide not to work with us, your information is safe.

Contact us to start the conversation.