Data Paradigm Shift #3: Doing Data Right Begins at the Data Source, Not the Data Lake

• clearly defined data product roadmaps,
• data ownership in and by the business,
• data priorities, budgets, and approvals in and by the business,
• data that is searchable, findable, understandable, usable, and reliable, and
• execution at a lower cost and shorter time-to-value to deliver on new data requests from the business

In contrast, I have heard a lot of statements over the years that immediately raise red flags in my mind as indicators the organization is likely not doing the right things with data, including:

• We have XX terabytes of data in our data lake
• We manage all of our customer data in XX (insert your favorite ERP or CRM vendor here)
• We are automatically cataloging all of our data when it comes into the data lake
• Our data scientists spend 90% of their time sourcing and cleaning up data

Each statement has multiple red flags for me but one common theme across them all is the underlying assumption that "data" -- data management, data modeling, data cataloging, data governance, data quality, etc. -- all begin when the data hits the data lake for analytics use cases. Don't rush pass this and miss the profound missed opportunity. These organizations are making huge investments in trying to manage their data in one location only for the single purpose of analytics. And yet, how many different systems across the organization capture, manage, and analyze customer data or sales data -- customer portal(s), ERP(s), CRM(s), CDP(s), eCommerce platforms, quoting and pricing systems, and on and on -- multiplied across several divisions, acquired companies, and external business partners.

To do data right, we must shift our paradigm to a mindset where "data" and all of its capabilities must begin at the source system and must continue through to the consumption of that data by a person (e.g. analytics) or by another system (e.g. transaction processing). Do you see the difference of managing data everywhere it exists for all of its uses? Do you begin to see and understand the implications of this paradigm shift on org/op models, funding, data sourcing, technology and platforms, and every other aspect of doing the right things with data? This is a profound paradigm shift, one that most companies have not yet recognized, and many that have tried to make the shift have been marginally successful at best, and completely failed at worst.

If your measures of success with "data" are the amount of data in the data lake and the number of systems feeding the data lake, then your current approach is probably pretty good. However, you might need a different approach if your (or at least your business sponsor's) measures of success include things like:

• Increasing repeat sales through frictionless customer experiences by sharing orders, shipments, and inventory data with our most valuable customers in real-time
• Increasing customer satisfaction by providing customer support representatives using the CRM system with an accurate and complete picture of the customer's account, orders, shipments, and past complaints
• Decreasing shipment delays, back orders, and production downtime by integrating with suppliers and logistics partners for visibility into potential supply chain challenges before they become your customer's problem
• Grow revenues to more than pre-Covid levels with half the pre-Covid staff by increasing marketing effectiveness

One of the biggest and most often overlooked implications of this paradigm shift is how data is sourced from our transactional systems. In the old paradigm, simply replicating tables from transactional systems is a quick and easy way to land data in the data lake, however, I would argue this is actually an indicator of failing to do the right things with data. With measures of success focused on increasing sales and customer satisfaction and reducing downtime and delays, a new approach is needed. Enter the data integration architecture.

From a technology and an execution perspective, the data integration architecture might be the single biggest indicator of an organization's commitment to do the right things with data. I say this for several reasons:

1. This approach requires an upfront investment to lay the foundation for building business assets without immediate value delivered back to the business.
2. This approach demonstrates a willingness to move to a product-centric mindset, even if it means a delay in IT projects to do the right things with data.
3. This approach is a move from application-centric thinking to data domain-centric thinking, requiring data product ownership, data governance, and data stewardship, all of which require engagement from and investments by the business.
4. This approach shows a long-term, strategic approach by IT that allows for transactional systems to be loosely coupled so that entire applications can be swapped out in the future without multi-year transformation projects that also rip out all the previous data work.
5. This approach is intentional, purposeful, and strategic by sourcing data one time from transactional systems and truly treating data as an asset beginning at the point where it is created.

Now, you might be saying, this sounds like its own multi-year, large-budget investment, but it doesn't need to be. With a start-small, iterative approach, no/low-code technologies, and good data leadership, this can begin quickly, show success early, and grow over time.

To understand where your biggest opportunities are and how best to get started, I would encourage you to take The iBusiness Group's Data Readiness Assessment. In a very short period of time, we can assess your current-state compared to best practices and make specific recommendations across the Leadership, Business, Data, and Technology Disciplines that are specific to where you are today and where you need to get to. Ask us today. We would love to help you do the right things with data the first time, or at least the next time .

Question: Where does "data" begin in your organization -- at the source system, at the data lake, somewhere else, or nowhere? Leave a comment.