Data Governance, Videos, Data Warehousing, Business Intelligence, Snowflake, Blog

Data Architecture Lessons from the 1700’s?

Jared Hillam

Jared Hillam

February 7, 2018


In the late-1700s there was an innovation that would change the world forever. This innovation came from the French Engineer & Lieutenant General Jean-Baptiste Gribeauval. He was charged with reforming the french military’s artillery. What he came up with was a standardized set of cannons and cannonballs. See up until that point cannons were not standardized, for each cannon, you made cannonballs to fit that specific cannon's barrel. Using Jean-Baptists specifications, the French could freely produce cannon balls by simply staying within the tolerance of Jean-Baptiste’s design. The concept of interchangeability was introduced to the world, and it became an integral part of the industrial revolution.

I share that story because it's a great illustration of something called coupling. There are two types of coupling, tight coupling or loose coupling. For example, the cannonballs that are custom made for a custom-made cannon are tightly coupled to that cannon. Whereas Jean-Baptiste’s loosely coupled cannon design will allow you to exchange the cannonball  

inventory without any conflict. But the cannonball example is a fairly crude simplification. Karl Weick, the one that came up with the coupling concept, wrote about more intangible systems like organizations and information systems. He discovered that tightly coupled systems run very smoothly when all the steps in that system are present and accounted for, but these types of systems are fragile to any change. Like bumping one domino and having the rest of them fall.

Understanding coupling comes in really handy when you’re setting up a system to turn data into information. Let me give you a few common examples of tightly coupled scenarios that we often see in this space.

First example: Let's say that you have your “data guy” that gives you all the information you need. Well, this means that you are tightly coupled to that data guy. The day that analyst finds a new job is the day that you stop getting information. The more that “data guy” uniquely supplies and understands your organization's data, the more painful his or her exit is.

Second example: Your organization decides to custom build all the analytics into the CRM or ERP application. This means that you are tightly coupling yourself to your software vendor because all your information assets are custom coded therein. So when you go to negotiate your fees with that vendor you no longer have the upper hand. Or worse, if you have to replace that vendor you now are forced to completely rebuild how your turn data into information.

Third Example: Your organization needs to change the way it calculates its costs. This will require you to change some logic in your analytics. However, since you already have thousands of reports and analytics we need to be sure that change is accounted for across all of them. There’s only one problem, the solution you choose for data visualization, nests the SQL query logic in the data visualization itself. So now you have to open each one individually, edit the SQL, and repeat… a thousand times. Because the visualization and the query logic are tightly coupled together.

Fourth Example: Each night you load data into a data warehouse to supply analytics to your business. However, on one particular night, the load takes too long, so now it's running during the day, and users can’t even query the data warehouse because its compute power is tied up with loading data. So in this case the compute and the storage are tightly coupled together so you just have to wait until the data finishes loading until you can get access to the data again.

Fifth Example: As a Marketing Professional you have got to get your hands on meaningful analytics to help you make some critical daily decisions. You either don’t have an IT department or you can’t rely on your IT department to be responsive enough to your analytic needs. Your organization decides to purchase an all-in-one solution that delivers a data warehouse and visualization offering in a single SKU. Now all your data logic and visualization are tightly coupled together and they are coupled to a single vendor. The moment you find another powerful analytics tool you’re stuck because your Data Warehouse is coupled to a single BI layer provided by that one vendor.

For each one of these examples, I’ve written a loose coupling solution in a whitepaper which you can pick up by clicking here.  Hopefully, these scenarios can help you get a glimpse into how important it is to understand the big picture when deploying the process of turning data into information. To go through your scenario one on one, feel free to reach out to us to Talk with a Specialist

Related Post

Snowflake Data Breach... Now What?

Snowflake's data breach affected 165 customers through stolen credentials. Discover the security measures that could have prevented it and how to protect your data in the future.

Read More

Using AI for Code & Metadata Conversions on Data Systems

LLM-based code conversions can have challenges and successes. Explore real-world insights and best practices for navigating these projects.

Read More

Medallion Architecture, From the Late 90's?

Much like AI is all-the-buzz today, the early 2000’s buzz tech was the Data Warehouse. Like AI today, everybody said they were doing it, but few were actually aware of what they were doing...

Learn More