New Video: Modern center of excellence

Hi, I’m Jared Hillam.

In the early lifecycle of a breakthrough there seem to be boundless possibilities. Those boundless opportunities are dominated by early adopters and open thinkers. But like with any useful invention, there’s a primary use case that really drives the adoption. But as that primary use case becomes a standard, the focus becomes less about pushing the boundaries, and more about making the primary use case work at scale. And we’ve seen this over the course of decades in essentially every industry and discipline, and the data cloud is no different. The primary use case for the data cloud is to take data and compute it into actionable information. Now whether that information is used by an AI or a Human that primary use case is essentially the same.

So within the primary use case of cloud data systems, we’re starting to see some architectural stability. Yes there is still some shuffling around, but we’re really at the point where these represent the typical laggards. So what this means is that organizations need to start stabilizing and cost managing their primary use cases. Just like car manufacturers have to transition from their prototypes to their production line, companies need to do the same with their data clouds. 
Enter the Center of Excellence. Now the concept of a center of excellence is as old as scalability itself. How do we take humans, and define a process for them that maximizes the technology which ultimately becomes a culture.

Now some might look at that idea and think it sounds boomer-esque. I mean, didn’t we leave the old organized world behind because we had unlimited compute resources in the cloud? To some extent I sympathize with that. But at the same time, customers that haven’t grabbed the bull by the horns, are getting mired in compute costs, and it frankly wouldn’t matter what cloud compute layer they adopted, the cost structure would still be enormous. So yeah, if you’ve really built out an empire on these systems, it’s time to maximize their value.

So let’s talk about some of the modern pillars of today’s center of excellence, and I’m gonna point out where this lands amongst all the other pillars.

#1. Access Control Automation - This is possibly one of the most important things you can do to enforce the outcomes you want. After all, these cloud data systems are unlimited. So if you can dynamically hook the data access to the systems that flag the hiring, firing, promoting and demoting, you’ll be able to control a huge amount of behavioral problems and wasted compute.

#2. Data Graduation - By having different graduations of data from raw to highly aggregated, you can couple this with access control to ensure the compute resources aren’t wasted for the wrong audiences and queries.

#3. Compute Optimization - Now many of these cloud data stores claim to magically optimize compute. But we’ve seen performance increases of over 1000% by dramatically reworking the query strategy. So organizations need to literally have an internal practice of sniping the most persistently egregious queries and reworking them to their greatest potential. The annualized returns on doing this more than pay for the cost of the team.

#4. AI Auditing - This refers to auditing in both directions. ChatGPT for example supports both Snowflake and Databricks code outputs. So both the humans need to audit the AI’s code and the AI should be auditing the human code. This brings opportunities to expand into other optimization strategies, and avoid costly mistakes. I want to emphasize that the human really does need to know what they’re doing here. GPT sounds really smart to a lay reader.

#5. Organizational Structure - Who owns the data in an organization can be a really touchy subject. Is this an IT thing or does the data belong to the business? The answer to that question can really impact how the solution is received by its users. 

#6. Training & Education - In a large scale cloud data environment you’re going to have a many to many relationship between data objects and users. If we spend all our time just managing data objects and we don’t spend time training users, organization’s will find themselves chasing compute overruns simply because people don’t know what they’re doing. In the past this was managed simply by the 4 walls of the MPP database’s physical process limits, but in an unlimited compute world, this can easily happen. Of course, you can create programmatic guardrails, but nonetheless we don’t want team members gobbling up compute just because they don’t know any better.
So these are just 6 of the many modern Center of Excellence pillars. I’ve written a short whitepaper on just these 6 and if you want to dive deeper into building a Center of Excellence in your organization I recommend you reach out to Intricity to talk with a specialist.