The Client is a global real-time information discovery company and also a source of breaking news alerts. These alerts are used by a wide variety of companies and government entities interested in being alerted by breaking news. The Client has built their own complex event processing capabilities that monitor the latest event data, bubbling up critical alerts from a sea of constantly streaming events.
Intricity is a team of specialized Data Management, Data Warehousing, and Business Intelligence experts. The team members at Intricity have been handpicked over the course of 20 years and represent the top talent globally in data-oriented disciplines.
Challenge and Wins
The Client quickly pushed the limits of traditional databases with its constant flow of over 1200 Kafka streams. The speed of storage and the ability to orchestrate the constant ingestion of data was always a wall that was difficult to traverse. The Client spent a significant amount of time evaluating the right cloud data warehousing solutions which could deliver on the promise of concurrently storing and querying data. Snowflake was the clear leader in the end, but building the orchestration framework and designing Snowflake in a way that could support their use case was something the Client team needed help to accomplish.
The Client lives in the world of “now” so their streams of data had to constantly be loading into their data warehouse with the lowest possible compute overhead to ensure timely arrival.
The Client had 1200 Kafka streams making it difficult to ingest to a consolidated data warehouse.
The Client wanted to completely own the solution with Intricity providing guidance on a regular basis.
Win 1: Ingestion Automation Using Snowpipes, Streams & Tasks and Airflow
Intricity assisted the Client with the ability to bring the data together programmatically through Airflow for complex patterns and Snowpipes with Streams & Tasks for transformation events. This made Snowflake the primary computing layer for all the transformation logic from the 1200 Kafka streams. On the front end, Looker was the BI layer of choice.
Win 2: Near Real-Time Processing of the Data Warehouse
With the deployment of Snowpipes with Streams and Tasks, the Client is able to deploy a near real-time architecture. This was accomplished by splitting the streams between real-time and scheduled streams which deliver aggregated details. Along with this real-time capability, the Client’s environment can be scaled to onboard an unlimited ingest and egress demand.
Win 3: RBAC Security for Snowflake
Intricity’s RBAC/ABAC Rollup Best Practices delivered a guided framework which the Client used to appropriately deliver access controls in their complex environment. This included diagrams of how the access controls interact with the Snowflake environment.
Win 4: CoDev Model
Intricity leveraged its CoDev Model to help their team become proficient in Snowflake. This enabled them to meet the requirement of internal ownership of their product while instilling best practices based on over 100 Snowflake deployments under Intricity’s belt.
Call to Action
To schedule a time to discuss your landscape with an Intricity specialist, go to http://www.intricity.com/contacts and register to talk with a specialist or call the office number near you.
WHO IS INTRICITY?
Intricity is a specialized selection of over 100 Data Management Professionals, with offices located across the USA and Headquarters in New York City. Our team of experts has implemented in a variety of Industries including, Healthcare, Insurance, Manufacturing, Financial Services, Media, Pharmaceutical, Retail, and others. Intricity is uniquely positioned as a partner to the business that deeply understands what makes the data tick. This joint knowledge and acumen has positioned Intricity to beat out its Big 4 competitors time and time again. Intricity’s area of expertise spans the entirety of the information lifecycle. This means when you’re problem involves data; Intricity will be a trusted partner. Intricity's services cover a broad range of data-to-information engineering needs:
WHAT MAKES INTRICITY DIFFERENT?
While Intricity conducts highly intricate and complex data management projects, Intricity is first a foremost a Business User Centric consulting company. Our internal slogan is to Simplify Complexity. This means that we take complex data management challenges and not only make them understandable to the business but also make them easier to operate. Intricity does this through using tools and techniques that are familiar to business people but adapted for IT content.
Intricity authors a highly sought after Data Management Video Series targeted towards Business Stakeholders at https://www.intricity.com/videos. These videos are used in universities across the world. Here is a small set of universities leveraging Intricity’s videos as a teaching tool:
TALK WITH A SPECIALIST
If you would like to talk with an Intricity Specialist about your particular scenario, don’t hesitate to reach out to us. You can write us an email:firstname.lastname@example.org
(C) 2020 by Intricity, LLC
This content is the sole property of Intricity LLC. No reproduction can be made without Intricity's explicit consent.
Intricity, LLC. 244 Fifth Avenue Suite 2026 New York, NY 10001 Phone: 212.461.1100 • Fax: 212.461.1110 • Website:www.intricity.com