Mass Kafka to Snowflake Case Study


The Client is a global real-time information discovery company and also a source of breaking news alerts. These alerts are used by a wide variety of companies and government entities interested in being alerted by breaking news. The Client has built their own complex event processing capabilities that monitor the latest event data, bubbling up critical alerts from a sea of constantly streaming events.


The Client quickly pushed the limits of traditional databases with its constant flow of over 1200 Kafka streams. The speed of storage and the ability to orchestrate the constant ingestion of data was always a wall that was difficult to traverse. The Client spent a significant amount of time evaluating the right cloud data warehousing solutions which could deliver on the promise of concurrently storing and querying data. Snowflake was the clear leader in the end, but building the orchestration framework and designing Snowflake in a way that could support their use case was something the Client team needed help to accomplish.

Navigating Constraints

    1. The Client lives in the world of “now” so their streams of data had to constantly be loading into their data warehouse with the lowest possible compute overhead to ensure timely arrival.
    2. The Client had 1200 Kafka streams making it difficult to ingest to a consolidated data warehouse.
    3. The Client wanted to completely own the solution with INTRICITY providing guidance on a regular basis.

Click the link for the full article: