Case Study: Healthcare Data Ingestion
Written by Jared Hillam
[Healthcare Services Client] is one of the largest hospital benchmarking organizations in the United States. The Client had relationships with thousands of hospitals which used their surveys to score their patient experience. These survey results are used by the US Medicaid and Medicare programs for payment to providers.
[Healthcare Services Client] received survey data from many thousands of hospitals on a regular basis. The great number of hospitals sharing patient data for benchmarking was a daunting regular feed of data. The Client wanted to organize the onboarding of the data into a standardized process which could dynamically handle the many regular anomalies which occurred in data. While the thousands of hospitals provided data from CSV to HL7 formats, the data sets rarely onboarded with adherence to the required data standards. Often the HL7 data sets would have data populated which did not use the agreed industry norms.
Additionally they Client also had several business lines which all have specific needs for consuming data from hospitals. Each of these business lines had their own rules which they needed to be able to apply to the data, which were critical for its ingestion.
This many to many business case created a logistical challenge in onboarding customers and customer turn over.
Through an iterative design process between the client and Intricity’s architects, a flexible architecture began to emerge.
The Architecture Included:
- A REST front end (DreamFactory) to onboard the data into Informatica B2B DX and DT
- A data processing layer delivered through Informatica Developer/Data Integration Service (DIS)
- One of the critical requirements to this architecture was the capacity to allow each business line to easily assemble complex rules. To solve this problem Intricity designed a primitive rule foundation which would abstract some of the fundamental components of rulemaking in an easy to interpret naming convention. These rules would act as an abstraction layer so that the business lines could easily assemble some of their complex rules in the Informatica Developer interface. This is providing the business lines the autonomy they need to define business rules for ingesting customer data.
- A processing store which Informatica DIS could use while iterating through the file errors and changes delivered through Cassandra and PowerExchange for Cassandra
- The client leveraged Informatica’s PowerExchange for Cassandra connector to house the on-boarded customer documents and store the assembled log of errors and mismatches which the customer files may have which may need to be changed. Even if the files had catastrophic errors sprinkled throughout, the Informatica Data Integration Services was designed in such a way that it was able to still traverse the file to assemble the errors in a summary that their customers could review.
- A method to enable the 7 business lines to flexibly consume the on-boarded data, delivered through the Data Integration Hub
This Architecture Which is Being Rolled Out in 2016 Enables:
- The onboarding of customer data via a modern REST interface which ingests the data into through INFA B2B DX/DT/MFT. These capabilities were developed by Naveed Anjum who built this critical REST onboarding bridge to B2B.
- Rules that the Business Lines can share and control: The use of the Primitives Rules Architecture has dramatically increased the speed at which rules can be developed, and simplified the process for creating new rules. The first business use case for this Rules Engine included a series of complex file templates. These templates were evaluated according to hundreds of complex business rules. By creating reusable primitive rules and creating a simplified process of Mapping creation, our client was able to reduce the hundreds of individual rules to fewer than 10 primitive rules. This accelerated the development of the mappings and allowed for the rapid deployment of multiple mappings to apply large numbers of business rules with minimal development effort.
- Clearer feedback for customers: Now customers receive a summarized and very detailed error report which pinpoints exactly where their files are erroring out. This will enable our client to onboard customers in a much shorter period of time.
- Flexible and centrally managed distribution: Using DIH, our client will be able to access the data in their desired aggregation level, to their desired target system.
The Role of Informatica
Who is Intricity?
Intricity is a specialized selection of over 100 Data Management Professionals, with offices located across the USA and Headquarters in New York City. Our team of experts has implemented in a variety of Industries including, Healthcare, Insurance, Manufacturing, Financial Services, Media, Pharmaceutical, Retail, and others. Intricity is uniquely positioned as a partner to the business that deeply understands what makes the data tick. This joint knowledge and acumen has positioned Intricity to beat out its Big 4 competitors time and time again. Intricity’s area of expertise spans the entirety of the information lifecycle. This means when you’re problem involves data; Intricity will be a trusted partner. Intricity's services cover a broad range of data-to-information engineering needs:
What Makes Intricity Different?
While Intricity conducts highly intricate and complex data management projects, Intricity is first a foremost a Business User Centric consulting company. Our internal slogan is to Simplify Complexity. This means that we take complex data management challenges and not only make them understandable to the business but also make them easier to operate. Intricity does this through using tools and techniques that are familiar to business people but adapted for IT content.
Intricity authors a highly sought after Data Management Video Series targeted towards Business Stakeholders at https://www.intricity.com/videos. These videos are used in universities across the world. Here is a small set of universities leveraging Intricity’s videos as a teaching tool:
Talk With a Specialist
If you would like to talk with an Intricity Specialist about your particular scenario, don’t hesitate to reach out to us. You can write us an email: firstname.lastname@example.org
(C) 2023 by Intricity, LLC
This content is the sole property of Intricity LLC. No reproduction can be made without Intricity's explicit consent.
Intricity, LLC. 244 Fifth Avenue Suite 2026 New York, NY 10001
Phone: 212.461.1100 • Fax: 212.461.1110 • Website: www.intricity.com