Data Warehousing, Business Intelligence, Blog, Partners

Accelerated ETL using BI Markup Language (Biml)



April 25, 2016


The ETL Drag

A common complaint in the construction of a Business Intelligence (BI) solution is the length of time, and the associated cost with building the Extract, Transform, Load (ETL) routines to populate the associated data repositories (i.e., DataMart’s, Data Warehouse). Some estimates place this ETL development at up to 80% of the overall BI development effort.

There are a few reasons for this ETL conundrum that include:

  1. Consolidation of multiple source (operational) systems
  2. Efficiencies with data pulls (full vs. incremental)
  3. Proper interpretation of each systems’ business rules
  4. Maintaining historical perspectives

None of which is addressed in this article.

But perhaps an equally challenging problem with the construction of a Data Warehouse occurs with the process of coding the ETL logic, which is really an issue inherent to the ETL developer’s toolkit. In short, most ETL tools are centric to the graphical user interface (think very repetitive ‘drag and drop’, ’point and click’). These interfaces create a nice visual representation of data mapping, but at the same time this method generally abandons decades of proven software development techniques, such as ‘code reuse’.

Biml's Birth

Microsoft recognized this issue several years ago, and sought to address with project Vulcan (the remnant of this now defunct project can be viewed here: Fortunately, a key member of project Vulcan (Scott Currie) has formed a company called Varigence to continue this effort, which has given the BI community an important XML dialect called Biml, along with related products and technologies (Mist, BimlScript, BimlOnline, BimlExpress) which form a healthy part of the growing ‘Biml ecosystem’.

It should be mentioned that while most of the work leveraging Biml is centric to the Microsoft product line, nothing in the Biml language precludes developing emitters for other platforms. The beauty of Biml is it’s a highly readable XML declaration of Business Intelligence assets. As a side note, while database and cube definitions are also a part of Biml, what appears to be of most interest in the BI community is its application for ETL processes.

To read the remainder of Jim's Biml article which includes code snippets click here:



Related Post

Celebrating the Launch of Truelty

Truelty, Identity Resolution powered by Snowflake, makes its debut at Snowflake Summit 2022. Intricity is proud to introduce this tool to the public.

Read More

Intricity's Jared Hillam as Guest Speaker on Hedgeye Technology Podcast

Jared Hillam discussed the future of Best of Breed software versus IaaS product and more on the Hedgeye Technology podcast on June 1st, 2022.

Watch Now

Era of Enterprise Data Apps

Imagine being able to connect apps for the very function you need and have them run within your existing data landscape. That is an Enterprise Data App.

Learn More