It's Time To Rip Off The ETL Band-Aid

By David K. Nace MD & Bill Fox J.D., M.A.

It’s time to remove the ETL (extract-transform-load) band-aid, STAT. ETL served an admirable purpose for the last two decades, but today’s volume and complexity of data are overwhelming ETL tools. These technologies that were supposed to solve data problems have now become a hindrance — and it’s time to let them go.

There are new generation technologies available designed to best manage today’s modern data challenges by quickly and easily integrating data from silos. By tearing off the band-aid (which doesn’t hurt nearly as bad as most seem to think), healthcare companies and agencies can get an up-to-date, 360-degree view of their data they can operationalize in order to make their businesses smarter and deliver better services to consumers.

Thank You, ETL

Business Intelligence (BI) and data warehouses drove the adoption of ETL in healthcare. ETL held the promise of gathering relevant data into a unified view so organizations could make better business decisions. In the very simplest of terms, ETL aggregates information by extracting data from various locations, transforming it into a consistent format, and loading it into one database-based data warehouse where the information could be analyzed. However, ETL does not embody just these three initials — it also includes the design, transport, and cleaning of data, as well as administrative and operational functions.

For some organizations and agencies, successful ETL implementations resulted in successful BI implementations, and its promise came to fruition. It was the best tool for the job, at the time.

But It’s Time To Go

ETL was certainly a huge step forward from the mainframe, but as many healthcare organizations are learning, ETL has its limits — most notably in managing the ever-increasing and ever-changing volume, velocity and types of data they need to use to stay competitive. The amount of data in healthcare is exploding (IDC projects that healthcare data will reach 2,314 Exabytes in 2020) as is the type of data such as powerpoints, videos, texts and images. In fact, images alone can overwhelm ETL tools: it’s been estimated that healthcare providers conduct 600 million imaging procedures each year.

Many healthcare companies are trying to use ETL band-aids to fix a cut to the femoral artery. As a result, organizations’ data warehouse projects spend about 80 percent of time on ETL, and ETL dominates the same amount of time for BI projects. This percentage is simply unacceptable in today’s competitive environment and the value isn’t always realized.

Pfizer’s ETL strategy resulted in repetitious and error-prone requests, and found that “integration requests could take one year to achieve and cost $1 million, and sometimes be deemed unnecessary by the time they were completed.”

And Replace With New Generation Databases

NoSQL databases are a new generation technology: they more easily ingest structured and unstructured data, which dramatically reduces ETL time and money, if not eliminating it altogether.

The catch is that not all NoSQL databases are created equal.

Healthcare agencies and organizations need to adopt new generation technologies like Enterprise NoSQL databases as they provide the unique combination of NoSQL and enterprise features. NoSQL offers flexibility for managing various types of data, agility for making quick changes, the ability to operationalize and transact business, and affordable scalability to meet growing users and data. Equally important are enterprise features like government-grade security, high availability, support for ACID (Atomicity, Consistency, Isolation, Durability) transactions and disaster recovery.

New generation databases are designed for today’s data challenges, integrating data from silos in a much more affordable and rapid manner than ETL. This creates an agile infrastructure, which allows organizations to spend more time on creating innovative and productive new apps and services, versus spending an exorbitant amount of time and money trying to simply manage their data.

ETL had its moment, a good long one, but now is time to move on in order to remain competitive. New generation databases can help you rip off that ETL band-aid to heal your IT infrastructure and bring vibrant, good health to your business.