Clairvoyant

CategoryHealthcare

About The Company

BluePearl Veterinary Partners is a national provider of specialty and emergency veterinary care. Their network of pet hospitals offers advanced specialty services and compassionate care and treat the most critical emergencies. BluePearl recognizes that pets are part of the family. They honor the human-animal bond by providing the very best in advanced medical care for pets – including techniques that were once reserved exclusively for humans.

Goals

Reduced time to accumulate data from all the tables, from 20 different location and 3 different types of databases into hadoop.

Made querying faster for all the data in the data lake.

Configuring data with ease for each location and table.

Application readiness to handle larger volumes of Data.

Business Challenges

Creating a robust data lake with an ingestion pipeline combining data from 20 different data centers and more than 1000 tables imported daily.

Merging the tables from different types of databases and different schema into the corresponding final table in data lake.

Processing of each kind of table was being done in a different manner.

The data received was from different geographical locations, this was required to be maintained accordingly.

Key Technologies

The Solution

Clairvoyant maintained the metadata in a MySQL database with all parameters in separate columns to give the ability to configure ingestion for each table and each location with ease. The Sqoop import was configured to read from the database directly and generate the dag (Workflow). Clairvoyant also updated the parameters at column level to import data based on the requirement. Since there were many cases where no specific pattern was present to process and transform the tables, manual changes in metadata was required. To reduce the time of accumulation of data, two airflow worker nodes were kept running in parallel.