Clairvoyant maintained the metadata in a MySQL database with all parameters in separate columns to give the ability to configure ingestion for each table and each location with ease. The Sqoop import was configured to read from the database directly and generate the dag (Workflow). Clairvoyant also updated the parameters at column level to import data based on the requirement. Since there were many cases where no specific pattern was present to process and transform the tables, manual changes in metadata was required. To reduce the time of accumulation of data, two airflow worker nodes were kept running in parallel.