HPE Shadowbase Extract, Transform, and Load (ETL) Toolkit

1. Choose a supported format

HPE Shadowbase Application Integration provides a suite of adapters to easily integrate the source data with the target environment.  Available adapters include:

• Integrating with KAFKA
• Integrating with IBM MQ environments
• JSON (JavaScript Object Notation) output format
• Flat-files (CSV, fixed position, tab-delimited)

2. Feed high-volume data quantities into a data warehouse, and feed row-based data into columnar databases

Using the Toolkit, source database file and table data can be extracted and stored in a flat-file.

Optionally, change data can also be periodically extracted from the audit trail.

The extracted data is processed into a flat file format for incremental loading into the target environment using a bulk or micro-batch update approach (provided by the target environment) to keep the target synchronized with the source.

The Toolkit can also be used to extend the SOLV loading capabilities to allow for reading and injecting events from flat files into the Shadowbase replication engine for processing.

This ability is useful, for example, to ingest into Shadowbase application-generated logs of events that otherwise are not readily available via an audit trail.

3. Extend SOLV using the Toolkit

The Toolkit can also be used to extend the SOLV loading capabilities to allow for reading and injecting events from flat files into the Shadowbase replication engine for processing:

• Extract select data from a source database
• Transform/cleanse the data into the proper format
• Save the data into flat files understood by an ETL utility
• Load the files into a target database environment not directly supported by Shadowbase replication