Utilities Overview

Data replication is at the center of the HPE Shadowbase product portfolio. However, it is also important to be able to manage, monitor and, if necessary, correct that data in order to detect and resolve anomalous behavior, ensure continuation of proper business operations, and for audit compliance purposes. HPE Shadowbase software solutions include additional utility products to meet these needs. They include:

HPE Shadowbase Audit Compliance Utilities

HPE Shadowbase Audit Log (SAL) – Search and review what and when data was changed

Shadowbase Audit Log is a lense to audit trailsAvailable for the HPE NonStop Server platform as a source (Enscribe, NonStop SQL/MP, NonStopSQL/MX), Shadowbase Audit Log (SAL) software creates a searchable archival database of transactional activity (e.g., inserts, updates, and deletes) on a reporting database for application change data auditing purposes. When configured, SAL software will replicate the HPE NonStop TMF audit trail database change activity to the target reporting database, storing the change information as a series of rows, one per each of the insert, update, and delete statements that were originally issued against the source database. This report allows you to search and review, historically, what data was changed, and when, in the source database.

HPE Shadowbase Audit Reader (SAR) – Show what and when application transactions occurred

Shadowbase Audit Reader analyzes and displays database activitiesShadowbase Audit Reader (SAR) software analyzes and displays all audited database activities on HPE NonStop server systems. It shows what application transactions did, and when they did it, to database files and tables, enabling investigation of how data is being changed. Both current and historical transactional information can be analyzed using a variety of search criteria. SAR software mines this information from the HPE NonStop TMF audit trail files. It optionally reads “foreign” audit (audit generated on another HPE NonStop server system). SAR software supports both Enscribe and NonStop SQL I/O events. It also shows both before and after I/O images, as well as “undo” for aborted transactions. SAR software provides an application view of the change data formatted by data type by querying the NonStop SQL catalog or by accessing the Enscribe DDL information (when available).


HPE Shadowbase Compare

Validates that target database matches its source

Shadowbase-Database-Compare-Technology - CopyHPE Shadowbase Compare software allows you to compare a target Enscribe file or NonStop SQL table to its source, and reports on all discrepancies found between the two (discrepancies may occur, for example, if the user accidentally purges necessary audit trails). Shadowbase Compare is helpful for validating that a target database matches its source, and for satisfying regulatory requirements. It is meant for comparing HPE NonStop to HPE NonStop, like-to-like source/target environments where Expand is available between the nodes (for example, for business continuity environments).


HPE Shadowbase Data Recovery Utilities

HPE Shadowbase UNDO – Selectively “undo,” or rollback, changes made to a database

Shadowbase undo and redoThis Shadowbase utility allows the user to selectively “undo,” or rollback, all of the changes made to a database (Enscribe, NonStop SQL/MP, NonStop SQL/MX), to a previous date/time. This option is useful, for example, to restore a database to an earlier set of values using a selective “as of” approach. Shadowbase UNDO software restores a corrupted database by undoing corrupting database changes while retaining correct database changes, thus leaving the database in a known, consistent, and current state. By maintaining an “Undo Queue” of changes that have been made to the database, Shadowbase UNDO software can follow the Undo Queue in reverse time order to the initial point of corruption and can reverse any corrupting DML changes that have been made. The rollback of corrupted data can be accomplished by Shadowbase UNDO software while the application stays online and continues its processing functions.

HPE Shadowbase REDO – Roll database changes forward to a current and correct state

 Shadowbase REDO software maintains a “Redo Queue” of all changes that have been made to a database (Enscribe, NonStop SQL/MP, NonStop SQL/MX). If a new version of an application is to be run, or if a new database version is to be deployed, or if some other system change is to be made, then a copy of the database modification is created and saved before executing the system. This step is often accomplished by unmirroring the application database and saving one of the mirrors. The upgraded system is then run with the remaining mirror (and can be re-mirrored if sufficient additional disk space is available). If serious problems result in database corruption, the saved mirror can be restored quickly. Shadowbase REDO software deletes corrupting changes from its Redo Queue and applies only the valid database changes from its Redo Queue to the saved mirror in order to roll it forward to a current and correct state.


Other HPE Shadowbase Utilities

HPE Shadowbase Enterprise Manager (SEM)

Graphically manage HPE Shadowbase environments

green-traffic-lightShadowbase Enterprise Manager (SEM), a Windows-based GUI command and monitoring interface, is available to control and monitor Shadowbase replication on all of the platforms that Shadowbase software supports. Ease-of-use features, such as simple graphical stoplights that are configured for each Shadowbase process or component, make the product easy to use for new users and operators. Error messages from all processes and platforms are viewable from the SEM interface, and SEM provides an email/warning interface to pagers to alert operators of impending problems.


HPE Shadowbase Online and Offline Database Loading

HPE Shadowbase Online Loading (SOLV) – Online loading facility for high-volume initial and refresh loading

puzzle piece coming down into it's place The Shadowbase Online Loading (SOLV) utility provides online loading of a source database into a target database, meaning that the source and/or target databases can be open for reading and updating while the load (or copy) occurs. In contrast to other loading products, there is no need to take either the source or target databases offline, making them inaccessible to applications, while the load occurs. The Shadowbase SOLV utility can load audited and non-audited HPE NonStop Enscribe source files and HPE NonStop SQL tables into any target environment and database combination supported by the Shadowbase line of data replication products (e.g., HPE NonStop Enscribe or HPE NonStop SQL targets, or other server targets such as Oracle, SQL Server, DB2, and MySQL). The Shadowbase SOLV utility can perform the load while regular replication is also taking place, consuming the source database changes and merging them with the data being loaded to keep the target fully synchronized with the source while the load takes place.

HPE Shadowbase Extract, Transform, and Load (ETL) – Extract database changes or initial load data into flat files

Online support. Toolbox with tools on laptop. The Shadowbase Extract, Transform, and Load (ETL) Toolkit is used to extract database changes or initial load data into flat files for subsequent ETL loading into a data warehouse using comma-separated value (CSV), fixed-position, and tab-delimited formats. More specifically, for certain applications, you may want to use an existing vendor’s ETL loading utility to load either initial data, or change data, into a data warehouse. The Shadowbase ETL toolkit allows a Shadowbase user to perform these tasks. For example, when coupled with the Shadowbase SOLV loader, the Shadowbase ETL toolkit can be used to extract select data from a source database, transform/cleanse it into the proper format, and save it into flat files that your ETL loading utility can understand and load into a target database environment not directly supported by HPE Shadowbase replication. Using the Shadowbase ETL Toolkit, you can also periodically extract source database change data from the source database’s audit trail (change log), and process it into a flat file format that can be incrementally loaded into the target environment using a micro-batch incremental update approach to keep the target environment synchronized with the source.


Enabling Technology: HPE NonStop AutoTMF, HPE Shadowbase File Chaser, HPE Shadowbase SOLV Snapshot Loading, HPE NonStop AutoSYNC

Help enable the use of HPE Shadowbase data replication

Smachine-cogshadowbase data replication software uses the source database transactional audit logs in order to capture and replicate the database change data. However, not all data updates are audited. Shadowbase File Chaser reads non-audited application-created-and-maintained log files/tables and injects those events into the replication stream to be applied into the target environment, thereby avoiding the need for the source database to be audited, while helping to ensure that such changes are also replicated. Shadowbase SOLV Snapshot Loading is a technique for reading and replicating TMF-audited or non-audited source Enscribe files and SQL tables to a target environment. A backup system needs copies not only of current production data, but of all the other objects necessary for applications to execute correctly; these objects are replicated by the HPE NonStop AutoSYNC product. An alternative to enable Shadowbase replication of non-audited HPE NonStop Enscribe files is to use the HPE NonStop AutoTMF product. AutoTMF automatically converts applications which perform operations on non-audited Enscribe files, making those operations audited (TMF protected), without any application modification.


HPE Shadowbase Queue Manager (QMGR)

Process between Collector and Consumer that safely stores replicated data on disk

queue-manager

Shadowbase data replication software supports extracting the data from the source environment, transferring it to the target environment, and applying it directly into the target database. Alternatively, the data can be queued to disk while it is being replicated to separate the extraction and delivery of the data from the replaying of the data being replicated. This ability allows the target database to be taken offline, for example, without affecting the delivery of the data to the target environment. It is also much faster to write the replicated data into a sequential queue file than to perform random I/O to the actual target database, resulting in less data loss in the event of a failure. This configurable option is provided by the Shadowbase Queue Manager (QMGR). In parallel, Shadowbase replication software will read the data from the queue file and apply it to the target database.