2011 Spring Newsletter — Shadowbase Development News
Shadowbase NonStop Development:
Since our last newsletter, NonStop Development has completed the first release of Shadowbase Online Verification & Validation (SOLV VnV) as well as Shadowbase Releases 4.091 and 4.092. We have also completed the Shadowbase ZDL™ (Zero Data Loss) synchronous replication prototype. NonStop Development is currently adding additional support for SQL/MX as a source database as well as some new features for replicating unaudited files.
More specifically, since the last newsletter:
- The first release of Shadowbase Online Verification & Validation (SOLV V&V), a “Database Compare” utility, was completed. SOLV V&V is a key set of technology for ensuring that the source and target databases match. Gravic won some clever patents in this area, particularly for the cases where the source or target schemas differ or either of the databases is actively being updated. The first release is for NonStop business continuity environments using Enscribe data; subsequent releases will add in SQL support, as well as the more heterogeneous replication environments we serve.
- Shadowbase Release 4.090 is now the official shipping release for our NonStop platform and Shadowbase Release 4.092 is nearing general release status after undergoing extensive testing in QA (Shadowbase Release 4.091 was only released as a beta to select customers for testing.) These releases incorporate an extensive mix of features and enhancements in response to our customers’ needs. Please refer to the SOFTDOCS for a complete list of enhancements, new features, and bug fixes. Key release items available in 4.092 include the following:
- Enhanced TCP/IP processing: you can now specify DNS host names instead of IP addresses, more options for switching when the primary connection fails, and faster detection of failed connections on idle circuits.
- Customer requested usability enhancements, including: an optional confirmation before shutdown message, the ability to send PULSE requests immediately (expedited), enhanced user exit APIs for transaction control, and significant enhancements to the Shadowbase configuration generation utilities.
- Improved handling of certain complex (e.g., TMF Volume Recovery) transactions in a bi-directional environment to allow the restart point to advance more quickly.
- The addition of a drain command to the QMGR, to allow data queued on the target system to be easily applied to the target database before the QMGR stops and to specify the processing for partial transactions, should they be aborted or applied? (Knowing this information is useful when the source system has disappeared in a disaster.)
- Significantly enhanced statistics output to allow the user to quickly see the activity associated with a specific file, file partition, or DBS.
- Upper and lower case PATHNAME support for those target databases that use case to distinguish characters.
- Enhanced SUSPENDUPD facility for processing by Open System user exits. The event can now be passed through to the Open System prior to the replication stream being suspended, allowing user exits to intercept and process the event. This capability is useful, for example, when the target side wants to coordinate one or more replication threads to a similar point in the processing stream, or to take a backup, or to shutdown the Open Server target appliers at a specific point in processing.
- Improved date, time, and timestamp support, eliminating the need for most user exits. Intervals are being enhanced as well and will be released in a future version.
- Extensive work continued on Shadowbase Zero Data Loss (ZDL), Gravic’s first synchronous replication engine. It is based on the patented “Coordinated Commits” algorithm for providing efficient synchronous transactional replication while using a high performance asynchronous replication engine as the transport. Shadowbase ZDL provides for zero data loss in the event of catastrophic source system loss by safe-storing the replication data on the target node before allowing the source transaction to commit. Ultimate data replay into the target is still asynchronous. Shadowbase Plus SR™, a future/planned offering, extends this capability to replay the data against the target environment before the source transaction is allowed to commit, and is suitable for active/active environments that need to avoid collisions. We have completed an internal prototype Shadowbase ZDL which implements the Coordinated Commits algorithm and are incorporating some of the findings from the prototype in our current work. If interested in these capabilities, contact Shadowbase Product Management (SBProductManagement@gravic.com) for additional information about these future offerings.
- We are currently concentrating our development efforts on improving our support of SQL/MX databases, particularly as a source. The initial release, scheduled for this summer, will be for uni-directional replication. It will include enhanced Collector capability for processing native SQL/MX as a source audit event, and features a port of our Open Server software to run under OSS to apply the data into the target sql/mx database. As part of this project, we are porting our Collector into a C++ framework using lessons learned from the Shadowbase ZDL prototype to lay the groundwork for additional future enhancements.
- We are also working on customer requested enhancements to our SOLV suite of products, adding to its capabilities in three areas.
- First, we are extending SOLV’s “file chaser” capability. This enhancement extends SOLV’s ablility to replicate a set of application “log” files, in sequence, by monitoring the files and using SOLV to periodically inject newly appended entries into the replication engine so they can be applied to the target environment. This feature is particularly useful if the files being monitored/replicated are nonaudited, since it avoids having to add an intercept library to capture the changes made to the file(s) by your applications.
- Second, a related enhancement will monitor/replicate a set of files and will use SOLV to periodically reload any/all of the files when they change. This monitoring is particularly useful if the files are relatively small, nonaudited, and heavily updated (e.g., an application sequence number file).
- Finally, we are working to automate Shadowbase replication’s existing ability to rollback (or undo) a set of changes made to an existing source file or table, using the event type and before/after image information available from the database audit trails. The working project name of this capability is SOLV UNDO. The approach is to undo a set of changes to records or rows in a source database and to restore them to their original state at a particular point in time, which is often referred to as a “flashback” or “rewind” capability (some refer to it as depressing the “oops” button).
Shadowbase Open Server Development:
Since our last newsletter, we are pleased to report that we have completed our port of our Open Server environment into a C++/object oriented framework using the Visual Studio (Windows) and Eclipse IDE’s. This Shadowbase Release 4.052 was quite a feat as it standardizes the Shadowbase releases for the Linux, Solaris, AIX, HP-UX, and other platforms. All future Shadowbase Open Server enhancements will be built on this code base.
Enhancements to the Open Server code include:
- Expanded use of DLL’s and loadable shared libraries for personality, user exit, and specific database configurations. For example, the Shadowbase Open Server code is now one executable that dynamically loads the personality it is configured as (e.g., DOC Writer, Direct Writer, TRS, TFS, etc.), and also as any user exit library defined for it and database interface it needs (e.g., Oracle, Sybase, SQL Server, MySQL, DB2, etc.).
- Sample IDE support (e.g., Eclipse and Visual Studio projects), as well as script MAKE files, for building Shadowbase user exits for the new environments.
- Extensions to our Execute Immediate and Statement Caching engines, allowing for the various types of engines to interconnect depending on the customer’s specific replication needs. For example, a non-cached source capture can now feed a cached target applier.
- Support for additional 32 and 64-bit environments and database versions, including: Oracle 11g series, Sybase 15 series, SQL Server 2008, etc. Check with Shadowbase Support for the full list of supported platforms and database versions at SBSupport@gravic.com.
- Series of extensions called “consumptive processing” were added. Normally, a replication product provides source database event capture, event filtering/transformation services, transport from the source to the target environment, followed by replicating the events into a target database. With consumptive processing, the replication engine still provides the event capture, filtering/transformation services, and transport from the source to the target environment; however, now the event is consumed by some business logic that is embedded into the engine and is not “replicated” in a classic sense. For example, when feeding ATM/POS events into a real-time fraud detection engine, often the ATM/POS events will need to be injected directly into the fraud detection application via an API and not written into a target database. Shadowbase replication provides the full set of facilities to implement these real-time, event-driven, architectures. If this is of interest to you, contact Shadowbase Product Management at SBProductManagement@gravic.com for a free white paper on Shadowbase’s capabilities in this area.
Presently, the Open Server development team is working on porting the entire Open Server code base to run under OSS on a NonStop platform. Shadowbase software has supported replicating to native sql/mx databases as a target for quite some time; however that replication was typically via the ODBC/MX driver running on a Windows or Linux platform. By porting this function into OSS, Shadowbase replication will now have on-platform support for native SQL/MX as a target.
Currently, Shadowbase products support DB/2 as a target. However, when the target is an IBM mainframe, Shadowbase runs on an external data appliance (e.g., AIX or Linux platform) and uses ODBC to replicate into DB/2. Going forward, we are looking at porting the Shadowbase Open Server functionality to an IBM mainframe environment, for example USS or zLinux. If you are interested in this capability, please speak with Shadowbase Product Management at SBProductManagement@gravic.com.
Be sure to read the softdocs and manuals that come with your release(s) to make sure you stay abreast of the full suite of enhancements in the Shadowbase product line.