Replication at the Speed of Change – a Fast, ScalableReplication Solution for Near Real-Time HTAP Processing

Dennis Butterstein Daniel MartinKnut Stolze Felix Beier

IBM Research & Development GmbHSchönaicher Strasse 220

71032 Böblingen, Germany

dennis.butterstein@ibm.com{danmartin,stolze,febe}@de.ibm.com

Jia Zhong Lingyun WangIBM Silicon Valley Lab

555 Bailey AveSan Jose, CA 95141, United States

jia.zhong@ibm.comwlingyun@us.ibm.com

ABSTRACTThe IBM Db2 Analytics Accelerator (IDAA) is a state-of-the art hybrid database system that seamlessly extendsthe strong transactional capabilities of Db2 for z/OS withthe very fast column-store processing in Db2 Warehouse.The Accelerator maintains a copy of the data from Db2 forz/OS in its backend database. Data can be synchronizedat a single point in time with a granularity of a table, oneor more of its partitions, or incrementally as rows changedusing replication technology.

IBM Change Data Capture (CDC) has been employed asreplication technology since IDAA version 3. Version 7.5.0introduces a superior replication technology as a replace-ment for IDAA’s use of CDC – Integrated Synchronization.In this paper, we present how Integrated Synchronization im-proves the performance by orders of magnitudes, paving theway for near real-time Hybrid Transactional and Analytical(HTAP) processing.

PVLDB Reference Format:Dennis Butterstein, Daniel Martin, Knut Stolze, Felix Beier, JiaZhong, Lingyun Wang. Replication at the Speed of Change –a Fast, Scalable Replication Solution for Near Real-Time HTAPProcessing. PVLDB, 13(12): 3245-3257, 2020.DOI: https://doi.org/10.14778/3415478.3415548

1. INTRODUCTIONIBM Db2 Analytics Accelerator (IDAA)1 is an enhance-

ment of Db2 for z/OS (Db2z) for analytical workloads. Thecurrent version of IDAA is an evolution of IBM Smart An-alytics Optimizer (ISAO)[17] which was using the BLINKin-memory query engine[14, 3]. In order to quickly broadenthe scope and functionality of SQL statements, the query

1http://www.ibm.com/software/data/db2/zos/analytics-accelerator/

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copyof this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Forany use beyond those covered by this license, obtain permission by emailinginfo@vldb.org. Copyright is held by the owner/author(s). Publication rightslicensed to the VLDB Endowment.Proceedings of the VLDB Endowment, Vol. 13, No. 12ISSN 2150-8097.DOI: https://doi.org/10.14778/3415478.3415548

engine was first replaced with Netezza (IBM PureData Sys-tem for Analytics2). Netezza’s design is to always use tablescans on all of its disks in parallel, leveraging FPGAs to ap-ply decompression, projection and filtering operations beforethe data hits the main processors of the cluster nodes. Therow-major organized tables were hash-distributed across allnodes (and disks) based on selected columns (to facilitateco-located joins) or in a random fashion. The engine itselfis optimized for table scans; besides Zone Maps there are noother structures (e. g., indices) that optimize the processingof predicates. With the advent of IDAA Version 6, Netezzawas replaced with Db2 Warehouse (Db2wh) and its column-store engine relying solely on general purpose processors.Various minor deviations between Db2z and Netezza wereremoved that way, which resolved many SQL syntax anddatatype incompatibilities faced by our customers.

Figure 1 gives an overview of the system: the Acceler-ator is an appliance add-on to Db2z running on an IBMzEnterprise mainframe. It comes in form of a purpose-built software and hardware combination that is attached tothe mainframe via (redundant) network connections to al-low Db2z to dynamically offload scan-intensive queries. TheAccelerator enhances the Db2z database with the capabil-ity to efficiently process all types of analytical workloads,typical for data warehousing and standardized reports aswell as ad-hoc analytical queries. Furthermore, data trans-formations inside the Accelerator are supported to simplifyor even avoid separate ETL pipelines. At the same time,the combined hybrid database retains the superior transac-tional query performance of Db2z. Query access as well asadministration use a common interface – from the outside,the hybrid system appears mostly like a single database.

Table maintenance operations (e. g., reorganization andstatistics collection) are fully automated and scheduled au-tonomically in the background. An IDAA installation in-herits all System Z attributes known from Db2z itself: thedata is owned by Db2z, i. e., security and access control,backup, data governance etc. are all managed by Db2z it-self. The Accelerator does not change any of the existingprocedures or violate any of the existing concepts. As aresult, the combination of Db2z and the Accelerator is a

2http://www.ibm.com/software/data/puredata/analytics/system/

3245

http://www.ibm.com/software/data/db2/zos/analytics-accelerator/http://www.ibm.com/software/data/puredata/analytics/system/

Db2 for z/OS

Client Application

TXN Log

Optimizer

Admin Stored Procedures

Database Runtime

UNLOAD Utility

Log Reader

Accelerator Services

Db2 Warehouse(Db2 LUW)

Accelerator Server

Batch Load

Incremental Update

Query Execution

Optimizer

Database Runtime

SQLDb2 LUW dialect

SQLDb2 z/OS dialect

SQLDb2 LUW dialect

ParallelUnload Streams

Change Records

INSERT /UPDATE /DELETE

Bulk INSERT

CALL AdminStored Procedure

Figure 1: IDAA Architecture

hybrid system that consists of two architecturally very dif-ferent databases systems: a traditional RDBMS that has itsstrengths at processing highly concurrent OLTP workloads,using well-established serialization and indexing concepts –and the Accelerator engine available for processing complexqueries.

Both systems not only complement each other, but canalso shield each other from workloads that may have a neg-ative impact like exhaustive use of resources if dispatched tothe less suitable system. For these query routing decisions,the Db2z optimizer implements a heuristic that decides if agiven query should be processed by Db2z itself or if it shouldbe offloaded to the Accelerator.

It is very important to understand that nothing has tobe changed on existing applications that are already usingDb2z in order to take advantage of the acceleration: theyare still connecting only to Db2z and use its specific SQLdialect. Deep integration into existing Db2z componentsensures that only little specific training is required to oper-ate the Accelerator. Db2z remains the owner of the data;data maintenance, backup and monitoring procedures donot change.

Because IDAA operates on copies of the tables in Db2z,any changes to these tables must also be reflected on theAccelerator. This is normally done by running the IDAAACCEL LOAD TABLES stored procedure that refreshes eitheran entire table or the changed (horizontal) partitions of thattable on the Accelerator. This refresh mechanism matchesmany use-cases of IDAA, as analytical systems are oftenupdated by ETL jobs that run in batch mode so that changesare applied in bulk on a scheduled basis.

Another use case, however, is to allow Accelerator-basedreporting over “online” data, providing a solution that com-bines the features of traditionally separated OLTP vs. re-porting systems that are connected by aforementioned ETLjobs. For such scenarios, the table or partition granularity ofthe refresh mechanisms is too coarse: if the total size of thechanges is rather low but the changes themselves are spreadover multiple partitions (or the tables themselves are notpartitioned), then re-loading the majority of partitions oran entire table implies a lot of unnecessary work. Also, thepractical minimum latency that can be achieved by runningthe aforementioned refresh procedures is too long; it is im-practical to run the stored procedures every few minutes.

Therefore, IDAA offers its “Incremental Update” featurewhich refreshes the copy tables on the Accelerator by asyn-chronously monitoring the Db2z transaction log for changes.Completed and committed transactions are replicated to theAccelerator. For efficiency reasons, those changes are ap-plied in “micro batches” that typically contain data fromall transactions that committed during a certain time in-terval (e. g., the last 60 seconds). This technology was im-plemented by the existing replication product called IBMInfosphere Change Data Capture (CDC)3.

Integrated Synchroniziation (InSync) is the successor forCDC in the context of IDAA. Since it is not a general pur-pose replication product, it is much more light-weight andspecifically tailored to the Accelerator. Installation and ad-ministration of InSync is significantly reduced, while repli-cation performance exceeds CDC.

2. RELATED WORKRelational database system vendors have been focusing on

analytical DBMS appliances for a while; popular examplesare Oracle Exadata4, the Teradata appliance5, EXASOL’sEXASolution Appliance6, and SAP’s HANA7 [7, 13]. Sim-ilarly, appliances running MapReduce implementations to-gether with an analytical DBMS have become increasinglypopular, sometimes under the name of “Big Data Analyt-ics Platform”. They use MapReduce programs for analyticson unstructured data and add additional ETL flows into ananalytical DBMS that runs on the same appliance for re-porting and prediction on cleansed, structured data. Essen-tially these systems promise an all-in-one solution for datatransformation and analytics of high-volumes of structuredand unstructured data. Examples are the EMC GreenplumDCA8 and the “Aster Big Analytics Appliance”9.

Common to all of these products is a shared nothing ar-chitecture that hash-partitions the data of each table anddistributes it across the available compute nodes of the clus-ter. There is a strong focus on optimizing long-running,scan-heavy queries and analytical functions. Data access ismostly read-only and data modifications are done in largebatches through bulk load interfaces. The distributed, sha-red nothing architecture makes these systems unsuitable forOLTP with a mixed read/write workload and the require-ment for very low response times. Clearly, it is impossibleto achieve the short access path and thus the low responsetimes of a traditional OLTP architecture because these sys-tems require a query to go through all nodes of a cluster forscanning a table and a coordinator node to combine interme-diate results before synchronizing and possibly re-distributework at runtime. By design, these systems have a minimumresponse time of several tens or hundreds milliseconds, asopposed to OLTP systems where the lowest response timestypically are in the single digit milliseconds range.

3http://www.ibm.com/software/data/infosphere/change-data-capture/4http://www.oracle.com/us/products/database/exadata/overview/index.html5http://www.teradata.com/data-appliance/6http://www.exasol.com/en/exasolution/data-warehouse-appliance.html7http://www.sap.com/solutions/technology/in-memory-computing-platform/hana/overview/index.epx8http://www.greenplum.com/products/greenplum-dca9http://www.asterdata.com/product/big-analytics-appliance.php

3246

http://www.ibm.com/software/data/infosphere/change-data-capture/http://www.ibm.com/software/data/infosphere/change-data-capture/http://www.oracle.com/us/products/database/exadata/overview/index.htmlhttp://www.oracle.com/us/products/database/exadata/overview/index.htmlhttp://www.teradata.com/data-appliance/http://www.exasol.com/en/exasolution/data-warehouse-appliance.htmlhttp://www.exasol.com/en/exasolution/data-warehouse-appliance.htmlhttp://www.sap.com/solutions/technology/in-memory-computing-platform/hana/overview/index.epxhttp://www.sap.com/solutions/technology/in-memory-computing-platform/hana/overview/index.epxhttp://www.greenplum.com/products/greenplum-dcahttp://www.asterdata.com/product/big-analytics-appliance.php

This is the reason why OLTP and analytics require sep-arate software and hardware architectures. IDAA is no ex-ception to this – it also comes in form of two fundamentallydifferent systems. However, IDAA is based on the notionof an Analytics Accelerator, a transparent attachment to anOLTP system that automatically classifies analytical work-loads and chooses the “best” platform for the work to beprocessed. In contrast to other systems, Db2z and the Ac-celerator are tightly integrated so that users experience asingle coherent system in terms of usage, monitoring, andadministration.

There are several proposals of architectures in the researchcommunity that resemble the hybrid design of IDAA: [1,2] propose an RDBMS that uses the MapReduce paradigmfor the communication and coordination layer and a modi-fied PostgreSQL database for the individual nodes. In [4],a review of the feasibility of architectures and implementa-tion methods to achieve real-time data analysis is presented,[5] proposes a theoretical architecture for a hybrid OLTP /OLAP system. [15] proposes a hybrid row-column store toavoid having to operate on duplicate data and deal with con-sistency and data coherence problems. HYRISE [9, 8] is anin-memory column store that automatically partitions tablesbased on data access patterns to optimize for cache locality.OLTP-style access yields to partitions with a higher num-ber of columns vs. analytical access that yields to smallerpartitions.

In contrast to these approaches that all propose a single,unified system architecture for a hybrid DBMS, we believethat the required properties for OLTP and analytics are fun-damentally different and in some cases even mutually exclu-sive as noted in Section 1. The transactions costs per query(network I/O, scheduling, etc.) as exposed by the differ-ent designs are so different that we believe that a singlesystem can never fit analytical and OLTP access patternsequally well. As a result, our approach is to implement sep-arate systems, using the OLTP system at the front (so thatthe performance characteristic of OLTP workloads are notnegatively impacted) and to use query routing (somewhatcomparable to what has been proposed in [6]) and data syn-chronization mechanisms to integrate the analytics system.Of course, this means that the data may not be perfectlyin sync, but IDAA provides several mechanisms to achievecoherency at the application level, i. e., although there aretemporary inconsistencies, applications will not notice thisbecause the refresh mechanisms have been chosen and inte-grated into the application’s data loading mechanisms (seeSection 3.3 for details). Besides IDAA, some systems follow-ing an approach with separated OLTP and OLAP systemshave been proposed [11, 10] with BatchDB being closelyrelated to the presented approach. InSync additionally in-cludes inter- and intra-transaction optimizations speedingup replication and does not execute OLAP queries on onedata snapshot only. Instead, issued HTAP queries will in-dividually be handled to fulfill per-query data freshness re-quirements.

3. DESIGN AND OPERATION OF INCRE-MENTAL UPDATE

The following section describes the architecture of theIDAA incremental update based on InSync. Several signifi-cant aspects are discussed, which lead to the replacement ofCDC.

3.1 High-Level Replication ArchitectureAll replication products have two major components: (1)

a Capture Agent that processes the transaction logs of thesource database system, and (2) an Apply Agent that pro-cesses the changes from (1) and applies them to the tar-get database system.10 Figure 2 denotes these as Db2z LogReader and Integrated Synchronization, respectively.

Contrary to CDC, the Log Reader does not actively pro-cess the transaction log. Instead, it is the Integration Syn-chronization component that requests a range of log recordsactively. Another noticeable difference of the Log Reader isthat it does not try to filter log records for committed trans-actions. All the transaction-related processing as well asforming the micro batches and applying them to the targettables is handled by the Integrated Synchroniziation com-ponent on the target side.

Similar to CDC, InSync has to deal with heterogenousdatabase systems. Db2z and Db2wh use different page for-mats and different structures in the transaction log. There-fore, commonly known techniques for log shipping in ho-mogenous environments cannot be adopted, at least notwithout a significant amount of effort.

3.2 Management InterfaceAdministration of InSync is fully handled through the

IDAA Server, providing a seamless user interface. It re-ceives requests whether to enable/disable replication for aspecific table, or to start/stop the replication process itself.Such user-driver requests as well as other internal requestsare passed on to Integrated Synchronization.

Internal requests are needed because InSync operates di-rectly on the underlying database. Concurrent operations ofthe IDAA Server – like schema manipulations or bulk loads –have to be synchronized. Many of the IDAA Server’s actionsthat manipulate data are only available when replication issuspended for the affected tables for the duration of the op-eration.

3.3 Integration with Bulk LoadAn important aspect is the integration of InSync with

IDAA’s bulk-LOAD mechanism. InSync itself does not han-dle the transfer of an entire table, e. g., after it has been firstdefined – bulk load is specialized and highly tuned for that.Therefore, IDAA allows to combine both mechanisms in thefollowing fashion:

1. IDAA LOAD is used for the initial transfer of the fulltable data from Db2z to the Accelerator. During theprocessing of the LOAD, InSync remembers the cur-rent write position in the Db2z transaction log, i. e., itsets a capture point.

2. Subsequently, InSync uses this capture point as startposition for requesting transaction log records from the

10Additional components may be used, for example, to manage metadata about the table mappings, access control, etc.

3247

CDC Agent for Db2 Warehouse

Db2 for z/OS

IDAA Studio

TXN Log

Admin Stored Procedures

Database Runtime

Db2 Warehouse(Db2 LUW)

Accelerator Server

IncrementalUpdate

Controller

Change Records

CALL AdminStored Procedure

CDC Agent for Db2 for z/OS

Log Reader

INSERT

UPDATE

DELETE

Client Application

MetadataCatalog

CDC Access Server

Log Apply

Insert /Update /Delete

Datasource &SubscriptionManagement

Integrated Synchronization

Db2 for z/OS

IDAA Studio

TXN Log

Admin Stored Procedures

Database Runtime

Db2 Warehouse(Db2 LUW)

Accelerator Server

IncrementalUpdate

Controller

CALL AdminStored Procedure

Db2 for z/OSLog Reader

INSERT

UPDATE

DELETE

Client Application

MetadataCatalog

Dynamic Apply

Datasource &SubscriptionManagement

InSyncController

LogParserChange Records

(Db2 for z/OS format)

Change Consolidation

External /Memory

Table

Figure 2: Architecture of the Incremental Update Feature

Log Reader. Thus, it obtains all changes on the Db2ztables resulting from inserts, updates and deletes.

3. When large scale changes for the table occur or the ta-ble is modified by Db2z utilities that bypass the trans-action log, the administrator can reload the table viathe bulk load, which temporarily suspends replication.Replication must remain suspended while the table isloaded to avoid the risk of applying an update twice(once as part of the IDAA load and again via InSync).

4. Once the table has been synchronized, replication is re-activated so that all changes occurring after the newcapture point can again be replicated by CDC.

Note that a table is still fully enabled for writes in Db2zduring bulk-load. Such changes are spilled, and applied byreplication in a special apply mode when the load phasefinished. This special apply mode merges the spill queue forthe table with the changes that came from the bulk load,and makes sure to apply row changes not captured by load,and to ignore them otherwise.

This way, IDAA offers a best-of-both-worlds approachwhere log-based replication is used for continuous propa-gation of small, dispersed changes while efficient full tableor partition-level transfer is used after bulk changes or op-erations that bypass the log. The bulk load can be appliedto reload the full table, or only a selected set of partitions.In the second case, it is enforced that change detection isused to identify at least all those partitions that must bereloaded. If no clear decision can be made whether a par-tition was changed or not, it is reloaded in order to guar-antee data consistency. Db2z collects run-time statistics onpartition-level already, and that is exploited to make thosedecisions.

To actually enable this level of integration, InSync is awareof the concept of table partitions in Db2z, which is the gran-ularity on which bulk load operates. In order to map thepartition granularity from the Db2z table to the table inDb2wh, the table’s schema is augmented by the Accelera-tor with a hidden column used to record the partition IDfor each row. The partition ID is not using a 1:1 mapping;instead, the rows of one partition from the Db2z side mayoccur multiple times (e. g., due to multiple subsequent bulk

loads) in the corresponding Db2wh table using different par-tition ID values. IDAA controls the data access such thatonly those rows from the most recent bulk load are visible fornew queries. Rows from previous bulk loads are kept untilall queries accessing them are finished, and then those rowsare purged physically. Thus, IDAA implements a variantof multi-version concurrency control to enable concurrentquery execution and bulk loads.

As a consequence, InSync needs to maintain the hiddenpartition ID column while updates are being propagatedfrom Db2z to Db2wh. The partitioning information foreach modified row can be deduced from Db2z transactionlog information already. However, the mapping applied byIDAA’s bulk load is not considered by InSync as the syn-chronization of this mapping incurs a too high performancepenalty. InSync does apply a 1:1 mapping, and IDAA isaware of that and applies special treatment for replicatedrows in subsequent bulk loads. [16]

3.4 Integration with Query ProcessingFor the query processing, the introduction of incremental

update means literally no change. The decision to executequeries in the Accelerator is based only on the availabil-ity of data and not on its currency, so the query offloadheuristics are not affected. As transactions are continuouslypropagated from the source to the target database, they areisolated from concurrently running queries by the normalisolation mechanisms in the target database. Of course, thefact that data continuously changes shows up at the appli-cation level, e.g. running the same analytic report twice onthe Accelerator may yield different results – the same wayas if the report query had been executed twice on the sourcesystem.

3248

3.5 Where is the problem then?Why invest effort to develop a high speed, low latency

replication system despite having a proven solution in thefield? Why would any customer take the step to migratefrom CDC to InSync? Although IDAA presents a transpar-ent solution for applications, stale data is not acceptable fora lot of use cases. For example, a credit card fraud detec-tion system does not protect from fraudulent transactions.For such kind of applications, being able to run queries onguaranteed current data is key.

The guarantee IDAA gives is that any change committedin the source database before a query started will be seenwhen the query is offloaded to the Accelerator. To imple-ment this guarantee, a delay protocol is employed. It worksin combination with micro-batching of captured changes.As a result, the query may see changes from later commits,but it will never miss any commit that happened before thequery started:

1. Db2z optimizer offloads eligible queries to the IDAA-Server, along with the position in the transaction logfor the commit of the last change that happened toany of the referenced tables.

2. IDAA-Server blocks the query and sends a request toInSync containing the position in the transaction logalong with a maximum wait time.

3. Upon each batch commit, InSync checks whether aparticular query request can be satisfied (request trans-action log position

Log Reading

Read Log Records(buffer, active, archive)

Decrypt(storage)

Log Preprocessing

Sort Log Records(across all members)

Decompress(row images)

Filter Log Records(UR control record only,

DBID / OBID)

Log Decoding

Cache Log Records(L1 in-memory, L2 disk)

Decoding(each column

value from row)

TCP/IP Network Transfer

(committed URs only in CDC format,

commit order)

Log Staging(staging space,

in-memory unit of recovery buckets)

Log Formatting(into CDC-internal format once UR is

ready)

Log Compensation(remove rows in staging space)

Figure 3: CDC Capture Agent pipeline

Log Reading

Read Log Records(buffer, active, archive)

Decrypt(storage)

Log Preprocessing

Sort Log Records(across all members)

Decompress(row images)

Filter Log Records(UR control record only,

DBID / OBID)

TCP/IP Network Transfer

(raw log stream)

Figure 4: Log Reader pipeline

3. Read data must be decoded. This means, it has toextract every column from a log record in Db2z pagerecord format (reordered row format, basic row for-mat). More precisely, it has to convert the source’sinternal row format to actual column formats, e. g.,text or decimal numbers. CDC employs a canonical,intermediate data format to transfer data from sourceto target. This has to be done for every single col-umn of every single row. In fact, processing overheadis even worse. Update log records contain a before im-age as well as an after image and therefore more thanthe actual amount of columns in a table row have tobe decoded. These operations are very expensive andsum up quickly.

4. Stage rows: The decoded rows are collected in UR(Unit of Recovery) buckets in the staging space. Usu-ally, these buckets contain a whole set of transactions.UR buckets are kept in the staging space until thecapture agent has read the commit record completinga given UR. Until then, data is accumulated in thestaging space which might cause the staging space tooverflow.

5. Format rows into CDC-internal format: As soon as anUR bucket is ready (commit was read), the log data isformatted into CDC-internal format to transfer it tothe target. Again, the decoded data is converted to adatabase-independent intermediate representation.

6. Process compensated changes: Rollbacks write com-pensation records to the transaction logs. These spe-cial log records contain inverse operations to the spe-cific action they revert. CDC is designed to save net-work traffic as well as target database load aiming ata low latency. Following this principle, compensatedoperations are removed from the staging space easingbefore they would be transmitted. This directly savesthe network traffic for transmission along with the cor-responding processing effort in the target database.

These savings come at the expense of more complexstaging space management. Every compensation logrecord needs a look up in the staging space main mem-ory and on disk). The removal steps are done one byone for every compensation record causing a significantamount of staging space management overhead.

Summing it up, out of the 6 steps two (1 to 2) cannotbe avoided. These steps are inherent to read the trans-action log. In contrast, steps 3 to 6 are required due toCDCs platform independent design and can be removed ormoved to the target system, which InSync does (cf. Fig-ure 4). Support of multiple source and target databaseplatforms imposes the need for intermediate representationsalong with the required conversions between source, interme-diate and target formats. The effort for this design accumu-lates throughout the whole pipeline and becomes apparentin both:

1. generated CPU processing costs (MIPS: Millions In-structions per Second): System Z uses a consumption-based pricing model for executed instructions on a pro-cessor. Thus, customers are interested in keeping theamount of used MIPS as low as possible.

2. throughput: number of inserts, updates and deletes(IUDs) per second on the target. Throughput directlyaffects the LOAD-strategy of a customer. Higher IUDrates enables users to solely rely on replication tech-nology. The lower the IUD rates the more likely it isthat customers have to use periodic reloads because ofconsistently growing latency.

Above metrics are key differentiators for customers. Con-sequently, every reduction in costs and every increase inthroughput will directly affect customer success.

3250

4.2 Log Reader EnhancementsThe key design points in log reader development were to

tightly integrate the capture agent into Db2z and removeprocessing steps from the capture agent component. Tightintegration of the log reader component simplifies installa-tion and maintainability lowers the hurdle to start and op-erate IDAA. Moving processing steps from source to targetdemands for technical finesses presented in the following.

4.2.1 Move processing steps to targetPushing required processing steps towards the Accelera-

tor moves computation intensive tasks away from the sourcesystem towards the appliance. The cost model on the appli-ance does not include any computational costs and thereforedirectly lowers total cost of ownership for an IDAA system.

1. Decoding of the log records is no longer done on thesource. In fact, encoded log record are given to thetarget database (Db2wh). There, the external tableinterface has been extended with the ability to handlenative Db2z log records. No intermediate representa-tion for data transmission is needed for this replicationsolution.

2. A staging space is no longer used. All read transactionlog records – regardless of whether a transaction will berolled back or committed in the future – is transmittedto target system. This helps to keep latency low as notime is wasted waiting for end of a transaction whilethe throughput is kept high.

3. With the removal of staging space, net-effect and com-pensation processing have moved to a different placein the processing pipeline. It is now done when data isprepared to be applied to the target database. At thispoint in time, it is already know if a transaction hasbeen committed. Net-effect / compensation processingare done on whole batches of transactions rather thanon single transactions which is much more efficient.

The combination of these modifications leads to a highlyefficient appliance-centric processing model. It saves a sig-nificant amount of computation resources on the source side,which is the OLTP system.

5. DATA PROCESSINGThe Log Reader on the source database system (Db2z)

provides an RACF 12 or Passticket 13 protected API forinteraction with InSync. The InSync component employsa single thread continuously interacting with the providedAPI to download chunks of variable size from the trans-action log and store them in a cache. A single log parserthread retrieves data from the cache, reads through thelog, classifies the log records according to their particu-lar type (such as BEGIN, INSERT, DELETE, UPDATE,COMPENSATION, COMMIT or ROLLBACK) and trig-gers actions in the Transaction Manager defined by the type

12Resource Access Control Facility managing access to resources onz/OS operated mainframe systems (https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zsecurity/zsecc_042.htm).

13Temporarily valid one-time password, generated from an ap-plication dependent name, a secret token and current time in-formation (https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.icha700/pass.htm).

of the log record read. The Transaction Manager collectsdata manipulating log records classified by transaction in in-memory representations of transactions – the Transaction-WorkItem. TransactionWorkItems can either be processedstand-alone or as a batch (5.3) together with other trans-actions. Batch processing exhibits optimization possibilitiesbetween the transactions. Single TransactionWorkItems orbatches of TransactionWorkItems are finally applied to thetarget database by constructing delete and insert statementsusing Dynamic Apply Path Selection (7). In the followingsections, the data processing depicted in figure 5 will beexplained in more detail.

5.1 Log ParserThe Log Download thread continuously downloads chunks

of the transaction log to the first in first out (FIFO) orga-nized cache. Cache contents are processed single-threadedby the Log Parser. FIFO processing is essential, as it guar-antees replay of the transmitted changes in log sequenceorder. Violations of log sequence order would lead to in-consistencies of the target database state. The Log Parserthread, reads through the cache contents on a per recordbasis. Records are structured as figure 7 illustrates (onlyshowing data relevant for parsing). Each record14 contains– besides the raw row information – a header that is re-quired for Db2z processing. Within these headers, infor-mation about the log records’ context are provided. Thisinformation is typical for write-ahead database systems em-ploying Algorithms for Recovery and Isolation ExploitingSemantics (ARIES [12]) and makes our approach generallyapplicable for such databases.

Some of the information important for parsing the log are:Type: Denotes the type of a log record. All of the replica-

tion relevant types for InSync trigger corresponding methodsin Transaction Manager. Relevant types are:

• Unit of Recovery Control Records: Mark transactionstart (BEGIN), rollback (ABORT) or commit (COM-MIT). The Log Record Identifier of a BEGIN recordis used as URID field for all records wrapped in theopened transaction.

• Undo / Redo log records: Redo log records denotemodifications to database state (inserts / updates /deletes). These records contain information to unique-ly identify the table they operate on. In Db2z thisis implemented in the form of the database identifier(DBID) and the object identifier (OBID) along withthe actual row data in database internal format. Incase a transaction is rolled back, compensation logrecords are written to the transaction log. Compen-sation log records contain the inverse operation of theaction in the log record they undo.

• Utility Log Records: Not only data modifications writelog records to the transaction log, but also utilities formaintenance of similar. Depending on the utility logrecord, InSync is able to process the log record or itremoves the affected table from replication.

• Diagnostic Log Records: On top of above required andwell known log records, a new class of records is added:

14https://www.ibm.com/support/knowledgecenter/en/SSEPEK_12.0.0/admin/src/tpc/db2z_logrecord.html

3251

https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zsecurity/zsecc_042.htmhttps://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zsecurity/zsecc_042.htmhttps://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.icha700/pass.htmhttps://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.icha700/pass.htmhttps://www.ibm.com/support/knowledgecenter/en/SSEPEK_12.0.0/admin/src/tpc/db2z_logrecord.htmlhttps://www.ibm.com/support/knowledgecenter/en/SSEPEK_12.0.0/admin/src/tpc/db2z_logrecord.html

Commit Queue

Commit Thread Pool

Per-UR Worker Pool

Main Thread

Db2/z Log Parser

Transaction Manager

committed work itemsin source commit order

One queue per Unit of Recovery,work items in log stream order

Transaction Work Item

Transaction Work Item

Db2/z Log Record Stream

Figure 5: InSync Technical Overview

Tran

sact

ion

2

Table 2

Table 1

Deletes Inserts

ID 2ID 1

Deletes InsertsID 3

Table 3

Table 1

Deletes InsertsID 2

Deletes InsertsID 4

Tran

sact

ion

1

Batch

Table 2

Table 1

Deletes Inserts

ID 2ID 1

ID 2

Deletes InsertsID 3

Table 3

Deletes InsertsID 4

Merged Changes

Deletes Inserts

Deletes InsertsID 3

Deletes InsertsID 4

Pipe Contents

Figure 6: Batch Processing

Data…OBIDDBIDTypeURIDLog Record Identifier

Headers

Figure 7: Log Record Layout

Diagnostic log records. These records are written e. g.,when the database schema is modified. This enablesInSync to detect schema modifications using their re-spective log records. If a compatible schema change isencountered, it is processed accordingly. Unsupportedschema changes cause affected tables to be removedfrom replication.

Log record identifier: Used to identify a particular logrecord within the transaction log. In the case of Db2z theused identifier varies depending on whether the system op-erates in data sharing mode or not. In non-data sharingmode the unique Relative Byte Address (RBA) whereas indata sharing environments the (potentially) non-unique LogRecord Sequence Number (LRSN) is used.

Unit of Recovery ID: Undo / Redo Log Records carrya URID set to the log record identifier of the BEGIN recordthe undo / redo record belongs to.

The parsing process itself is designed to be as efficientas possible. To achieve optimal performance, only necessaryinformation is read by using numeric offsets. Extraction andconversion of any row data is strictly avoided. Offset usage,allows for direct access to the relevant information withoutthe need to touch any unnecessary log record data. Theonly data fields extracted from the (opaque) log records, arekey columns required for further processing. The log recordrow data is not converted or processed any further by theparser. Instead, it is passed unmodified to the TransactionManager for further processing.

5.2 Transaction ManagerThe Transaction Manager collects all information about

transactions stored in the transaction log. As we transmit alldata – no matter if a transaction has been committed or willever be committed, the Transaction Manager maintains anassociative data structure which maps a URID to its corre-sponding in-memory transaction representation. This datastructure helps the Transaction Manager to assign undo /redo log records to the wrapping transaction and keep trackof the order of the transactions (ascending order with respectto the URID). The Log Parser calls a method according tothe log record type read from the transaction log and triggersone of the following actions in the Transaction Manager:

• BEGIN: Creates an in-memory representation (Trans-actionWorkItem) associated with its URID that de-picts a transaction in the transaction log. All subse-quent insert, update, delete or compensation recordsassociated with the above URID along with their re-spective LRSNs are collected in a transaction.

• INSERT / DELETE: Adds row image and LRSN tothe transaction the records belong to. The DBID andOBID fields are used to store change records in thetransactions separated by table.

• COMPENSATION: Adds the LRSN of the log recordto be undone to the compensation list.

• UPDATE: Update records, in contrast to delete or in-sert records, contain two row images: a before imagewhich contains the full row image before the updateand an after image – a full row image of the row af-ter the update. To add update information to theTransaction Manager, it is decomposed in a deleteconstructed from the before image and an insert con-structed from the after image. The constructed imagesare added similar to native insert / delete records.This update decomposition allows for optimizationslike net-effect processing 5.3.

3252

• COMMIT: Signals successful completion of a transac-tion. When encountering this record, the correspond-ing transaction identified by its URID is removed fromthe open transaction heap and either added to the cur-rently open batch (5.3) or send to a thread for process-ing right away.

• ABORT: Signals that the transaction has been rolledback. The URID is extracted and the transaction in-cluding all its occupied resources are released. Thetransaction is removed from the transaction heap.

5.3 Batch ProcessingA batch is a set of committed transactions. Its size is de-

termined by two configurable parameters: maximum num-ber of transactions and batch timeout. Either the maximumsize of a batch or the batch timeout is reached. Both of thecriterions lead to closing a batch (called sealing). Sealedbatches are added to the commit queue (FIFO) which isprocessed single-threaded.

The first processing step for a batch is to merge the col-lected changes across all transactions. This is done by iterat-ing through the transactions (in commit order) and appendthe changes of every transaction to table objects identifiedby a combination of DBID and OBID. The result is a collec-tion of table objects accumulating the changes of all trans-actions in a batch in commit order. Keeping the commitorder is essential to guarantee a consistent target databasestate.

In the second step, unnecessary database operations areremoved – a process called net-effect processing. But whatare unnecessary database operations? To answer this ques-tion, recall that updates are decomposed in a delete andan insert row (see 5.2). When executing queries, for eachtable in a batch two queries will be issued: a delete queryand a insert query. Delete queries are executed before theinsert queries for a table. This processing scheme has twoimmediate consequences:

1. Deletion of a row appearing after the same rows insertin the same transaction will lead to incorrect targetdatabase state: The delete will be applied to the targetdatabase before the insert of the row.

2. Deletion of a row with an ID that has been insertedin a preceding transaction can be safely removed fromthe batch without affecting correctness.

To ensure correct target database state when situation 1appears and to optimize the amount of database operationsconducted on the target in situation 2 InSync employs thefollowing techniques:

1. Intra transaction net-effect: A delete issued after aninsert of the same row in the same transaction is notcollected in the transaction in-memory representation.Instead, the collected insert is marked for removal dur-ing net-effect processing.

2. Inter transaction net-effect: When a delete on a rowwith an ID that has been inserted in a preceding trans-action occurs, both records are marked for removalduring net-effect processing.

Identification of rows net-effect can be applied to, is doneby comparing the records using a unique key. Before be-ing able to enable a table for replication a unique key (real

unique key or informational unique key) has to be definedon the table. This key is the only data which is extractedfrom the row data images.

Assume InSync collects the transmitted transaction log ina single batch and the log sequence received was (transactionID, table number, operation unique key): BEGIN 1 (1, 1,INSERT 1) (1, 1, DELETE 1) BEGIN 2 (2, 3, DELETE 4)(2, 1, DELETE 2) (1, 2, INSERT 3) COMMIT 1 COMMIT2

The constructed in-memory representation of the transac-tions is depicted in figure 6. First BEGIN 1 causes creationof a in-memory representation of transaction 1. Log record(1, 1, INSERT 1) causes the row image for INSERT ID tobe added to the inserted rows for table 1. The following logrecord triggers intra transaction net-effect processing: therow identified by ID 1 has been added before in the sametransaction (TX). Consequently, the delete for that row isnot added to the in-memory representation. Instead, thedelete causes the preceding insert of ID 1 to be marked forremoval during net-effect processing. BEGIN 2 creates ain-memory representation for transaction 2. The next logrecords add an insert of ID 3 to table 2 of TX 1 and deletesof ID 2 and ID 4 for tables 1 and 3 in TX 2. The commitsfor TX 1 and TX 2 cause the transactions to be completedand added to the currently open batch. After the batch hasbeen sealed, the collected changes are merged. Table 1 nowcombines all changes of table 1 of TX 1 and TX 2. Mergingof table 2 and 3 does not have any effect as these tables arecontained in one of the transactions only. After merging,intra transaction net-effect processing is applied. In table 1ID 2 has been inserted in TX 2 and subsequently deleted inTX 2. Hence, without applying both operations to the tar-get database, the database state will still be correct. Whenwriting the changes to the data structure (pipes / memoryareas) as input for query processing, both the insertion andthe deletion of ID 1 will be ignored. For tables 2 and 3 thecollected changes will be written to the data structure asexpected.

While intra transaction net-effect processing is required tokeep database state correct (a delete will always be executedbefore an insert, even if it was after the insert in the logstream), inter transaction processing is a optimization only.Log sequences like continued updates to a row with the sameID exhibit the optimizations’ full potential. In this case,we only need to conduct the first delete (if the row waspresent in the table before) and the last insert. All updatesin between are unnecessary work and hence can be removedby net-effect processing.

Be aware that this processing scheme modifies the trans-action pattern between source and target. On the source,each transaction collected in the batch is recorded as singletransaction. After they have been collected in a batch, all ofthe transactions are consolidated in a single transaction onthe target. This implies that either a whole batch is appliedto the target or none of the changes of a batch is applied atall.

Besides batch processing, InSync employs another han-dling strategy for transactions exhibiting a particular be-haviour. If a transaction has not been committed before aconfigured threshold of collected rows is reached, no com-pensation records and no deletes have been read, it can beprocessed in preapply mode.Instead of waiting for a commitfor the transaction and adding it to the current batch, preap-

3253

ply immediately starts to feed changes into an open trans-action but does not commit the executed SQL statementin the target database. The changes remain uncommitteduntil the commit log record for the transaction has beenprocessed. This is comparable to the concept of speculativeexecution – to speed up overall processing, we assume thatthe transaction will eventually commit since rollbacks aremuch less likely than commits in production transactionaldatabase systems.

The benefits of preapply is improved latency: if a largetransaction is only processed when the commit is seen, alarge number of changes have to be applied as part of pro-cessing the commit. However, the time can be used alreadyfor streaming the changes in smaller chunks into the tar-get table. At commit time, only a small tail of remainingchanges has to be processed. In addition, preapply helps toreduce memory pressure. Instead of channeling the transac-tions’ changes through the whole batch processing and po-tentially keeping memory resources reserved by large trans-actions, preapply bypasses the full blown processing andmakes sure to free resources as fast as possible and prioritizelarge transaction that might potentially increase latency.

However, this comes at a cost: If the preapplied transac-tion does not commit as expected, a delete or compensationrecord is read, there is some overhead involved for restoringa safe system state. First of all, the changes applied to theopen connection need to be rolled back. Afterwards, thepreapplied transaction is completed as any other transac-tion. Upon reading its’ commit log record there is a specialhandling however. The currently open batch will be sealedand processed. This is what we call a batch breaking changewhich has impact on the performance of InSync.

The field of application for preapply also is very narrow.Transactions can only be preapplied as long as there haven’tbeen any deletes and compensation records. As you may re-member from section 5.3, deletes are always processed beforeinserts. As a consequence, if a delete is read in a transactionafter a insert has already been applied, this order would beviolated and can lead to results in the target database state.The same is true for compensation records – log records al-ready applied to the connection cannot be easily removedwhen a compensation record is read.

6. MEMORY MANAGEMENTMemory Management is an important aspect for InSync.

All row images extracted from the transaction log alongwith the data structures required for their management, arechannelled through InSync’s limited amount of main mem-ory. Long running transactions might accumulate a largeamount of row images while millions of small transactionsare processed in parallel. Figure 6 shows the data flow ofthe extracted row images.

When the Transaction Manager appends row images tothe corresponding in-memory transaction representation, itkeeps track of the data size of the row images stored in mainmemory. The collected row images keep accumulating untila configured threshold (typically 1 MB) would be exceededby adding the current row image. Instead of adding therow image to the current 1 MB block, the block is writ-ten (spilled) to disk. It contains a header keeping track ofthe block id and some additional administration informationenabling the system for easy removal of row images hav-ing been compensated or identified as unnecessary during

net-effect processing. The rows register for removal duringnet-effect processing are associated with the block id, an off-set marking the start of the record and the record length.This allows for easy skipping when row images are written(drained) to the data structures passed to the queries. Anew block is allocated, prefilled with the header informationand collects the subsequent row images added. As a conse-quence, a maximum of 1 MB is used for each – inserts anddeletes – per table per transaction in memory. In memorycritical situations, the block threshold can be adjusted tosmaller values which will cause the row images to be spilledto disk earlier. This process is done before the merge (5.3) ofthe changes is conducted. Note that this technique is vitalto support long running transactions. InSync keeps collect-ing the data for these transactions until they finally commitand are added to a batch after hours or even days. In theworst case, only 2 MB per table of this transaction will bekept in memory at any time.

At some point in time, the processing pipeline will beready to prepare the row data for application to the tar-get database. To write the row images into the source datastructures to be applied, the spilled row images are processedby memory mapped I/O. A 1 MB block is mapped from diskinto memory one by one to keep memory consumption as lowas possible. While a block is mapped into memory, it is writ-ten (drained) into the source data structures for databaseapplication. Not all row images are written - instead, rowimages detected as unnecessary (compensation, net-effect)are skipped during the write process using the associatedblock id, offset and length of the record. This techniquemakes it possible, to remove rows identified as unnecessaryand draining the surviving row images into the source datastructures for target database application in just one pass.

7. DYNAMIC APPLY PATH SELECTIONDepending on the workload, InSync is confronted with dif-

ferent demands defined by workload characteristics. Thesecharacteristics vary along dimensions like number of changesper transactions, number of changes per table and numberof tables per transaction. InSync chooses from two differentpaths to feed data to the target database - ”external table”and ”memory table”. Both paths have their strengths andweaknesses defining their area of application.

An external table is a table stored in a database externalfile (or pipe) while still being accessible through SQL. Theexternal table is expensive to use in terms of time consump-tion for preparation. A new (fenced) process is spawnedfor each external table operation, leading to a preparationtime of approximately 200ms without having processed asingle row. Once the external table is build however, it pro-cesses rows very fast due to heavy internal parallelizationand pipelining. Consequently, a particular amount of rowshas to be processed by an external table for the preparationtime to be amortized.

On the other hand, the memory table is designed to avoidany form of preparation times, like plan prepare, thread ormemory preallocation etc. It operates on fixed-size chunksthat represent a number of rows in source-internal format,and does all of its processing streamlined on this chunk ofdata. In return, for large number of rows its processingis slower than the external table. Still, memory table iscapable of processing a significant amount of rows duringthe preparation period external table processing requires.

3254

Transaction Work Item /Transaction Work Item Batch

Table Changes

Pipe Writer Thread

Transaction Manager

Insert

Memory-Mapped I/O

Row Cache(row images)

Header

Row Cache(row images)

Header

Row Images

Header

External TablePipe

Row Images

Header

Row Images

Header

Row Images

Header

Delete

1 MB off-JVMHeap Memory Block

Spill

Disk

Memory

Figure 8: Memory Management Figure 9: Workload Characteristics

To decide what query path to employ, InSync uses a pre-configured row threshold defining the amount of rows thatcan be processed before external table should be used. Thedecision is done for both, insert and delete operations on aper-table level. This way, the apply paths operate at theirsweet spot guaranteeing high speed and low latency datareplication.

Figure 9 represents an exemplary workload distribution.Most of the query execution calls, no matter if deletes orupdates, contain a small amount of changed rows only withan emphasis between 1 and 9999 changed rows. This areais a perfect match for employing memory table. The lessfrequent cases from 100000 to 9999999 would be processedby external table.

8. AVOIDING DATA CONVERSIONTo be able to support combinations of different source

and target data sources, general-purpose replication prod-ucts typically employ an internal, canonical data format tomediate between different source and target systems. Eachreplication source then transforms all captured rows intothis intermediate format, sends it to the target which thentransforms it again into a target-native format. It’s a trade-off between interoperability and performance, favouring in-teroperability.

InSync is not a general purpose replication product, itsfocus is row to column-store synchronisation with a strictidentity mapping (exactly the same schema on source andtarget) for tables. Interoperability between different sourceand target systems is a secondary priority. For this reason,we use the source-native data format as the general data en-coding scheme for data exchange between replication sourceand target. This not only avoids any form of row-level dataconversions in the capture and apply parts, but also comeswith the benefit of making it extremely lightweight to cap-ture changes on the source system. The source system is justforwarding transaction log records to the target in additionto flushing them to stable storage.

On the target system, the transaction log is parsed, butonly up to the point where the actual before- and after rowimages are stored. The rows itself remain encoded in source-native format and the target replication system does nottouch them. Instead, these are treated as opaque bytes,stored, staged and fed into the target system by means ofthe external- or memory table (cf. 7).

The target database system has been enhanced with con-version logic to its external- and memory table operations tobe able to parse and interpret the source-native row format.This has proven to increase efficiency for small “trickle” styleoperations and for large, bulk-style changes alike. The trans-formation operators run directly in the insert codepath, andshare the source-format parser. For low-latency trickle op-erations (memory table), the parser control block is cachedbetween calls by means of a binary descriptor, to minimizeper-row latency added by the row parser and conversionlogic. Putting the data transformation logic into the targetdatabase also decreased the complexity of the replicationsystem. It is unaware of the actual encoding used by sourceor target, and “just” moves bytes between both sides.

9. EVALUATIONThe performance evaluation was conducted on an IBM

zEnterprise z14 system with 6 general processors, 2 co-pro-cessors, and 96 GB of main memory, connected via 10 GBnetwork to a 1-rack IBM Integrated Analytics System (IIAS)(M4002-010). Software-wise z/OS 02.02.00 together withDb2z v12 was applied. The used test workload consisted of500 tables concurrently receiving inserts and updates witha peak transaction rate of 64, 400 inserted rows per secondand 64, 400 updates per second and 30 concurrently runningHTAP query threads.

9.1 Log Reader ResultsFor Capture Agent / Log Reader tests, the recovery log

has been pre-populated to measure log reader work in iso-lation. Figure 10 shows the performance result, comparingexecutions times of different phases in the CDC and InSynclog reader pipelines (cf. Figures 3 and 4). Processing shareson general processors and co-processors are shaded in differ-ent colors.

The CDC log processing phase, which reads log records,decrypts, decompresses, sorts, and filters them before con-verting them into the CDC-internal canonical format andapplying change compensation, consumes the largest amountof processing time. The InSync log reader, which skipsdata conversion operations and change compensation, justrequires roughly half of CDC’s processing time. Becausemore data is transmitted via the network in the InSync-case, more time is spent in this phase compared to CDC,which is negligible compared to the overall processing costs.

3255

2114.42

1076.44

2094.92

966.34

16.14

82.10

3.36

28.00

0 500 1000 1500 2000

CDC

InSync

CDC

InSync

CDC

InSync

CDC

InSyncTO

TAL

Log

Processing

Datab

ase

Access

Network

Stack

Processing Time in Seconds

General ProcessorCoprocessor

Figure 10: Log Reader Performance Comparison

0

50000

100000

150000

200000

250000

300000

10

100

1000

10000

100000

1000000

0 10 20 30 40 50 60 70 80

Late

ncy

in m

s

Thro

ughp

ut in

Ope

rati

ons /

s

Time

Comparison of InSync and CDC Performance

Inserts / s - InSync Deletes / s - InSync Inserts / s - CDCDeletes / s - CDC Latency in ms - InSync Latency in ms - CDC

Figure 11: Performance Evaluation

The biggest advantage of InSync can be seen when com-paring processing shares of general and co-processors. Whileonly 0.13 % of CDC’s workload is eligible for co-processoroffloading, 99.87 % can be offloaded to inexpensive coproces-sors for InSync. This results in both – a significant amountof saved time and reduced processing cost.

9.2 End-to-End Performance ResultsIn the end-to-end case, we focus on comparison of through-

put and latency metrics.For the workload described above, figure 11 shows CDCs

latency continuously increasing until it reaches about 3 min-utes. In fact, the figure only shows only an excerpt of theworkload data for clearness. CDC latency builds up as highas 30 minutes. Throughput for inserts peaks at about 35, 000rows per second. Delete throughput reaches about 12, 000rows per second.

InSync latency peaks at about 10 seconds (cf. 11) whichmeans that InSync is capable of keeping the latency 180times lower than CDC despite the fact, that the latency doesnot constantly grow. Instead, the latency is stable and doesnot show any signs of falling behind. Throughput for insertspeaks at about 200, 000 processed records per second whiledelete throughput reaches about 100, 000 processed recordsper second.

The average query wait time has been determined to be51 seconds in the CDC workload whereas InSync causes anaverage query wait time of about 6 seconds. Single HTAPquery execution proved to exhibit stable wait times around6 seconds for InSync.

Comparing above results, it turns out that InSync exhibitsa 180× lower latency while increasing the insert throughputby about 6× and the delete throughput by about 8× for thetest workload. This is a significant improvement over CDCreplication in IDAA. For the sake of completeness, pleasenote that the throughput numbers displayed are for only onethread. Both systems usually run with 8 parallel threadsin a IDAA setup – the displayed number do only show afraction of the actual system performance. The query waittime reduced by 8.5× and proved to be very stable withInSync.

10. LIMITATIONSThere are some limitations that exist with the introduc-

tion of Integrated Synchronization:Data type support: IDAA does not support the data

types ∗LOB, TIMESTAMP WITH TIME ZONE and XML

due to their special internal storage concepts. This restric-tion existed in earlier IDAA versions.

Indexes are not enforced on the target database:As in earlier IDAA versions, we do not enforce referential in-tegrity on the target database. It is enforced on the sourcedatabase and therefore guaranteed on the target databaseas well. However, there are situations when referential in-tegrity is and has to be violated on the target system (e.g.continuous replication: bulk load a table during replication)

Tables containing partitions with reordered as wellas basic row format: As of this writing, Integrated Syn-chronization is not capable to replicate tables containingpartitions in reordered as well as basic row format. Thislimitation will only be hit, when a transaction collects bothrow format types.

Replication delay: Very special workload characteris-tics might cause HTAP queries to unexpectedly time out. Ifthe source system is flooded with exceptional high amountsof data irrelevant for replication you might observe this be-havior as data collected in a batch might not get applied tothe target database. This is a very unrealistic scenario in acustomer environment but relevant for our testing.

11. SUMMARYIn this paper we have presented a new, fast, scalable, low

cost replication technology to enhance IBM Db2 AnalyticsAccelerators’ Incremental Update feature. Fast data repli-cation from a high performance transactional database sys-tem to a column store database system needs to be guar-anteed in high load situations while data integrity betweenthe database systems needs to be maintained. The imple-mented solution turned out to be both, performant – latencydropped 180× while throughput increased 6× for inserts and8× for deletes – and affordable – processing time decreasedby about 50% for the test workload.

The next steps of our work will be to gradually add schemachange replication. At the time of this writing, IDAA hasonly rudimentary schema change support (ADD COLUMN)when using CDC. InSync has no support yet. Starting withADD COLUMN we will add support for schema changes toInSync step by step. Performance-wise we strive to furtherdecrease latency while increasing throughput. This is a vitalstep for bringing OLTP and OLAP workloads together withIDAA’s own implementation of Hybrid Transactional An-alytical Processing HTAP. The lower the latency, the lessquery delay time HTAP queries will exhibit to guaranteetransactionally-consistent query results.

3256

12. REFERENCES[1] A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi,

A. Silberschatz, and A. Rasin. HadoopDB: AnArchitectural Hybrid of MapReduce and DBMSTechnologies for Analytical Workloads. PVLDB,2(1):922–933, Aug. 2009.

[2] K. Bajda-Pawlikowski, D. Abadi, A. Silberschatz, andE. Paulson. Efficient Processing of Data WarehousingQueries in a Split Execution Environment. InProceedings of the 2011 international conference onManagement of data, 2011.

[3] R. Barber, P. Bendel, M. Czech, O. Draese, F. Ho,N. Hrle, S. Idreos, M. Kim, O. Koeth, and J. Lee.Business Analytics in (a) Blink. IEEE DataEngineering Bulletin, 2012.

[4] S. Conn. OLTP and OLAP Data Integration: AReview of Feasible Implementation Methods andArchitectures for Real Time Data Analysis. InSoutheastCon, 2005. Proceedings. IEEE, 2005.

[5] J. Dittrich and A. Jindal. Towards a One Size Fits AllDatabase Architecture. In Outrageous Ideas andVision Track, 5th Biennial Conference on InnovativeData Systems Research, CIDR, 2011.

[6] S. Elnaffar. A Methodology for Auto-RecognizingDBMS Workloads. In Proceedings of the 2002conference of the Centre for Advanced Studies onCollaborative research, 2002.

[7] F. Färber, S. Cha, J. Primsch, C. Bornhövd, S. Sigg,and W. Lehner. Sap hana database: data managementfor modern business applications. ACM SigmodRecord, 2012.

[8] M. Grund, P. Cudre-Mauroux, J. Krüger, S. Madden,and H. Plattner. An overview of hyrise-a mainmemory hybrid storage engine. Bulletin of the IEEEComputer Society Technical Committee on DataEngineering, 2012.

[9] M. Grund, J. Krüger, H. Plattner, A. Zeier,P. Cudre-Mauroux, and S. Madden. HYRISE: A MainMemory Hybrid Storage Engine. PVLDB,

4(2):105–116, Nov. 2010.

[10] J. Lee, S. Moon, K. H. Kim, D. H. Kim, S. K. Cha,and W.-S. Han. Parallel Replication across Formats inSAP HANA for Scaling out Mixed OLTP/OLAPWorkloads. PVLDB, 10(12):1598–1609, Aug. 2017.

[11] D. Makreshanski, J. Giceva, C. Barthels, andG. Alonso. BatchDB: Efficient Isolated Execution ofHybrid OLTP+OLAP Workloads for InteractiveApplications. Proceedings of the 43th SIGMODinternational conference on Management of data,pages 37–50, May 2017.

[12] C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, andP. Schwarz. ARIES: A Transaction Recovery MethodSupporting Fine-Granularity Locking and PartialRollbacks Using Write-Ahead Logginig. ACMTransactions on Database Systems, pages 94–162,March 1992.

[13] H. Plattner. A Common Database Approach for OLTPand OLAP Using an In-Memory Column Database. InProceedings of the 35th SIGMOD internationalconference on Management of data, 2009.

[14] V. Raman, G. Swart, L. Qiao, F. Reiss, V. Dialani,D. Kossmann, I. Narang, and R. Sidle. Constant-TimeQuery Processing. In IEEE 24th InternationalConference on Data Engineering (ICDE), 2008.

[15] J. Schaffner, A. Bog, J. Krüger, and A. Zeier. AHybrid Row-Column OLTP Database Architecture forOperational Reporting. Business Intelligence for theReal-Time Enterprise, 2009.

[16] K. Stolze, F. Beier, and J. Müller. Partial Reload ofIncrementally Updated Tables in Analytic DatabaseAccelerators. Datenbanksysteme für Business,Technologie und Web (BTW) 2019 : Tagung vom 4. -8. März 2019 in Rostock, pages 453–463, March 2019.

[17] K. Stolze, F. Beier, K. Sattler, S. Sprenger,C. Grolimund, and M. Czech. Architecture of a HighlyScalable Data Warehouse Appliance Integrated toMainframe Database Systems. Database Systems forBusiness, Technology, and the Web (BTW), 2011.

3257

IntroductionRelated WorkDesign and Operation of Incremental UpdateHigh-Level Replication ArchitectureManagement InterfaceIntegration with Bulk LoadIntegration with Query ProcessingWhere is the problem then?

The Capture Agent on a DietChange Data Capture: Capture AgentLog Reader EnhancementsMove processing steps to target

Data ProcessingLog ParserTransaction ManagerBatch Processing

Memory ManagementDynamic Apply Path SelectionAvoiding data conversionEvaluationLog Reader ResultsEnd-to-End Performance Results

LimitationsSummaryReferences