Comprehensive Enterprise Mainframe-to-Amazon Web Services (AWS) Data Replication

As a Technology Partner in the AWS Partner Network (APN), we understands the necessity of having a straight forward solution for enhancing business agility. tcVision provides a proven Mainframe-to-AWS data replication solution, and customers benefit from modernizing their data within within the flexible, scalable, pay-as-you-go offering on AWS.Customers that still have business-critical data locked in mainframes want to exploit this data with Amazon Web Services (AWS) agile services for analytics purposes, to create new communication channels, and for quickly developing new innovations.

tcVISION for Amazon Web Services is a powerful and comprehensive solution for real-time, bi-directional replication of mainframe data to AWS.

Mainframe Data to AWS Customer Use Cases

Mainframe data stores often hold large amounts of complex, critical data in proprietary legacy formats. This data can be difficult to extract and inconsistent with modern databases, data types, and data tools.

tcVISION for Amazon Web Services is a powerful and comprehensive solution for real-time, bi-directional replication of mainframe data to AWS.

The tcVISION replication software solves this problem by replicating data to modern AWS databases in real-time, and allows for the following use-cases:

Analytics

As soon as the mainframe data is unlocked and available within an AWS data store, such as Amazon Simple Storage Service (Amazon S3), customers can use the wide array of analytics and machine learning services for easy access to all relevant data, without compromising security or governance. Customers select AWS data services from data catalog and data processing to interactive analytics, real-time analytics, operational analytics, dashboards, and data warehousing.

Innovation

Once mainframe data is on AWS, customers innovate by creating new functions with cloud speed. For example, some choose to create microservices with a complete serverless stack via AWS Lambda, accessing their mainframe data. Others decide to make mainframe data available to new channels, such as mobile users via Amazon API Gateway or voice devices such as Amazon Alexa. Mainframe data can also be easily moved into machine learning models.

Finally, customers can take advantage of the AWS global infrastructure to deploy applications with key mainframe data globally, quickly delivering innovations worldwide.

Incremental Migration

When piece mealing a large mainframe-to-AWS migration, some customers have to synchronize data between their mainframe and AWS. Bi-directional, real-time data replication allows incremental migration without manually developing data synchronization code.

Amazon Web Services

tcVISION on AWS architecture overview.

tcVISION’s GUI modeling and mapping, and ease of migrating data to AWS makes it an ideal choice for modernizing large mainframe environments.

Additionally, tcVISION for Amazon Web Services can synchronise mission critical data from a mainframe system on AWS. Real-time, bi-directional data synchronization enables changes on either system to be reflected on the other system (e.g., a change to a PostgreSQL table on AWS is reflected on the mainframe database).

This allows businesses to modernise an application on AWS without disrupting the existing critical work on the legacy system.

tcVISION is ready to meet all AWS Cloud requirements, technologies, and challenges. tcVISION’s support for AWS Cloud as a target is fully integrated alongside traditional Linux/Unix/Windows (LUW) targets such as PostgreSQL, Oracle Database, IBM Db2 LUW, Software AG ADABAS LUW, IBM Informix, Sybase, Microsoft SQL Server, and ODBC.

tcVISION supports many mainframe data sources for both online and offline scenarios. Data can be replicated from IBM Db2 z/OS, Db2 z/VSE, VSAM, IMS/DB, CA IDMS, CA DATACOM, or Software AG ADABAS. tcVISION can replicate data to many targets including Amazon Aurora, Amazon Relational Database Service (Amazon RDS), or Amazon S3. To learn more, see the complete list of supported tcVISION sources and targets.

tcVISION has software components installed on the mainframe and on a Windows or Linux Amazon Elastic Compute Cloud (Amazon EC2) instance.

Often, customers establish multiple environments, such as development, quality assurance (QA), and production, with each being associated with a different mainframe LPAR hosting a tcVISION Manager and communicating with a corresponding tcVISION Manager installed on an Amazon EC2 instance. These components communicate over TCP/IP or SSL/TLS using a VPN or AWS Direct Connect.

tcVISION stores metadata in a relational database, such as Amazon RDS. The tcVISION Manager components are administered by the tcVISION Control Board, which can be installed on-premises or in an Amazon EC2 instance. This allows tcVISION users to create metadata, create and control replication scripts, and control database interaction. tcVISION’s product architecture is designed to minimize the mainframe resource utilization.

Metadata from the source and target environments is acquired via the tcVISION Control Board. Sources and targets can be mapped one-to-one, one-to-many, many-to-one, and many-to-many. There is built-in intelligence to understand mainframe and relational database management system (RDBMS) data types and stores.

The Control Board facilitates the mapping of the mainframe copybooks, redefines, data dictionaries, data catalogs, code pages, data type mapping, and more via the user-friendly interface. The Repository Editor allows users to control data transformations.

tcVISION Replication Modes

tcVISION’s synchronization process requires an initial bulk load of the mainframe source database into AWS data targets such as Aurora, Amazon RDS, or Amazon S3. After the initial bulk load, tcVISION’s Change Data Capture (CDC) is utilized to keep the mainframe data and AWS data source in constant synchronization.

The entire process is designed for minimal impact on the mainframe, meaning no source database outage during the bulk load, and minimal mainframe resource utilization during the bulk load and ongoing replication.

Bulk Load

tcVISION’s bulk load performs the initial target database load, using mainframe source data. Source data can be read directly from the mainframe data store, or can be read from a mainframe backup or unload. The bulk load provides automatic translation of mainframe data types such as EBCDIC packed fields.

Generally, the highest performance is attained by using the backup or unload data versus a direct read of the mainframe database. Moving unload or backup data to the requisite tcVISION Amazon EC2 instance and using native database loaders minimizes network IO, and reduces load time.

Real-Time Replication

tcVISION CDC enables real-time synchronization between the mainframe and AWS data sources, such as Amazon RDS. tcVISION utilizes native logging associated with each mainframe database to capture the data changes on the mainframe platform. This includes adds, updates, and deletes to specific data records.

For reliability, tcVISION operates on an ACID transactional basis, only applying committed transactions, and can restart CDC automatically.

Bi-Directional Replication

When data needs to be replicated from the mainframe to an AWS data source and back from the AWS data source to the mainframe, tcVISION uses CDC on both source and target databases. It has built-in capabilities to fully support bi-directional replication:

‘Looping prevention’ ensures only data changes not made by tcVISION are acted upon.‘Conflict detection’ allows users to pre-define specific actions to be taken when there are data conflicts encountered during bi-directional replication. For example, a conflict detection rule can be specified to change an INSERT to an UPDATE when a database record already exists.

Security, High Availability, and Scalability

tcVISION provides the quality of service required by enterprise data workloads for security, availability, and scalability.

From a security perspective, authentication and access control for tcVISION can be controlled by LDAP, Active Directory, or a mainframe SAF product, such as RACF, ACF2, or Top Secret. In-transit data between tcVISION managers (mainframe-to-AWS) and the Control Board can be encrypted via SSL/TLS. Temporary block storage-based CDC files can reside in encrypted form on disk.

AWS Security

tcVISION high availability architecture on AWS.

During tcVISION’s CDC processing, high availability must be maintained in the AWS environment. The Amazon EC2 instance, which contains the tcVISION Manager, is part of an Auto Scaling Group spread across Availability Zones (AZs) with minimum and maximum of one Amazon EC2 instance.

Upon failure, the replacement Amazon EC2 instance tcVISION Manager is launched and communicates its IP address to the mainframe tcVISION Manager. The mainframe tcVISION Manager then starts communication with the replacement Amazon EC2 tcVISION Manager.

Once the Amazon EC2 tcVISION Manager is restarted, it continues processing at its next logical restart point, using a combination of the LUW and Restart files. LUW files contain committed data transactions not yet applied to the target database. Restart files contain a pointer to the last captured and committed transaction and queued uncommitted CDC data. Both file types are stored on a highly available data store, such as Amazon Elastic File System (EFS).

For production workloads, CCA Software recommends turning on Multi-AZ target and metadata databases.

tcVISION’s scalability is dependent on the type of replication process it performs. tcVISION can run parallel concurrent bulk load processing simultaneously on a single Amazon EC2 instance, or on multiple instances, giving horizontal scalability. Very large tables can be bulk loaded faster by splitting the process into multiple tasks, either by arbitrary intervals, or via row filtering. Row filtering can use a key, partition key, date, etc.

tcVISION scaling for CDC processing can be achieved by running multiple parallel replication streams. The first step is to analyze the files included in logical transactions, as these files must be processed together in sequence.

tcVISION’s CDC process ensures the integrity of each logical transaction, and these files must be processed together. For instance, sets of tables that do not participate in common transactions may be divided into parallel tasks by creating multiple processing scripts.

Transactional consistency is maintained within a task, so it’s important that tables in separate tasks do not participate in common transactions. This approach utilizes multiple tcVISION scripts to create separate replication streams that parallelize reads on the source data transformation, and writes to the target database.

tcVISION Replication from Mainframe Db2 z/OS to Amazon Aurora

tcVISION’s Control Board is a Windows Graphical User Interface (GUI) that allows users to configure the replication stream between various database platforms, including the IBM mainframe and AWS. Using the Control Board and built-in wizards, users can define the metadata and mappings between the mainframe and AWS database target.

tcVISION data scripts are created through wizards. Data scripts control the replication of data from the source (Db2 z/OS) to the target (Aurora). tcVISION bulk load scripts are a type of data script that performs the initial load of the Aurora database.

The script below shows data being accessed directly from the mainframe Db2 z/OS database. Another alternative reducing MIPS consumption is to read the data from a Db2 image copy.

Environments Supported

Mainframe Databases

Non-Mainframe Databases

Big Data / Hadoop

IBM z Systems

z/OS
z/VSE
Linux on z Systems

Distributed Systems

Linux on IBM Power Systems
IBM AIX
Microsoft Windows
Unix
Linux

IBM Db2
IBM IMS/DB / DL1
VSAM
Software AG ADABAS
CA IDMS/DB
CA DATACOM/DB
PDS/PS

IBM Db2 LUW
IBM BLU Acceleration
IBM Informix
IBM NETEZZA
Oracle
Sybase
Microsoft SQL Server
Software AG ADABAS LUW
PostgreSQL
Teradata
MongoDB
Flat File Integration
SAP Hana
MySQL / MariaDB

JSON
Avro
KAFKA
with Avro
with CSV
with JSON
Hadoop Data Lakes
HDFS
CSV

Cloud

Aurora MySQL
Aurora PostgreSQL
AWS S3
Amazon Web Services
Amazon Kinesis
Microsoft Azure
Amazon Kinesis
Azure SQL-Database
Azure Database for MySQL/MariaDB
Azure Database for PostgreSQL
Azure Event Hubs

tcVision Summary

tcVISION considerably simplifies mainframe data exchange processes.

The structure of the existing mainframe data is analyzed by tcVISION processors, then automatically mapped to a target data mapping.

The data mapping information is presented in a user-friendly and transparent format – even for users with no mainframe knowledge.

The mapping information is saved in a meta data repository hosted on a relational database, and can easily be made available to other applications.

The Windows-based Control Board of tcVISION provides an easy-to-use facility to administer the data flow.

tcVISION provides a variety of interfaces to allow seamless integration with ETL or EAI solutions.

It also offers different CDC methods to identify mainframe data.

The change data capture method deployed depends on the source database (Db2, VSAM, DL/1, IMS/DB, ADABAS, CA-IDMS, DATACOM/DB, SQL Server, Oracle), the data volume, the volume of changed data and the required currency of the information.

Changes are automatically transferred to the targets in time intervals or in real-time by tcVISION data change publishing facilities.

tcVISION has ETL bulk load functionality and uses native change data capture (CDC) mechanisms, keeping the data between mainframe databases and modern databases in constant real-time synchronization. Its support of bi-directional replication allows changes to be captured and updated in two different environments, including modern databases.

tcVISION is a powerful tool for data and application modernization, allowing legacy mainframe teams to continue their work while modern teams use leading edge technology on the same data. It enables modern data warehouses and databases to have access to key corporate data residing on legacy systems.

While tcVISION can be utilised in your corporate data center, using tcVISION with AWS has several key advantages:

  • Rapid global deployments to 18 key geographic regions
  • Database as a Service– Relational Database Instances (for Oracle, PostgreSQL, MySQL, SQL Server, MariaDB, and Aurora) greatly simplifying administration
  • Powerful, low cost, highly available databases, such as Aurora
  • Extreme high availability using availability zones
  • Numerous infrastructure options to optimise the environment and enhance performance
  • Bursting capabilities to accommodate peak loads
  • Controlling costs through infrastructure optimisation
  • Moving capital expense to operational expenses
  • Extreme security at every layer, maintained by world class security experts

“Security, scalability, resiliency, recoverability, and cost of applications in the cloud are better than what almost any private enterprise could achieve on its own”:

Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud.

Amazon RDS is available on several database instance types – optimised for memory, performance, or I/O – and provides you with six familiar database engines to choose from, including Amazon Aurora, Maria DB, PostgreSQL, Oracle, and Microsoft SQL Server.

tcVISION enables you to migrate or perform real-time replication between your on-premises mainframe and Amazon RDS databases. Your business can rapidly deploy databases globally within minutes with minimal administration requirements.

There are several advantages of a Amazon RDS databases. One especially powerful database is AWS Aurora which can be established as either mySQL or PostgreSQL API compatible.

Your business can leverage these key advantages:

  • Serverless Aurora: With serverless Aurora there is no database size pre-allocation required. The database automatically scales up and down based on load and you pay only for the resources utilised.
  • High availability: HA is built into the architecture of Aurora, with six copies of data maintained across three Availability Zones. That is two copies in each Availability Zone (data center in a region), boosting availability with minor disruption even if an entire Availability Zone is down. In addition, the database is continuously backed up to Amazon S3, so you can take advantage of the high durability of S3 (99.999999999) for your backups.
  • Performance: Amazon Aurora provides 5X the throughput of MySQL and 3X the throughput of standard PostgreSQL running on the same hardware
  • Manageability: Amazon Aurora simplifies administration by handling routine database tasks such as provisioning, patching, backup, recovery, failure detection, and repair.
  • Storage Scaling: Aurora storage scales automatically up to 64TB per database instance, growing and re-balancing I/O across the fleet to provide consistent performance.

Using AWS tools, Auto Scaling, and Elastic Load Balancing, your application can scale up or down based on demand.

Unlike a traditional data center, with the AWS infrastructure environment, you pay only for what you use, and your company is charged only for resource utilization. Users can bring up EC2 virtual Windows or Linux instances for any length of time.

For example, if the mainframe database bulk loads only take an hour to complete, the tcVISION EC2 instances can be brought up for the two hours needed, and shut down when the processing is finished. AWS EC2 instances can be dynamically stopped and restarted while retaining its data on AWS Elastic Block Storage.

This way, your business pays only for what is needed and the instance sizes (CPU, memory, and I/O bandwidth) can be optimised for the required mainframe data migration. or data replication process.

AWS resources, such as EC2 instances, storage, and databases can be rapidly provisioned and de-provisioned. The allows your corporation to approach the architecture with an agile mentality.

Once the initial design and architecture is complete, it can be rapidly tested and adjusted as required to optimise all aspects, including security, availability, performance, and operation readiness. The AWS environment allows users to experiment with various architectures and sizing.

Our Partners

Fujitsu Partner
Oracle Partner
HP-PARTNER
IBM-Partner-Emblem-Web
BOS Logo
Microsoft Partner
Useful Links:
HOMEtcVISION ArchitecturetcVISION OverviewtcVISION Frequent QuestionstcVISION – Amazon Web ServicestcVISION Big DataReal Time Data Replication
Mainframe to CloudData SynchronisationDownloadstcVISION z/StreamtcACCESSNewsRead MoreDEMOPrivacy PolicyContact us
tcVISION is distributed in Australia and New Zealand by CCA Software Pty. Ltd. a division of FreeSoft Asia-Pacific
All Rights Reserved Copyright © 2019 Privacy Policy
error: Alert: Content is protected !!