Data Movement - Architecture Essentials

Modified on Wed, 12 Mar 2025 07:32 by Biswajit Dash — Categorized as: blueprint, classical architecture, data architecture, whiteprint

Problem Statement

What is data-movement, and when does a system require this? What are the different architectural solutions available for such a system? How to determine the most appropriate solution for data-movement?

Table of Contents [Hide/Show]

Problem Statement
Solution Abstract
   What? When? How?
   The Inner-workings
      Replication Pipe-line
      ETL Pipe-line
      Decision Matrix
Glossary
References

Solution Abstract

What? When? How?

What is data-movement? The act of moving a copy of data from it's current physical location(s) to different location(s), reliably and repeatedly.

When to use data-movement? A data-movement from source-system to target-system is needed if :-

the need is to provide a redundant copy of the data to the target-system;
the target-system is restricted from accessing the source-system data;
the systems have different data structure requirements;
the target-system requires - read-only access, or does not require updates made to it to persist;
the availability of data from source-system is no more relevant, e.g., a 8x5 availability becomes 24x7 resulting in no window for offline processing;
the network or platform stability of the source-system is not reliable; and/or
the network bandwidth is inadequate to support the real-time data access and performance needs.

How to determine the solution? This is a multi-step process, i.e., :-

identify the driving forces which warrant the need of a data-movement solution;
determine the high-level architectural approach that can be applied; and
finally determine design approach at a more precision level.

The Inner-workings

Any data-movement system is composed of three building blocks - AMW i.e., Acquire, Manipulate, Write.
Each block implements the below.

(A) Acquire : The extraction of data from the source-system.
(M) Manipulate : The enrichment of data acquired from source-system.
(W) Write: Writing the acquired and manipulated data to the target-system.

The exact nature or responsibility of each of these building blocks can led to two architectural solutions, i.e.,

Replication, and
ETL or Extract-Transform-Load.

Replication Pipe-line

ETL Pipe-line

Decision Matrix

With the basic data-movement understanding above, the below two bus-matrix can be used to derive the most applicable architecture and design approach. In the matrix - higher the number of crosses 'X' better aligned is the solution to the problem statement.

Glossary

Master	The source data store which is to be copied. This is considered the original source of data.
Slave	The target data store.
Master-Master Replication	A bi-directional replication between two source and target systems. Based on the replication direction the master and slave switch role.
Master-Slave Replication	An uni-directional replication from source (master) to target (slave).
Master-Master Row Synch	A special Master-Master replication in which conflict resolution is done at row level.
Master-Slave-Snapshot Replication	A special Master-Slave replication, in which the complete data from source is copied to the target at a point-of-time.
Master-Slave-Cascade	A Master-Slave replication topology where replication from source target is achieved through a cascade of intermediate master/slave.
Batch ETL	An ETL in which the data from source systems are copied to target schema in a batch process, primarily to be used as Data Warehouse based reporting.
Real-time/Streaming ETL	An ETL in which continuous stream of data is processed and copied incrementally to the target schema, primarily used for real-time analytics.

References

Patterns & Prctices - Data Patterns (Microsoft)
The Data Warehouse ETL Toolkit (Ralph Kimball)

Paper Code: TWP_1001.10, Version: 1.0, Author: Biswajit Dash, License: CC-BY-ND, Published: Jan-2016