1. Context: Why the Legacy System Failed

The legacy platform failed not just because it was “old,” but due to structural architectural constraints:

1.1 Metadata Explosion

~400,000 tables in a single schema with no lifecycle management, causing query planner degradation and catalog lookup latency.

1.2 Batch-Oriented Architecture

Monthly/yearly scheduled ingestion with no incremental strategy, causing long freshness gaps and failure propagation across pipelines.​

1.3 Tight Coupling Between Layers

Ingestion, transformation, and reporting tightly bound —changes required full pipeline rewrites with no reusability.

1.4 Lack of Observability

No lineage tracking, limited monitoring, and manual debugging via logs made incident response slow and painful.
‍

2. Architecture Design: Detailed Breakdown

2.1 Design Principles

The new system was built around:
Decoupling: Storage (Iceberg) separated from Compute (Trino) — each scales independently.​
Immutability: ​​Append-only, versioned tables prevent data corruption and enable time travel.
Streaming-First: ​CDC-based ingestion replaces batch jobs for near real-time data freshness.
Idempotency: ​Deterministic pipeline execution ensures safe retries without duplicate data.
Observability​: Metadata and lineage tracking built in from day one via DataHub.

2.2 Data Layering Strategy

We implemented a multi-layered data model:

Layer 1: Raw (Bronze)

• Source-aligned schema
• Minimal transformation
• Append-only ingestion via CDC

Layer 2: Refined (Silver)

• Data cleansing
• Schema normalization
• Deduplication logic

Layer 3: Curated (Gold)

• Business-ready datasets
• Aggregations and joins
• Optimized for query performance

2.3 Storage Design with Apache Iceberg

Why Iceberg?

• Schema evolution without rewriting data
• Partition evolution (critical for long-term scalability)
• ACID transactions on data lake
• Time travel for auditability

Table Design Decisions

• Partitioning Strategy
  
  • Based on query access patterns (e.g., date, region)
  • Avoided over-partitioning (common anti-pattern)
• File Size Optimization
  
  • Target file size: 512MB–1GB
  • Avoided small file problem using compaction jobs
• Compaction Strategy
  
  • Scheduled compaction via Airflow
  • Merge small files → optimize read performance

2.4 Query Layer (Trino)

Cluster Design

• Coordinator + multiple worker nodes
• Auto-scaling enabled via Kubernetes

Query Optimization Techniques

• Predicate pushdown
• Partition pruning
• Broadcast joins for small tables
• Cost-based optimizer tuning

Concurrency Handling

• Resource groups configured
• Query prioritization (BI vs ad-hoc)

2.5 Ingestion Layer (CDC + NiFi)

CDC via Striim

• Log-based change capture
• Handles:
  
  • Inserts
  • Updates
  • Deletes

CDC Design Challenges

• Out-of-order events
• Duplicate events
• Late-arriving dat

Solutions

• Event timestamp ordering
• Deduplication keys
• Watermarking strategies

NiFi Pipelines

• Used for:
  
  • Batch ingestion
  • External data sources
• Back-pressure handling configured

2.6 Orchestration (Airflow)

DAG Design Principles

• Modular DAGs (per domain)
• Idempotent tasks
• Retry policies with exponential backoff

Dependency Management

• Task-level dependencies
• Dataset-based triggering (where applicable)

‍

3. Infrastructure Provisioning

3.1 Kubernetes Architecture

Cluster Setup

• Multi-node cluster
• Node pools:
  
  • Compute-heavy (Trino workers)
  • Storage-heavy (data nodes)
  • General-purpose (Airflow, NiFi)

Key Configurations

• Horizontal Pod Autoscaler (HPA)
• Pod disruption budgets
• Node affinity rules

3.2 Deployment Strategy

Helm-Based Deployments

• Parameterized configurations
• Environment-specific values

Namespace Strategy

• Separate namespaces for:
  
  • ingestion
  • processing
  • orchestration
  • governance

3.3 Storage Infrastructure

• Object storage (S3-compatible)
• Persistent volumes for stateful components

3.4 Networking

• Internal service mesh (optional)
• Secure communication via TLS
• Role-based access control (RBAC)
  ‍

‍4.Development Approach

4.1 Pipeline Framework (DigileEdge)

• Config-driven pipelines (YAML/JSON-based)
• Reusable templates:
  
  • CDC ingestion
  • Batch ingestion
  • Transformation jobs

4.2 Idempotent Processing

• Each pipeline designed to be re-runnable
• No duplicate data on retries

4.3 Schema Evolution Handling

• Iceberg schema evolution
• Backward compatibility maintained

4.4 Version Control

• Git-based
• Code + config versioned together

4.5 Data Quality Framework

• Validation rules:
  
  • Null checks
  • Range checks
  • Referential integrity

5. Implementation Strategy

5.1 Migration Approach

• Incremental migration (table-by-table)
• Parallel run:
  
  • Legacy vs new system‍

5.2 Cutover Strategy

• Shadow mode testing
• Gradual switchover

5.3 Rollback Strategy

• Versioned datasets
• Rollback to previous snapshot

‍

6. Testing Strategy

6.1 Data Validation

• Row-level reconciliation
• Hash-based validation

6.2 Performance Testing

• Benchmark queries
• Load testing with synthetic + real data

6.3 Chaos Testing

• Node failures
• Network interruptions

6.4 Pipeline Testing

• Unit tests for transformations
• Integration tests across DAGs

6.5 SLA Validation

• End-to-end pipeline timing
• Alert thresholds
  ‍

7. Observability & Operations

Monitoring Stack

• Metrics:
  
  • CPU, memory
  • Query latency
• Logs:
  
  • Centralized logging
• Alerts:
  
  • SLA breaches
  • pipeline failures

Data Observability

• Lineage tracking (DataHub)
• Data freshness metrics
• Data quality dashboards
  ‍

8. Final Outcome

• Real-time ingestion: CDC-powered pipeline delivering near-instant data freshness across all domains.​
• Scalable query layer: Trino on Kubernetes auto-scales to handle concurrent BI and ad-hoc workloads.​
• Resilient infrastructure: Kubernetes-native deployment with chaos-tested recovery and versioned rollback.​
• Fully automated workflows: Airflow DAGs orchestrate end-to-end pipelines with observability built in.​

Closing Thought

This wasn’t just modernization—it was a shift from:

data as storage → data as a real-time platform

For more updates, follow us on LinkedIn, Twitter, Facebook, Instagram, and YouTube

‍