Evaluate log configurations to recommend tiered storage, retention policies, and access controls.

Design stream processing topologies that implement join patterns, aggregation windows, and state management for real-time data transformation.

Optimize real-time data flows by analyzing throughput bottlenecks, partition strategies, and resource allocation to meet SLAs within budget limits.

Explain core patterns for schema evolution (backward/forward/full compatibility, additive vs. breaking changes) and select the right strategy.

Implement versioned event/data contracts with Avro or Protobuf using a schema registry and enforce compatibility rules in CI/CD.

Orchestrate real‑time rollout plans across producers, consumers, and storage (Kafka topics, CDC sinks, warehouses) with monitoring and rollback.

Stream pipeline design by analyzing failure scenarios and business requirements to prevent data loss or duplication.

Implement exactly-once processing semantics across producer, processor, and sink layers using transactions, checkpoints, and idempotent operations.

Evaluate watermarking and windowing configurations to optimize the tradeoff between latency and data completeness.

Explain the execution model of Spark Structured Streaming and build a simple pipeline from a file source to a console sink.

Develop streaming pipelines that integrate with Kafka, apply event-time processing with watermarks, and write reliable outputs to Delta Lake.

Build an end-to-end Spark streaming pipeline that can be deployed in real-world production environments.

Inspect Spark UI and metrics (task duration, shuffle I/O, executor CPU/mem) to find bottlenecks and recommend actionable optimizations.

Apply partitioning and skew mitigation (salting/custom partitioner) & reduce shuffle (broadcast joins, avoid groupByKey, AQE) to improve parallelism.

Configure executors, cores, memory, dynamic allocation and parallelism/caching settings to maximize throughput while meeting defined SLA targets.

Architect a streaming data solution by differentiating between batch, micro-batch, and streaming patterns to solve a specific business problem.

Develop real-time analytics pipelines using window functions and watermarking to aggregate and analyze streaming data.

Optimize a production streaming application by diagnosing performance bottlenecks like data skew and implementing mitigation techniques.

Explain Spark’s streaming model and produce a dashboard-ready table from a simple file source.

Construct a real-time pipeline that ingests from Kafka, processes with Spark, and stores result in Delta using event-time windows and watermarks.

Operate a production-oriented dashboard with refresh policies, monitoring, and failure recovery.

Transform nested and streaming data into analytics-ready tables using programming tools and platforms.

Implement automated data quality checks and integrate these checks into CI/CD pipelines to enforce quality gates.

Build and manage scalable real-time analytics pipelines that block low-quality data and connect curated datasets to Power BI dashboards.

Build and schedule streaming and batch-adjacent workflows using a modern orchestrator, such as Airflow or Prefect.

IImplement reliability patterns like idempotence, checkpointing, DLQs, and backfills for fault-tolerant and exactly-once-ish processing.

Design multi-region recovery strategies (mirroring/replication) and run playbooks to restore pipelines after partial or regional failures.

Explain CDC fundamentals (binlog/WAL) and schema evolution strategies.

Configure a Schema Registry pipeline locally using Debezium and Kafka.

Use streaming SQL (Flink/ksqlDB) to map, cast, and merge divergent schemas into a canonical model.

Examine core real-time data principles and how Kafka and Spark support streaming architectures.

Create real-time pipelines by connecting Kafka topics with Spark Structured Streaming.

Improve and deploy streaming systems using monitoring, fault tolerance, and tuning.