Senior Data Engineer

. Functional Programming & Apache Spark

• Scala Core Mastery: You must understand functional programming paradigms, immutable data structures, pattern matching, and implicit parameters.
• Spark Core & Architecture: Deep knowledge of the internal workings of Apache Spark, including the Catalyst Optimizer, Tungsten execution engine, lazy evaluation, Directed Acyclic Graphs (DAGs), and memory management (execution vs. storage memory).
• Performance Tuning: Ability to identify and resolve performance bottlenecks like data skew, handling OOM (Out Of Memory) errors, optimizing joins (Broadcast vs. Sort-Merge), managing partition sizes, and avoiding expensive shuffle operations.
• Structured APIs & Streaming: Proficiency in Spark DataFrames/Datasets APIs and Spark Structured Streaming for low-latency, real-time data processing.

2. Next-Generation Storage & Table Formats

• Apache Iceberg: Expertise in implementing Iceberg as your open table format over a data lake. You must master features like ACID transactions, time travel, schema evolution (hidden partitioning), and row-level updates/deletes.
• Apache Ozone: Understanding Ozone as a scalable, redundant, and distributed object store designed specifically for Hadoop environments. You should know how it replaces or coexists with HDFS to handle billions of small and large files efficiently.
• Storage Optimization: Skills in managing data compaction (merging small files), snapshot isolation, and choosing optimal file formats like Parquet, ORC, or Avro.

3. The Hadoop Ecosystem Foundation

• HDFS & YARN: While industry focus is shifting toward object storage, you still need a strong understanding of HDFS architecture (NameNode, DataNode) and YARN resource management (Resource Manager, Node Manager) to debug legacy systems or manage hybrid environments.
• Hive & Metastore Management: Ability to manage catalog metadata and run distributed SQL queries over your distributed storage system.

4. Workflow Orchestration

• Apache Airflow: Mastery of building, scheduling, and monitoring complex data pipelines using Python-based DAGs.
• Advanced Airflow Concepts: Utilizing TaskFlow API, custom XComs, dynamic task mapping, and setting up efficient Task Groups.
• Orchestration Integration: Knowing how to safely trigger, monitor, and pass parameters to external Spark jobs or Cloud/Databricks operators) without overloading the Airflow worker nodes.

5. Architectural & Cross-Functional Skills

• Data Lakehouse Architecture: Designing unified platforms that combine the cost-effective storage of data lakes with the data management structures of data warehouses.
• CI/CD & DataOps: Writing clean, testable Scala/Python code using unit-testing frameworks (like ScalaTest) and automating deployments using Git, Docker, and CI/CD pipelines.
• Advanced SQL: Writing complex query logic, analytical window functions, and diagnosing execution plans—even when writing Spark code, SQL remains foundational.