Senior Data Engineer | 10-12 Years Experience | Must Have: Proven expertise in building scalable batch and streaming data pipelines using Databricks (PySpark) and Snowflake.
Lead the design, implementation, and optimization of application data stores using PostgreSQL, DynamoDB, and advanced SQL.
Strong programming skills in SQL, Python, and PySpark for efficient data processing.
Work with structured (CSV, SQL), semi-structured (JSON, XML), and unstructured (PDF, logs) data formats.
Deep knowledge of Azure and AWS data ecosystems, including Microsoft Fabric and distributed computing frameworks.
Design low-latency, high-throughput pipelines using Spark and cloud-native tools.
Implement CI/CD, automation, schema versioning, and enforce data security in production environments.
Ensure adherence to best practices for performance, scalability, and compliance.
