Key Responsibilities:
Data Architecture & Solution Development
- Translate business and functional requirements into robust, scalable data solutions aligned with overall architecture.
- Design and implement data integrations and data quality frameworks.
- Participate in full development lifecycle including design, implementation, testing, documentation, delivery, support, and maintenance.
Pipeline & API Development
- Develop and maintain scalable data pipelines.
- Build new API integrations to support data flow and analytics requirements.
- Implement large-scale, high-volume, low-latency applications for efficient data processing.
Big Data & ETL Development
- Design, develop, and maintain ETL pipelines.
- Apply dimension modeling and data warehousing best practices.
- Write complex SQL queries for data transformation, extraction, and validation.
Streaming & Advanced Processing
- Build stream processing platforms using Kafka and Spark Streaming.
- Optimize data pipelines for performance and scalability.
- Independently deliver end-to-end projects with minimal supervision.
Required Skills & Experience:
- 5+ years of development experience in big data using Spark.
- Strong programming skills in Python.
- Expertise in SQL and complex query writing.
- Experience with dimension modeling, data warehousing, and ETL pipelines.
- Knowledge of streaming platforms like Kafka and Spark Streaming.
Nice to Have:
- Experience with job orchestration frameworks like Airflow, Oozie, or Luigi.
- Familiarity with AWS services such as S3, EMR, RDS.
- Understanding of cloud data warehouses like Snowflake.
- Knowledge of SQL distribution engines like Presto, Druid.
- Familiarity with streaming frameworks like Flink.
- Exposure to NoSQL databases such as HBase or Cassandra.
