DATA ENGINEER (SENIOR)
Project: Large-scale feature engineering for predictive analytics
Experience: 5-7 years in Data Engineering
Key Responsibilities:
- Design and implement feature engineering pipelines (50-150 features/device type)
- Build ETL workflows using Apache Airflow and PySpark
- Validate data quality (95% quality score across 8 dimensions)
- Develop Bronze Silver Gold data transformation layers
- Process 10K ServiceNow incidents/day + device telemetry streams
Must-Have Skills:
- Python (Pandas, NumPy, PySpark)
- Apache Airflow for ETL orchestration
- AWS S3, RDS PostgreSQL, data lake architectures
- Data quality frameworks (completeness, accuracy, consistency)
- SQL and data modelling
- Implementation experience of Projects having large data-sets
Nice-to-Have:
- Real-time stream processing (Kafka, MSK)
- ServiceNow data structures
- AWS Glue, EMR
