Role Overview
We are looking for a Data Engineer to support and enhance the CS-AT platform, which processes large-scale machine log data (15–20 years) to enable predictive and proactive maintenance solutions. The role involves building and optimizing data pipelines, log processing systems, and analytics platforms on Azure.
Key Responsibilities
- Design, build, and maintain scalable data pipelines
- Handle ingestion and processing of large-scale log data
- Monitor and optimize pipeline performance and reliability
- Work on log parsing and pattern extraction
- Collaborate with data scientists on predictive and proactive maintenance use cases
- Develop and optimize queries using KQL (Kusto Query Language)
- Build and support dashboards using Azure Data Explorer
- Ensure data quality, monitoring, and operational stability
- Support integration with downstream systems such as OneAI and MI Log Interpreter
Required Skills
Core Skills
- Strong experience in data engineering (5–6 years)
- Expertise in writing KQL
- Expertise in building data pipelines and ETL/ELT processes
- Experience with log data processing and analysis
- Python and C# language skills
Azure Technologies
- Azure Functions
- Azure Event Grid
- Azure Data Explorer (Kusto)+KQL
- Azure Data Factory
- Databricks/Spark
- Delta Lake
Data & Engineering Skills
- Pipeline monitoring and optimization
- Performance tuning of large data systems
- Handling high-volume historical datasets
- Experience in distributed data processing
- Documentation according to CS standards
Good to Have
- Experience with predictive maintenance use cases
- Exposure to machine logs / IoT / telemetry data
- Understanding of data science workflows
Soft Skills
- Strong problem-solving ability
- Data Product Ownership mindset (Development + Operations)
- Ability to work in cross-functional teams
- Proactive approach to optimization and operations
Team Structure
Part of a (4 – 6) member cross-functional team including Development and Operations.
Key Success Metrics
- Stable and optimized data pipelines
- Improved processing efficiency
- Reliable log ingestion and analysis
- Smooth integration with AI systems
