Description
We are looking for a skilled Data Engineer to design, build, and optimize scalable data platforms that enable advanced analytics, real-time decision-making, and data-driven applications.
The role focuses on modern data infrastructure, distributed systems, and high-performance data processing to support business and product needs.
Key Responsibilities
Data Platform & Architecture :
- Design and develop scalable, high-performance data platforms and data ecosystems
- Build robust data architectures supporting real-time and batch data processing use cases
- Ensure high availability, fault tolerance, and scalability of data systems
Streaming & Real-Time Processing
- Develop and maintain real-time data processing systems using technologies like Kafka, Flink, or Spark Streaming
- Build event-driven architectures for low-latency data consumption and processing
- Optimize streaming pipelines for performance and reliability
Data Storage & Management
- Design and manage large-scale distributed data storage solutions (data lakes, warehouses, lakehouses)
- Work with structured, semi-structured, and unstructured data formats
- Optimize data storage for query performance and cost efficiency
Performance Optimization
- Tune data systems for performance, scalability, and efficiency
- Optimize query performance and data access patterns
- Implement partitioning, indexing, and caching strategies
Data Quality & Governance
- Ensure data reliability, consistency, and integrity across systems
- Implement validation, monitoring, and observability for data systems
- Support data governance, security, and compliance requirements
Cloud & Infrastructure
- Build and manage data platforms on cloud environments (AWS, Azure, GCP)
- Utilize cloud-native services for storage, processing, and orchestration
- Automate infrastructure using Infrastructure as Code (IaC) tools
Collaboration & Integration
- Work closely with data scientists, analysts, and product teams to enable data access and usability
- Integrate data platforms with downstream applications, APIs, and analytics tools
- Provide technical guidance and best practices to stakeholders
Required Skills & Qualifications
- 6 to 9 years of experience in Data Engineering or Backend/Data Platform Engineering
- Strong programming skills in Python, Scala, or Java
- Hands-on experience with distributed data processing frameworks (Apache Spark, Flink, etc.)
- Experience with streaming technologies (Kafka or equivalent)
- Strong understanding of distributed systems and data architecture principles
- Proficiency in SQL and data modeling techniques
- Experience with cloud platforms (AWS, Azure, or GCP)
- Familiarity with containerization and orchestration tools (Docker, Kubernetes)
Preferred Qualifications
- Experience with lakehouse architectures (Delta Lake, Iceberg, Hudi)
- Exposure to data mesh or domain-oriented data architectures
- Experience with query engines like Presto, Trino, or Snowflake
- Knowledge of security best practices in data systems
- Experience with monitoring tools (Prometheus, Grafana, etc.)
Key Competencies
- Strong problem-solving and system design skills
- Performance optimization mindset
- Collaboration and stakeholder management
- Attention to detail and data accuracy
- Ability to work in fast-paced environments
(ref:hirist.tech)
