We are looking for an experienced Data Engineer with strong expertise in real-time data streaming and distributed data processing technologies. The ideal candidate should have hands-on experience in building scalable, event-driven data platforms using Apache Flink, Kafka, Java/Scala, and PySpark. The candidate will be responsible for designing, developing, and maintaining high-performance data engineering solutions within large-scale enterprise environments.
Requirements
Key Responsibilities:
- Design, develop, and maintain real-time streaming data pipelines and event-driven architectures
- Build scalable and fault-tolerant data engineering solutions using Apache Flink, Kafka, and Java/Scala
- Develop and optimize distributed data processing applications for enterprise-grade systems
- Implement best practices for performance tuning, resiliency, security, and compliance requirements
- Collaborate with cross-functional teams to design technical and application architectures
- Ensure production-grade monitoring, troubleshooting, and operational excellence of distributed services
- Contribute to CI/CD implementation and DevOps best practices
- Work closely with engineering teams to drive technical improvements and architectural decisions
Mandatory Technical Skills:
- Minimum 4+ years of development and design experience in:
- Apache Flink
- Java or Scala
- Apache Kafka
- PySpark
- Real-time data streaming technologies
- Event-driven architectures
- Hands-on experience with:
- Apache Flink (Beam or Spark Streaming experience is also valuable)
- Kafka ecosystem and streaming platforms
- JVM tuning and performance optimization
- Distributed systems design and implementation
- Docker and Kubernetes containerization
- Linux OS administration and Shell scripting
- SQL and NoSQL databases
- CI/CD tools such as GitHub and Jenkins
- Design patterns and their implementation
- Production monitoring and troubleshooting of distributed services
Nice to Have Skills:
- Redis or other caching technologies
- Experience with Spark Streaming or Apache Beam
- Experience working with banking or fintech platforms
Data Engineering & Security Requirements:
- Strong understanding of data security principles and controls
- Experience implementing secure data transfer mechanisms including:
- CRON jobs
- ETL processes
- JDBC and ODBC scripts
- Knowledge of encryption, anonymization, data integrity, and policy controls in large-scale infrastructures
- Understanding of data sensitivity requirements, including:
- Protection of PII data
- Secure logging practices
- Secure in-memory data handling
- Ability to identify security design gaps and recommend enhancements
- Experience implementing wrapper solutions for legacy or third-party components to ensure compliance requirements are met
Infrastructure & Networking Knowledge:
- Working knowledge of:
- DNS
- Proxy servers
- ACLs
- Networking policies
- Troubleshooting network-related issues
Functional Requirements:
- Experience working in Agile environments
- Ability to design and implement scalable technical architectures
- Conduct technology research and benchmarking activities
- Ability to influence engineering teams on technical best practices
- Experience working in enterprise-scale environments
- Banking, Financial Services, or FinTech domain experience is preferred
Soft Skills:
- Excellent communication and interpersonal skills
- Strong problem-solving and analytical abilities
- Self-driven and capable of working independently
- Ability to collaborate effectively with cross-functional teams and stakeholders
- Strong presentation and listening skills
- Positive attitude and ability to foster a collaborative team environment
- Passionate about engineering excellence and continuous improvement
