Job Description: Data Engineer (Big Data/Kafka)
We are seeking a highly experienced Senior Data Engineer with a deep background in Big Data technologies to join our team. This is a contract role for a major project in the Banking sector.
Key Details:
- Role: Data Engineer
- Industry: Banking (Financial Services)
- Work Location: Remote (India)
- Contract Duration: 6 months initial contract (with a strong likelihood of extension to long-term)
- Notice Period: 30 days or less
- Note: This role is for professionals seeking a contract engagement, not for freelancers or independent contractors.
Key Responsibilities:
- Technical Execution: Design, write, and tune complex data processing jobs using Java, MapReduce, Hive, and Spark.
- Team Leadership & Management: Lead and mentor a team of data engineers (including full-time associates and vendor partners), ensuring project prioritization aligns with business needs. Remain hands-on with code and architecture.
- Big Data Ecosystem Management: Work extensively across the Hadoop stack (HDFS, Pig, Hive, HBase, Flume, Sqoop) and NoSQL stores (e.g., Cassandra).
- Data Modeling & BI: Design and build dimensional data models. Manage the business intelligence function, developing high-quality dashboards using industry best practices.
- Data Integrity & Deployment: Analyze source data integrity and lead the deployment and auditing of data models and attributes for accuracy.
- Stream Processing: Develop and maintain solutions using stream-processing systems like Spark-Streaming.
- SQL & Performance Tuning: Demonstrate proficiency in writing and performance-tuning advanced SQL queries.
Required Skills & Qualifications:
- Experience: 6-10+ years of proven experience as a Big Data Engineer.
- Big Data Frameworks: In-depth knowledge of Hadoop (Cloudera distribution preferred), Spark, and similar frameworks.
- Programming & Scripting: Strong expertise in Java. Experience with object-oriented/object function scripting languages like Python and Scala.
- Querying Tools: Good knowledge of Big Data querying tools such as Pig, Hive, and Impala.
- Databases: Experience with both relational SQL (e.g., Postgres) and NoSQL databases (e.g., Cassandra).
- Streaming Tech: Hands-on experience with stream-processing systems like Spark-Streaming or Storm.
- Data Modeling: Proven experience in designing and building dimensional data models.
- Leadership: Demonstrated experience leading a Backend/Distributed Data Systems team while remaining technically hands-on.
Preferred Skills (Pluses):
- Experience with Azure cloud services.
- Knowledge of data science and machine learning tools and technologies.
- Experience in the Financial Services/Banking industry.
Personal Attributes:
- Able to effectively plan, organize, and own deliverables.
- Strong interpersonal and communication skills.
- Ability to solve complex networking, data, and software issues.
- A collaborative team player who builds and maintains cooperative work relationships.
