We at Coforge are hiring for a detail-oriented Data Engineer (Apache Flume) to join our team. The ideal candidate will have 4+ years of hands-on experience in development, with strong expertise in Apache Flume, data modeling, and migration. This role involves leading the migration of existing logs from SQL Server databases to an Apache Flume application and establishing a robust pipeline for daily log ingestion.
Key Responsibilities
- Log Migration:
- Design and implement solutions to extract and transfer historical logs from SQL Server to Apache Flume.
- Ensure data integrity, completeness, and consistency during migration.
- Daily Log Ingestion:
- Develop and automate pipelines to move newly generated logs from SQL Server to Flume daily.
- Monitor and maintain pipeline performance and reliability.
- System Integration & Optimization:
- Collaborate with infrastructure and application teams for seamless integration.
- Optimize data flow and troubleshoot issues in real-time.
- Documentation & Reporting:
- Maintain detailed documentation of architecture, processes, and configurations.
- Provide regular updates and reports to stakeholders.
Required Skills & Qualifications
- 3+ years of experience in data engineering or ETL development.
- Strong expertise in SQL Server and Apache Flume.
- Proficiency in scripting languages (e.g., Python, Shell) for automation.
- Experience with log management, data pipelines, and system integration.
- Familiarity with big data ecosystems (e.g., Hadoop, Kafka) is a plus.
- Excellent problem-solving and communication skills.
Share your resume over [Confidential Information]
