Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a
Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Remote Opening
Shift: Day Shift (9-hour shift, 5 days a week, weekends and holidays on rotation).
This role requires covering weekends and holidays on a rotational basis to ensure 7-day-a-week coverage.
The critical time window to be covered is 11:00 AM to 7:00 PM IST
Responsibilities:
- Team & Operational Management
- Team Management: Manage the team of 3-4 Data Engineers.
- Roster & Scheduling: Create, manage, and maintain team rosters and schedules to ensure full 8-hour coverage, 7 days a week, including rotational weekend and holiday coverage.
- Availability: Must be able to work a day shift and manage a team that provides 7-day-a-week coverage, including weekends and holidays on rotation.
- Coverage Assurance: Handle all team leave requests (planned and unplanned). You are responsible for ensuring coverage is always maintained, especially during medical or other unexpected absences.
- Performance & SLAs: Act as the primary operational point of contact, ensuring the team executes all tasks efficiently, makes sound decisions, and adheres to all agreed-upon SLAs.
- Technical Leadership & Engineering
- Final Escalation Point: Serve as the final technical escalation point for the team; personally handle the most critical job failures and system issues.
- Expert RCA: Lead all high-priority Root Cause Analysis (RCA) efforts, documenting solutions and implementing permanent fixes to prevent recurrence.
- Strategic Improvement: Drive the team's efforts in performance tuning and AWS Glue cost optimization.
- Automation Strategy: Define and lead the development of new automation jobs (using Glue, Lambda, and Step Functions) to reduce manual support tasks.
- Complex Data Management: Oversee all data integrity and synchronization tasks, including CDC reloads and write-backs to external databases like Teradata.
- Knowledge Transfer: Own the knowledge transfer process to internal resources and ensure the team maintains excellent documentation and runbooks.
Qualifications:
- Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services.
- Demonstrated expertise with a minimum of 5-8 years of relevant experience.
- Deep understanding of Data engineering, ETL principles and data warehouse design.
- Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue.
- Automation: Proven ability to design and develop automation solutions from the ground up.
- Problem-Solving: Expert-level troubleshooting and RCA skills.
- Leadership: Ability to work independently, make critical decisions, and provide technical guidance.
- Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
- Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.
- Preferred Skills: AWS Certified.
