- Develop & Optimize Data Pipelines Architect, build, and enhance scalable data pipelines for high-performance processing.
- Troubleshoot & Sustain Identify, diagnose, and resolve data pipeline issues to ensure operational efficiency.
- Data Architecture & Storage Design efficient data storage and retrieval strategies using Postgres, Redshift, and other databases.
- CI/CD Pipeline Management Implement and maintain continuous integration and deployment strategies for smooth workflow automation.
- Scalability & Performance Tuning Ensure the robustness of data solutions while optimizing performance at scale.
- Collaboration & Leadership Work closely with cross-functional teams to ensure seamless data flow and lead engineering best practices.
- Security & Reliability Establish governance protocols and ensure data integrity across all pipelines.
Technical Skills Required:
- Programming: Expert in Python and Scala
- Big Data Technologies: Proficient in Spark, Kafka
- DevOps & Cloud Infrastructure: Strong understanding of Kubernetes
- SQL & Database Management: Skilled in SQL administration, Postgres, Redshift
- CI/CD Implementation: Experience in automating deployment processes for efficient workflow
