We are looking for a Senior Data Engineer with expertise in SQL, Python, AWS, and containerization to build and maintain a scalable data platform. The role involves working with web scraping, data pipelines, and DevOps practices while collaborating with cross-functional teams to onboard new data providers and support data migration projects.
Responsibilities:
- Perform web scraping using scripts, APIs, and tools.
- Build and maintain a greenfield data platform on Snowflake and AWS.
- Understand existing data pipelines and enhance them for new business requirements.
- Manage onboarding of new data providers and execute data migration projects.
Required Skills & Experience:
- 10+ years of experience as a Data Engineer.
- Proficiency in SQL and Python for data processing and automation.
- Strong experience working in Linux environments.
- Hands-on experience with containerization technologies: Docker, Kubernetes.
- Expertise in AWS cloud services.
- Strong DevOps knowledge: Kubernetes, Docker, Jenkins.
- Excellent communication skills and ability to collaborate across teams.
Nice to Have:
- Experience in market data projects or capital markets.
- Hands-on experience with Snowflake (big plus).
- Knowledge of Airflow for workflow orchestration.
