We are building a scalable data collection, storage, and distribution platform to consolidate data from vendors, research providers, exchanges, prime brokers, and web scraping sources. The platform will serve systematic & fundamental portfolio managers (PMs), as well as enterprise teams including Ops, Risk, Trading, and Compliance. The role involves developing internal data products and analytics while optimizing data pipelines.
Key Responsibilities:
- Web Scraping & Data Acquisition
- Utilize scripts, APIs, and web scraping tools to collect market data.
- Data Engineering & Pipeline Development
- Build and maintain a greenfield data platform on Snowflake and AWS.
- Enhance and optimize existing data pipelines to meet new business needs.
- Onboard and integrate new data providers efficiently.
- Data Storage & Migration
- Handle data migration projects ensuring seamless transition and accuracy.
- Implement efficient storage solutions to support analytical workloads.
- DevOps & Cloud Infrastructure
- Work with Docker, Kubernetes, and Jenkins for scalable deployment.
- Optimize cloud infrastructure on AWS for performance and cost-effectiveness.
Mandatory Skills & Experience:
- 10+ years of experience as a Data Engineer.
- Strong SQL & Python programming skills.
- Proficiency in Linux environments.
- Experience with AWS-based data solutions.
- Containerization expertise (Docker, Kubernetes).
- DevOps knowledge (Jenkins, CI/CD pipelines).
- Excellent communication and problem-solving skills.
Nice-to-Have Skills:
- Experience with Market Data / Capital Markets projects.
- Knowledge of Apache Airflow for workflow automation.
