Dynamic Yield, a Mastercard company, is seeking a Data Engineer II to join our Data Engineering & Analytics team. We're focused on developing robust data and analytics solutions from vast datasets collected across various consumer-focused businesses.
In this role, you'll be instrumental in creating high-performance algorithms, leveraging cutting-edge analytical techniques, and designing intuitive workflows. Your work will empower users to derive actionable insights from big data. You'll work with large-scale datasets and front-end visualizations to unlock the true value of big data, supporting business needs through innovative, data-driven solutions.
The Role
As a Data Engineer II, you will:
- Platform Evolution: Drive the evolution of data and services platforms with a strong emphasis on data engineering and data science, ensuring impactful advancements in data quality, scalability, and efficiency.
- Data Generation & Curation: Develop and fine-tune methods and algorithms to generate precise, high-quality data at scale. This includes creating and maintaining feature stores, analytical stores, and curated datasets to enhance data integrity and usability.
- Complex Problem Solving: Solve complex data challenges involving multi-layered datasets and optimize the performance of existing data pipelines, libraries, and frameworks.
- Support & Resolution: Provide support for deployed data applications and analytical models, identifying data issues and guiding resolutions.
- Data Governance: Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, and other relevant measures.
- Data Integration: Integrate diverse data sources, including real-time, streaming, batch, and API-based data, to enrich platform insights and drive data-driven decision-making.
- Tool Experimentation: Experiment with new tools to streamline the development, testing, deployment, and running of our data pipelines.
- Best Practices: Develop and enforce best practices for data engineering, including coding standards, code reviews, and documentation.
- Security & Privacy: Ensure data security and privacy compliance, implementing robust measures to protect sensitive data.
- Global Collaboration: Communicate, collaborate, and work effectively within a global environment.
All About You
- Education: Bachelor's degree in Computer Science, Software Engineering, or a related field.
- Data Engineering Experience: Extensive hands-on experience in Data Engineering, including implementing multiple end-to-end data warehouse projects in Big Data environments.
- Programming & Frameworks: Proficiency in application development frameworks (Python, Java/Scala) and data processing/storage frameworks (Hadoop, Spark, Kafka).
- Orchestration Tools: Experience in developing data orchestration workflows using tools such as Apache NiFi, Apache Airflow, or similar platforms to automate and streamline data pipelines.
- Performance Tuning: Experience with performance tuning of database schemas, databases, SQL, ETL jobs, and related scripts.
- Agile Environment: Experience working in Agile teams.
- Data-Driven Application Development: Experience in developing data-driven applications and data processing workflows/pipelines, and/or implementing machine learning systems at scale using Java, Scala, or Python. This includes all phases such as data ingestion, feature engineering, modeling, tuning, evaluating, monitoring, and presenting analytics.
- Cloud Integration: Experience in developing integrated cloud applications with services like Azure, Databricks, AWS, or GCP.
- Analytical Skills: Excellent analytical and problem-solving skills, with the ability to analyze complex data issues and develop practical solutions.
- Communication & Collaboration: Strong communication and interpersonal skills, with the ability to collaborate effectively with and facilitate activities across cross-functional, geographically distributed teams and stakeholders.
