We are actively seeking a skilled Big Data Engineer to join our client's dynamic team through Acme Services. This pivotal role requires solid experience in ETL / Data Engineering / Data Warehousing principles. The ideal candidate will possess strong proficiency in Python, PySpark, and SQL or other ETL tools, along with hands-on experience in various database and big data technologies such as MySQL, SQL Server, MongoDB, Hadoop, Hive, Pig, and Spark. Experience with orchestration tools like AirFlow and container platforms like OpenShift would be a significant advantage.
Key Responsibilities
- Data Pipeline Development: Design, build, and maintain robust and scalable ETL (Extract, Transform, Load) pipelines to ingest, process, and transform large datasets from various sources into data warehouses or data lakes.
- Big Data Technologies: Work extensively with Big Data platforms including Hadoop, Hive, Pig, and Spark, leveraging their capabilities for data processing and analytics.
- Programming & Scripting: Utilize Python, PySpark, and SQL for data manipulation, scripting, and developing data engineering solutions.
- Database Management: Interact with and manage data in various database systems such as MySQL, SQL Server, and MongoDB.
- Data Warehousing: Contribute to the design, implementation, and optimization of Data Warehouses, ensuring data quality and accessibility for business intelligence and reporting.
- Orchestration & Automation (Advantage): (If applicable) Use AirFlow for scheduling, monitoring, and orchestrating complex data workflows.
- Containerization (Advantage): (If applicable) Leverage OpenShift for deploying and managing data applications in a containerized environment.
- Troubleshooting & Optimization: Identify and resolve data-related issues, and optimize data processes for performance and efficiency.
Skills
- Solid experience in ETL / Data Engineering / Data Warehouse.
- Strong proficiency in Python, PySpark, and SQL or other ETL tools.
- Hands-on experience with databases including MySQL, SQL Server, MongoDB.
- Expertise in Hadoop, Hive, Pig, and Spark.
- Experience in AirFlow (big advantage).
- Experience in OpenShift (big advantage).
- Strong analytical and problem-solving skills.
- Ability to work with large, complex datasets.
Qualifications
- Proven professional experience as a Big Data Engineer or in a similar data-focused role.
- Demonstrable experience in building and managing data pipelines.
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
