Data Engineer

Basic Job Functions:

We are seeking an experiencedData Engineerwith a minimum of 5 years of experience, specializing inDatabricks,Python,SQL, SSIS, SAS,IoT,Azure Data Lake Services, andDataOps. The ideal candidate will have a strong background in building and maintaining scalable data pipelines, managing cloud-based data platforms, and collaborating with cross-functional teams to deliver high-quality data solutions..

Education/Experience:

Preferred Qualifications

a.Bachelor's or Master's degree in Computer Science, Engineering, or related fields.

b.Certifications such as Azure Data Engineer Associate or Databricks Certified Professional are highly desirable.

c.Familiarity with containerization technologies like Docker or Kubernetes is a plus.

d.Familiarity with agile methodologies for product development.

Key Technologies

a.Databricsk, SQL, Python, Azure Synapse, Kafka/MQTT, ETL Pltforms

b.IoT Platforms (e.g., AWS IoT Core, Azure IoT)

c.Familiarity Machine Learning frameworks (e.g., TensorFlow, PyTorch)

d.Cloud Computing

e.Predictive Analytics

f.Data Security Protocols

Required Skills/Competencies:

Databricks Expertise: 3+ years of hands-on experience with Databricks for building scalable data solutions using Spark SQL, Delta Lake, and other Databricks utilities.

Programming Skills: Proficiency in Python for scripting and automation. Experience with SAS for statistical analysis is a plus.

Azure Services: Strong experience working with Azure Data Lake Storage (ADLS), Azure Databricks, and other related Azure cloud services.

IoT Integration: Experience in handling IoT data ingestion pipelines from various devices into cloud storage systems.

DataOps Practices: Familiarity with DataOps methodologies for automating data pipeline deployments using CI/CD tools such as Jenkins or Azure DevOps.

SQL & Big Data Technologies: Strong knowledge of SQL for querying large datasets. Experience with distributed computing frameworks like Apache Spark is essential.

Cloud Platforms: Hands-on experience with Azure cloud infrastructure. Knowledge of AWS or GCP is beneficial but not mandatory.

Essential Responsibilities:

Data Pipeline Development: Design, build, and maintain robust and scalable data pipelines using Databricks, Python, and Azure Data Lake services.

Data Integration: Ingest, process, and integrate data from various IoT devices and external sources into Azure Data Lake for further analysis.

ETL/ELT Processes: Develop efficient ETL/ELT processes to transform raw data into structured formats suitable for analytics and machine learning models.

DataOps Automation: Implement DataOps practices to automate the deployment, monitoring, and management of data pipelines. Ensure continuous integration (CI) and continuous deployment (CD) of data workflows.

Data Quality & Validation: Ensure the quality, consistency, and reliability of data through validation checks and testing frameworks.

Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand requirements and deliver optimized data solutions.

Cloud Platform Management: Manage cloud resources on Azure, including Azure Data Lake Storage, Databricks clusters, and related services.

Performance Optimization: Optimize the performance of data pipelines for large-scale IoT datasets using Apache Spark on Databricks.

Documentation & Reporting: Maintain comprehensive documentation of data pipeline architecture, processes, and governance policies. Provide regular reports on pipeline performance and data quality.