Senior Data Engineer

We are hiring enthusiastic professionals who have zeal to turn data into information, information into insight and insight into business decisions also strong expertise in cloud-based data platforms and AI/ML technologies. Data Engineer would be responsible for developing and creating batch and data pipelines for ETL/ELT as per the User requirement. Data Engineer shall be responsible for leading the database related efforts (primarily MS SQL DWH and Redshift ) in the engagement which will include data warehousing, ETL, data modeling, database integration/migration, database development and database management efforts for the engagements.

Experience: 5 to 7 years.

Job Responsibilities:

• Architect, build, and maintain scalable, secure ETL/ELT pipelines for batch.
• Code, test, and deploy batch and data pipelines using PySpark, SQL, and Python.
• Developing scalable and re-usable, self-service frameworks for data ingestion and processing.
• Integrate AI/ML workflows into data engineering pipelines.
• Prepare, cleanse, and transform large datasets for machine learning models.
• Managing, implementation and regular monitoring of ETL activities right from Source systems to Power BI Layer using Azure and AWS cloud services
• Design distributed data lakes and data warehousing solutions using native cloud environments
• Track automated pipeline health in AWS Glue, Azure Data Factory, or Apache Airflow to fix failures. Coordinate with user group for issues/requirements
• Design scalable star and snowflake schemas for cloud data warehouses like AWS Redshift and Azure Synapse.
• Support for the existing project
• Bringing best practices in following areas: Design & Analysis, Automation (Pipelining etc.), Testing, Monitoring and Documentation.

Preferred Background:

• Proficiency in Python or Scala for data processing.
• Strong expertise in cloud-based data platforms and AI/ML technologies.
• Good knowledge of Apache Spark, PySpark, and Hadoop, Familiar with AWS Services like Glue, S3, Athena etc.
• Machine Learning fundamentals and Generative AI applications.
• Azure services like ADF, ADLS, Synapse Analytics, Databricks.
• Power BI is optional.