Search by job, company or skills

Azure Data Engineer

Quick Apply
  • Posted 2 hours ago
  • Over 100 applicants have applied

Job Description

  • Designing and building data pipelines to collect, process, transform, and store data from various sources. This involves understanding the data flow, selecting the appropriate technologies, and creating scalable and efficient data processing workflows.
  • Ensure that the data pipeline architecture is reliable and produces high-quality data. This involves monitoring the data pipeline, identifying and resolving issues, and implementing data validation and testing mechanisms.
  • Work closely with data scientists, data analysts, software engineers, and other stakeholders to understand their data requirements and ensure that the data pipeline architecture meets their needs.
  • Optimizing the data pipeline architecture to ensure efficient and scalable data processing. This involves identifying bottlenecks, optimizing data processing workflows, and selecting appropriate technologies and tools.
  • Ensure security and privacy of data in the pipeline architecture. This involves implementing appropriate security measures, such as data encryption, access controls, and auditing.
  • Documentation of the data pipeline architecture, data processing workflows, and data sources. This involves creating technical documentation, data flow diagrams, and reports that communicate the performance and quality of the data pipeline to stakeholders.
  • Qualification and Experience:
  • Qualification:
  • A bachelor's in computer science, data science, software engineering, or a related field
  • Experience:
  • 5+ years of relevant experience in data engineering, software development, or data architecture, in a cloud-based environment.
  • Technical skills in data engineering, software development, and data architecture. This includes experience with big data technologies.
  • Proficiency in one or more programming languages such as Python, Java, or Scala, as well as experience with scripting languages like Bash or PowerShell.
  • Data management skills, data modelling, data governance, and data security. Analytical skills and experience with data analysis, data profiling, and data quality assessment.
  • Project management methodologies, such as Agile or Scrum, and be able to manage projects independently.

More Info

Job Type:
Industry:
Function:
Employment Type:

Job ID: 105659337

Similar Jobs

Pune

Skills:

.NET and SSASAzure SqlData WarehousingData ModelingAzure Data LakeData PipelineSpark StreamingAzure FunctionsSQL ServerPythonETL Processes

Early Applicant
Pune, India

Skills:

virtual environment PytestAzure Synapse AnalyticsT-sqlTensorflowHadoopPysparkPandasAzure DatabricksHiveMatplotlibAzure DevOpsKerasSQL ServerAzure Data FactoryPostgresHBaseNumpyData ModellingNltkOraclePythonOracle Pl SqlPycharmAzure Data LakeIDLEAzure SQL DatabasesScikit-learnJupyterAzure AD IntegrationrequestsConda

Mumbai, Mumbai City, Navi Mumbai

Skills:

AdfAzure Data FactorySsisAZURE DATA ENGINEER

Pune, India

Skills:

SqlAzure DatabricksAzure Data FactoryPythonAzure Data LakeGitPysparkData ModelingPerformance Optimization

Pune, India

Skills:

Azure SynapseAzure Data FactoryCloud ArchitectureData WarehousingPythonSqlEtlADLS Gen2

Beware of Scammers

We don’t charge money for job offers