V
Architect
Job Description
The Senior Data Engineer is responsible for the design, development, administration, governance, and operational support of enterprise data platforms built on Databricks and Microsoft Fabric. This role combines deep data engineering expertise with platform administration capabilities to ensure scalable, secure, reliable, and cost-effective data solutions that support business operations, analytics, AI initiatives, and project delivery.
Own end-to-end technical design, development, and delivery of enterprise data engineering solutions.
Design and implement scalable ETL/ELT, data integration, and data transformation pipelines for structured and unstructured data sources.
Develop and maintain batch, near real-time, and streaming data ingestion pipelines from databases, APIs, files, cloud storage, and enterprise applications.
Build reusable notebooks, data transformations, and processing frameworks using Databricks, Microsoft Fabric, Python, PySpark, and SQL.
Create trusted datasets, curated data products, and semantic-ready data structures supporting analytics, reporting, AI, and machine learning initiatives.
Primary Skill
Databricks
Microsoft Fabric
Python
SQL
Secondary Skill
OneLake
Spark
Own end-to-end technical design, development, and delivery of enterprise data engineering solutions.
Design and implement scalable ETL/ELT, data integration, and data transformation pipelines for structured and unstructured data sources.
Develop and maintain batch, near real-time, and streaming data ingestion pipelines from databases, APIs, files, cloud storage, and enterprise applications.
Build reusable notebooks, data transformations, and processing frameworks using Databricks, Microsoft Fabric, Python, PySpark, and SQL.
Create trusted datasets, curated data products, and semantic-ready data structures supporting analytics, reporting, AI, and machine learning initiatives.
Primary Skill
Databricks
Microsoft Fabric
Python
SQL
Secondary Skill
OneLake
Spark
More Info
Key Skills
About Company
Skills:
.NET, telemetry , snowflake , S3, Distributed Systems, Automated Testing, Pyspark, Apache Spark, Containers, Git, Advanced Sql, Rest Apis, Python, AWS, Infrastructure as Code, batch integrations, HPS, Production Troubleshooting, ASP.NET Core, CI/CD, Experian PowerCurve Collections, PCC Cards Modernisation, Observability
Skills:
Large Language Models (LLMs), Apis, Microservices, Docker, Terraform, Databricks, Python, Kubernetes, Azure AI ML ecosystem, LangChain, Generative AI, MLflow, CI CD, AI ML Architecture, RAG, Event-driven architectures
Skills:
cda , Ccda, Api Integration, Ms Sql Server, Json, Xml, Rest Apis, Azure Integration Services, FHIR R4, Data Mapping Transformation, ccd, NextGen Connect, Mirth Connect, HL7 v2.x, EMR EHR Integration, Healthcare Integration Interoperability
Skills:
Keras, Deep Learning, Tensorflow, Machine Learning, Pytorch, Python, Predictive Analytics, AWS AI services, AI Infrastructure, LLM technologies, scikit-learn, Google Vertex Gen AI, Azure OpenAI, Agentic frameworks, ML Ops, Spacy, Generative AI, cloud-based AI services
Skills:
snowflake , Tableau, Databricks, Sql, Tensorflow, Jira, Hadoop, Power Bi, Prince2, Pytorch, Gcp, Confluence, XGBoost, Azure ML, Agile, Apache Spark, Togaf, AWS, Python, Azure, OpenAI, Azure OpenAI, LangChain, Pmp, CRISP-DM, Vertex AI, Kubeflow, MLflow, Safe, MS Project, Scikit-learn, Amazon SageMaker, R