V
Databricks
V
Databricks
Virtusa- Posted an hour ago
- Be among the first 10 applicants
Job Description
⃣ Sr. Data Engineers (2) – Databricks Pipeline Developers
Role Overview
Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.
Key Responsibilities
Role Overview
Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.
Key Responsibilities
- Develop PySpark pipelines for
- API ingestion
- FTP file ingestion
- Data cleansing
- Enrichment transformations
- Implement incremental & batch loads
- Develop Bronze → Silver → Gold transformations
- Translate legacy SQL stored procedures into Spark jobs
- Optimize Delta Lake tables
- Implement validation checks
- Support testing & reconciliation
- Strong PySpark / Spark SQL
- Strong SQL (read & interpret legacy procedures)
- Experience with API integration
- Experience handling flat files (FTP ingestion)
- Experience with Delta Lake
- Knowledge of partitioning & optimization
More Info
Key Skills
About Company
Skills:
Gcp, Apache Spark, Kafka, Databricks, Azure, AWS, Spark Structured Streaming, Delta Lake, Unity Catalog
Skills:
Pyspark, ELT, Software Development Life Cycle, Terraform, Docker, Python, AWS, Cloudformation, Sql, MLops, Spark, Databricks, Kubernetes, Etl, Airflow, MLflow, IaC, AI/ML, AI-assisted software development tools, Structured Streaming, LLM GenAI-enabled applications, Unity Catalog, Model serving, Delta Lake, data models, Feature stores
Skills:
Spark, Sql, Databricks, AWS, AWS CloudFormation, Python, Terraform, Delta Lake
Delivery Lead – Data Engineering & Databricks - Assistant Vice PresidentCurrenex State Street Trust Company
C
Skills:
Spark, Sql, Databricks, Python, cloud data engineering patterns, Delta Lake
Skills:
snowflake , Data Modelling, Unix Commands, Perl, Informatica, Python, Sql, pipeline design, Stakeholder Management