We are looking for a Databricks Data Engineer to support enterprise-scale data quality, governance, and cloud data platform initiatives. This role focuses on hands-on Databricks engineering, data quality automation (Databuck), and metadata/governance integrations (Collibra, AWS). The ideal candidate will work closely with platform, governance, and analytics teams to build, optimize, and operationalize scalable data solutions, including transitioning proofs of concept into production-ready implementations.
Key Responsibilities- Databricks & Data Engineering.
- Design, build, and optimize scalable data pipelines using Databricks and Apache Spark.
- Ensure performance, reliability, and cost efficiency across development, sub-production, and production environments.
- Lead and support platform upgrades and maintenance activities, ensuring stability and backward compatibility.
- Data Quality & Observability.
- Implement and manage enterprise data quality frameworks using Databuck or similar tools.
- Define, automate, and monitor data quality rules aligned with enterprise governance standards.
- Drive improvements in data observability and reliability across data platforms.
- Data Governance & Metadata Integration.
- Build and support integrations between AWS data platforms (e.g., S3) and governance tools such as Collibra.
- Enable metadata management, data lineage, and governance visibility across enterprise data assets.
- Collaborate with governance teams to define scalable integration patterns.
- AI / Automation.
- Drive AI/GenAI-led PoCs to automate data quality, governance, and documentation processes.
- Support the transition of PoCs into scalable, production-ready solutions.
- Cloud & Platform Integration.
- Work within AWS-based architectures, integrating Databricks with cloud-native services.
- Define and document integration patterns for enterprise data platforms and downstream systems.
- Ensure security, access control, and compliance across data pipelines.
Required Qualifications
- 6+ years of experience in data engineering or data platform development.
- Strong hands-on experience with Databricks and Apache Spark.
- Experience with data quality platforms (Databuck preferred or equivalent).
- Solid experience working with AWS services, especially S3.
- Experience with data governance and metadata tools (e.g., Collibra).
- Proficiency in Python and SQL.
- Experience supporting production-grade data platforms.
Preferred Qualifications
- Experience with AI/ML or GenAI use cases in data engineering or governance.
- Familiarity with enterprise data governance frameworks.
- Experience in large enterprise or regulated environments.
- Knowledge of CI/CD, Infrastructure as Code, or DevOps practices.
Benefits
- Health Insurance, Accident Insurance.
- The salary will be determined based on several factors, including, but not limited to, location, relevant education, qualifications, experience, technical skills, and business needs.
Additional Responsibilities
- Participate in OP monthly team meetings, and participate in team-building efforts.
- Contribute to OP technical discussions, peer reviews, etc.
- Contribute content and collaborate via the OP-Wiki/Knowledge Base.
- Provide status reports to OP Account Management as requested.
About Us
At OP, we help you harness the power of technology for maximum impact. A technology consulting and solutions company, we offer advisory and managed services, innovative platforms, and staffing solutions across a wide range of fields including AI, cybersecurity, enterprise architecture, and beyond. For nearly two decades, we've been challenging the status quo of the consulting industry, serving up fresh, ingenious thinking through a radically lean structure. Together, this strategy delivers unprecedented performance at an unparalleled pace for faster results that propel your business forward.
