Title: Staff Data Engineer (Lead Data Engineer – D3)
Location: Bangalore
Experience:
- 7-10 years of hands-on experience in Data Engineering, with at least 3 years in a lead or technical ownership role.
- Strong experience with Databricks Unity Catalog, Delta Lake, Spark optimization, and governance controls.
- Experience with ClickHouse for analytical workloads and ingestion optimization.
- Prior experience in fintech or financial services.
About the Role:
We are looking for a highly experienced Staff Data Engineer to lead the evolution, reliability, and governance of the Data Platform. This is a senior individual contributor role with strong architectural influence across the organization.
You will own critical technical decisions that shape how data moves from operational systems into trusted business intelligence and analytics platforms. The role demands deep expertise in modern data engineering practices, distributed systems, cloud-native orchestration, governance, and platform reliability within a regulated fintech environment.
You will work closely with platform engineering, analytics, compliance, legal, and product teams to ensure our data ecosystem is scalable, secure, observable, and compliant with evolving regulatory requirements including the SEBI CSCRF 2024 and DPDP Act 2023.
This role is ideal for someone who combines hands-on engineering depth with platform thinking, operational maturity, and strong technical leadership.
Key Responsibilities:
Platform Architecture & Technical Leadership
- Drive the end-to-end architecture of the data platform in collaboration with senior platform engineers and data architects.
- Design scalable ingestion, transformation, and delivery pipelines across:
- DBT Medallion architecture
- Databricks Lakehouse
- Airflow orchestration
- BI platforms including Metabase, Tableau, Redash, and Databricks Lakeview.
- Define and document technical standards for:
- Schema evolution
- Incremental modeling
- Cluster strategies
- Pipeline reliability
- Data quality frameworks.
- Lead the completion of the Databricks Unity Catalog migration, including:
- Schema governance
- Access policies
- Catalog standardization
- Stakeholder communication.
- Define long-term orchestration strategy with Airflow on Kubernetes as the primary orchestration layer.
- Evaluate emerging tools and frameworks to improve developer productivity and operational efficiency.
- Participate in lightweight MDM (Master Data Management) evaluations and architecture decisions.
Pipeline Reliability & Production Ownership
- Own production SLAs for critical data pipelines and reporting systems.
- Design pipelines with:
- Idempotency
- Re-runnability
- Safe backfill capabilities
- Fault-tolerant processing.
- Build and maintain observability frameworks using:
- Prometheus
- Grafana
- Graylog
- Structured audit logging.
- Lead incident response for data SLA failures and drive root-cause resolution.
- Conduct post-mortems and implement preventive remediation measures.
Data Quality, Governance & Compliance
- Architect and implement scalable data validation and anomaly detection frameworks.
- Establish governance standards for:
- RBAC/ABAC
- Service account access
- Data lineage
- Auditability.
- Enforce PII masking and protection standards, including:
- SHA-256 with salt for PAN/customer identifiers
- Migration away from legacy MD5-based masking approaches.
- Ensure all pipelines emit structured, queryable audit events with 5-year retention standards.
- Translate regulatory requirements from SEBI CSCRF and DPDP into enforceable platform-level controls.
Engineering Excellence & Mentorship
- Define and enforce engineering standards across the data platform team.
- Review architecture proposals, pipeline PRs, and schema changes.
- Establish documentation and operational playbooks for platform reliability and on-call readiness.
- Mentor engineers on:
- DAG design
- Data modeling
- DBT best practices
- Warehouse optimization
- Fintech data domain concepts.
Cross-Functional Collaboration
- Partner with backend and product engineering teams to ensure data instrumentation is built into products from day one.
- Work closely with analytics and BI teams to eliminate technical bottlenecks.
- Collaborate with compliance and legal stakeholders to operationalize regulatory requirements into technical controls.
Required Qualifications:
Must Have:
- 7-10 years of hands-on experience in Data Engineering, with at least 3 years in a lead or technical ownership role.
- Expertise in advanced SQL:
- Window functions
- CTEs
- Incremental processing
- Query optimization
- Performance tuning.
- Strong expertise in DBT:
- Models
- Macros
- Jinja
- Incremental strategies
- Testing
- Packages
- Manifest-driven orchestration.
- Production-grade Airflow experience:
- Complex DAG authoring
- Dependency management
- Failure handling
- Backfill strategies.
- Strong experience designing cloud data platforms using:
- Databricks
- Snowflake
- Redshift
- BigQuery.
- Advanced Python skills for:
- Ingestion frameworks
- Validation systems
- File handling
- Watermarking
- Data transformation.
- Hands-on AWS expertise:
- S3
- EC2
- EKS
- IAM
- Secrets Manager.
- Proven ability to author and defend technical design documents and architecture decisions.
Preferred Qualifications:
- Strong experience with Databricks Unity Catalog, Delta Lake, Spark optimization, and governance controls.
- Experience with ClickHouse for analytical workloads and ingestion optimization.
- Prior experience in fintech or financial services domains:
- Mutual funds
- NAV/AUM reporting
- XIRR calculations
- RTA integrations.
- Familiarity with:
- SEBI/AMFI data standards
- BSE StarMF
- CAMS
- KFintech ecosystems.
- Experience with metadata and governance platforms such as:
- OpenMetadata
- Alation
- Data lineage tooling.
- Strong Kubernetes operational understanding:
- Helm configurations
- Pod scheduling
- Resource management.
- Understanding of the Digital Personal Data Protection (DPDP) Act and regulated data environments.
