Job Description
We are seeking a Senior Full Stack Engineer with strong backend and data engineering expertise to build and maintain the Cloud FinOps platform's data pipelines, APIs, and backend services. This role focuses on data ingestion, transformation, and serving layer development using Python, Databricks, and Azure services.
Key Responsibilities
Data Pipelines: Build and maintain data ingestion pipelines from AWS, Azure, and OCI using Azure Functions and Event Hubs. Develop Delta Live Tables pipelines for data transformation across Bronze, Silver, and Gold layers.
API Development: Design and develop RESTful APIs and GraphQL endpoints using FastAPI and Strawberry.
Business Logic: Implement business logic for cost allocation, chargeback calculations, and credit tracking.
Job Processing: Build background job processing using Celery for scheduled reports and batch operations.
Optimization: Optimize Spark jobs for performance and cost efficiency. Implement data quality checks, schema validation, and error handling in ETL processes.
Infrastructure & Security: Integrate with Azure SQL, Cosmos DB, and Redis for storage/caching. Deploy and manage containerized applications on Azure Kubernetes Service (AKS). Implement OAuth 2.0/OIDC using Azure AD B2C and manage secrets via Azure Key Vault.
IaC: Work with infrastructure-as-code using Terraform for environment provisioning.
Required Technical Qualifications
Experience:
6+ years of software engineering experience with a focus on backend and data engineering.
3+ years of experience with cloud platforms, preferably Azure.
2+ years of experience with Databricks, Spark, or similar distributed data processing frameworks.
Technical Skills:
Languages: Strong proficiency in Python 3.11+ including async programming.
Frameworks: Experience with FastAPI, Flask, or Django for API development.
Data Engineering: Hands-on experience with Databricks, Delta Lake, PySpark, streaming data processing, and ETL patterns.
Azure Services: Knowledge of ADLS Gen2, Azure Functions, Event Hubs, AKS, and Azure SQL.
DevOps: Experience with Docker, Kubernetes, Terraform, Git, and CI/CD pipelines.
Database: Strong SQL skills (Relational and NoSQL).
