data engineering

Project Role : Data Engineering

Project Role Description : Own the technical direction and architecture of custom software solutions, leading teams through design and delivery. Set development standards and ensure code quality, scalability, and performance aligned to business objectives.

Must have skills : Data Engineering

Good to have skills : NA

Minimum 5 year(s) of experience is required

Educational Qualification : 15 years full time education

Summary:

As a Custom Software Engineering Lead, a typical day involves overseeing the technical direction and architectural design of bespoke software solutions. This role requires guiding teams through the entire development lifecycle, from initial design concepts to final delivery. The position focuses on establishing development standards and ensuring that the software produced meets high benchmarks for quality, scalability, and performance. Collaboration with various stakeholders to align technical efforts with business goals is a key aspect of daily activities, fostering an environment of innovation and continuous improvement.

Roles & Responsibilities:

- Expected to be an SME, collaborate and manage the team to perform.

- Responsible for team decisions.

- Engage with multiple teams and contribute on key decisions.

- Provide solutions to problems for their immediate team and across multiple teams.

- Lead the development and implementation of architectural strategies to support evolving business needs.

- Mentor and support team members to enhance their technical skills and professional growth.

- Coordinate cross-functional efforts to ensure seamless integration and delivery of software components.

Professional & Technical Skills:

- Must To Have Skills: Proficiency in Data Engineering.

- Strong knowledge of data pipeline design, data warehousing, and ETL processes.

- Experience with cloud platforms and big data technologies to manage large-scale data solutions.

- Ability to optimize data workflows for performance and scalability.

- Familiarity with programming languages commonly used in data engineering such as Python, Java, or Scala.

- Expertise in database management systems and data modeling techniques.

-Hands-on experience with ADLS, ADF, Synapse, and Purview.

-Experience with Databricks (Delta Lake, Spark, Notebooks).

Knowledge of Vector Databases, Embedding Pipelines, and GenAI/RAG solutions.

-Experience with Azure OpenAI, Cognitive Search, and AI Studio.

Exposure to LangChain, LangGraph, CrewAI, or similar agent orchestration frameworks.

-Understanding of Data Governance, Security, and Compliance.

Nice to Have

-Experience with Azure DevOps, GitHub Actions, Docker, and Kubernetes.

-Familiarity with Great Expectations, Deequ, or data quality frameworks.

-Knowledge of Kafka, Event Hubs, and real-time data processing.

-Experience in performance tuning for large-scale embeddings and distributed compute environments.