Senior Data Engineer

Role Senior Data Engineer (data architecture)

Seniority Senior: works autonomously, leads local design and guides other engineers- Platforms Microsoft Fabric, Databricks, Microsoft Purview, Prof isee

Role summary: A senior data engineer who combines hands-on PySpark delivery with data architecture input: leading local design, influencing design analysis, and owning data cataloguing and master data standards across Microsoft Purview and Profisee.

Activities & outputs:

Data architecture & design

• Lead local solution design for assigned data flows and influence wider design analysis, including Source-to-Target (S2T) mappings and medallion layer patterns
• Catalogue data assets in Microsoft Purview: metadata, classifications, business glossary terms and lineage
• Work with Profisee MDM: master data models, matching and survivorship rules, and integration of golden records with the lakehouse
• Challenge and improve designs, raising risks, trade-offs and dependencies early

Engineering & delivery:

• Deliver high-quality PySpark development in Fabric / Databricks notebooks within an existing Bronze–Silver–Gold architecture
• Extend and optimise existing ETL/ELT pipelines and downstream exports (batch, real - time and on-demand)
• Own data quality, performance tuning and issue resolution for assigned flows, applying modular, maintainable, version-controlled code
• Deliver against a high-volume backlog (200+ flows), prioritising critical milestones (e.g. June critical path) and adapting to changing requirements
• Collaboration & ways of working
• Act as a senior point of contact for technical teams and business stakeholders, communicating progress, blockers and dependencies clearly
• Guide and review the work of other engineers in the distributed/offshore team
• Follow client data policies, governance and security requirements; maintain documentation and traceability

Skills & experience- Essential:

• Microsoft Purview: data cataloguing, source scanning and classification, business glossary, lineage and data products
• Profisee MDM: entity modelling, match/merge and survivorship rules, data quality rules, stewardship workflows, and integration with Fabric / Azure Data Factory
• Data engineering: strong PySpark and SQL in Fabric / Databricks notebooks, with proven delivery in a medallion (Bronze–Silver–Gold) architecture
• Data design: data modelling and Source-to-Target mapping; turning business requirements into practical designs and leading local design decisions
• Engineering practice: data quality testing, performance tuning, Git version control and CI/CD
• Seniority: track record of working autonomously, mentoring engineers and engaging confidently with business stakeholders

Desirable:

• Certifications: Microsoft Fabric Data Engineer (DP-700) or Fabric Analytics Engineer (DP-600)
• Wider tooling: other MDM or catalogue tools, e.g. CluedIn, Informatica, Collibra
• Semantic layer: understanding of Power BI semantic models and Fabric IQ, and how mastered data feeds them
• Context: experience in regulated or data-intensive sectors, ideally utilities