Databricks Platform Engineer

We are recruiting a Cloud Data Platform Engineer to take ownership of the technology environment supporting our data engineering, analytics and machine learning teams.

Your focus will be on making the platform dependable, secure and efficient. You will combine hands-on engineering with operational ownership, helping technical teams deliver their work without unnecessary infrastructure or access barriers.

Working alongside engineering and analytics colleagues, you will develop the platform’s capabilities, strengthen its controls and improve the way it is deployed, monitored and maintained.

• Manage Databricks environments and the compute resources used for data processing, SQL analysis and modelling.
• Develop repeatable infrastructure deployments using Terraform and version-controlled configuration.
• Establish automated release processes, ensuring platform changes are tested, reviewed and deployed consistently.
• Maintain reusable configurations and engineering standards that simplify the setup of new workloads.
• Support workflow scheduling and integrations with the wider data ecosystem.

• Administer Unity Catalog and maintain appropriate permissions across data assets and environments.
• Configure identities, service accounts, authentication integrations and secrets management.
• Work with relevant colleagues to implement network controls, encryption and secure access.
• Maintain the controls and records needed to support auditing, data traceability and compliance requirements.

• Investigate and improve the performance of Spark workloads, Delta tables and SQL compute.
• Review resource consumption and identify ways to reduce unnecessary platform and cloud expenditure.
• Set up monitoring and alerting to identify failures, performance issues and capacity constraints.
• Resolve platform incidents, document their causes and introduce improvements that prevent recurrence.
• Maintain operational documentation and review changes for quality and reliability.

• Provide the platform foundations needed for data science experimentation and production modelling.
• Support capabilities such as MLflow, feature management and model deployment.
• Help users diagnose platform-related problems and adopt consistent engineering practices.
• Contribute to technical reviews, knowledge sharing and mentoring across the team.

• Practical experience managing Databricks in a production environment.
• A strong understanding of Apache Spark, Delta Lake and Unity Catalog.
• Experience working with one or more major cloud providers, including storage, networking and identity management.
• Confidence using Terraform and Git-based deployment pipelines.
• Good Python and SQL skills, with experience using PySpark and scripting for automation.
• Experience diagnosing performance issues and optimising distributed data workloads.
• Familiarity with single sign-on, automated identity provisioning and secure credential management.
• An understanding of data protection requirements and security controls within a commercial environment.