British Heart Foundation’s journey to a Databricks Data Intelligence Platform on Microsoft Azure with Simpson Associates 

Overview

Every 3 minutes, someone in the UK loses their life to cardiovascular disease. For over 60 years, British Heart Foundation (BHF) has been at the forefront of the fight against heart and circulatory diseases – a mission so effective it has helped to halve the annual death rate from these conditions.  

As the largest independent funder of cardiovascular research in the UK, BHF’s work is underpinned by a strong commitment to three core pillars. They pioneer research that drives breakthroughs in treatment and prevention. They also tackle health inequalities, influencing government policy to ensure everyone has access to the best care, and they empower the public with life-saving skills like CPR, turning bystanders into lifesavers. 

To power this colossal effort, BHF relies on a vast network of dedicated people – from innovative researchers to an army of volunteers. Providing them with timely and valuable data is critical to their success. This data covers everything from the geographical spread of heart conditions to the impact of public health initiatives. BHF also uses data from its retail arm, one of the UK’s largest, to support its vast network of over 700 stores nationwide, which play a crucial role in funding their research and services. 

To bring this vision to life, BHF recognised the need for a modern, cloud data platform to support all their initiatives. They partnered with Simpson Associates to implement the Databricks Data Intelligence Platform on Microsoft Azure, bringing the power of the Databricks Lakehouse to the Medical Directorate and Analytics teams across BHF. By analysing this vast amount of information, BHF can measure and evaluate the impact of its funded research, gather health statistics from across the UK, and build new statistical models to support key organisational decisions. This crucial work ensures a steady flow of funding for its groundbreaking research and services. 

Problem

British Heart Foundation had an ambitious vision for its data, but their technical reality was holding them back. While their Medical Directorate has a team of talented, driven data professionals, they were being held back by inadequate data platforms and processes. The team was hitting three frustrating obstacles that kept them from innovating and scaling their most impactful work: 

No Standard Data Platform 

BHF relied on a fragmented ecosystem of complex and disparate data sources, struggling with data silos. Information from their national defibrillator network was not centrally managed, making it difficult to combine for cross-project analysis. This lack of a unified cloud data platform reduced the ability to collaborate and slowed project delivery. 

Lack of Collaborative Framework 

The data team operated with siloed, localised development practices. Code was created on individual laptops with no central repository or version control, leading to duplication of effort and limited collaboration. This made it impossible to build a cohesive, reusable knowledge base. 

Lack of Compute 

With development tied to individual machines, complex and high value simulations were taking days to complete. This created project delays but also limited the scope of analysis the team could undertake, directly impacting the speed at which they could generate actionable insights. 

Solution

Working as part of a wider strategic data transformation program, Simpson Associates provided a strategic blueprint for British Heart Foundation by implementing the Databricks Data Intelligence Platform on Microsoft Azure. This modern data platform leverages key components like Delta Lake, Unity Catalog and Databricks Genie to provide a secure and scalable environment for their data science teams.  

The new platform serves as the Integration and Reporting Platform (IRP), a datahub built on Databricks. It is designed to support a range of programs, including data science work and the Enterprise Foundations (EF) Platform. The EF Platform, a collaborate effort with PwC, is designed to integrate and replace disparate systems like CRM, HubSpot and finance, ensuring a unified flow of data across the organisation.  

This new architecture enables the delivery of automated insights, high-impact reporting, and AI capabilities, allowing BHF to plan and execute strategic health objectives with reduced manual intervention and improved data quality. This strategic move has freed up data professionals to focus on advanced projects that directly impact BHF’s mission.  

Key Benefits

During the delivery, the BHF team focused on three core initiatives that have now been successfully deployed, demonstrating immediate value to the organisation. 

• RevivR App Analysis: British Heart Foundation’s RevivR app is a free digital CPR training tool designed to make life-saving skills accessible to the public.  
  

To maximise its impact, the BHF team uses Databricks Delta Sharing to analyse and securely share anonymised data from the app. This data-driven approach allows BHF to identify areas with low training coverage and target efforts to improve accessibility in underserved communities.  

Databricks Delta Sharing is crucial for this work as its secure, real-time protocol allows BHF to share this valuable data without having to replicate it, while also maintaining a full audit trail. 

• Defibrillator Distance Tool: BHF operates The Circuit – the national defibrillator network which provides ambulance services with vital information about defibrillators across the UK. However, identifying areas that lack adequate defibrillator access can be a difficult and time consuming task.  
  

This data-driven solution solves that problem. It optimises data from The Circuit alongside open-source data to pinpoint ‘priority areas’ with the greatest need. The tool uses H3 spatial indexing to partition areas into hexagonal grid cells, reducing computational load. 

By awarding defibrillators to applicants in these key priority areas, BHF aim to reduce bystander retrieval time for a defibrillator when some has a cardiac arrest. The new platform has helped to reduce complex spatial analysis time from over 3 days on a laptop to less than 6 minutes on the new platform by being able to select the perfect size and type of compute cluster on Databricks’ from a variety of options. 

• Research Intelligence Hub: It is key for BHF to be able to easily access information about the research it funds, along with its outcomes and impact. To this end, BHF collects and uses data from a variety of internal and external sources, but it lacked a central place to store, update and explore these datasets easily.  
  

This tool automates and consolidates various impact and evaluation data sources, providing a comprehensive and interactive view of BHF’s research activities. It enables actionable insights and facilitates data-driven decision-making, while using Unity Catalog to manage and secure data assets for consistent governance.  

Ultimately, this allows BHF to more easily mobilise research data and insight to inform strategic decision-making as well as supporting its fundraising and influencing activities. 

Beyond these core initiatives, the scalability and innovation of the Databricks Platform proved transformative for projects that were previously impossible. For example, the BHF team needed to work with the English Prescribing Dataset (EPD) to understand prescription patterns over time and the relationship with cardiovascular conditions. The sheer size of this dataset (a single month is approximately 6GB) made it impractical to work with locally. 

The new Databricks platform provided the compute and storage needed to seamlessly integrate the entire EPD dataset, going back to 2014. Now, the team is exploring Databricks Genie, an AI Assistant to ask questions of the data using natural language. This capability is currently being used for analysis on prescription patterns, making valuable insights instantly accessible to all team members, regardless of their technical level.

Conclusion

British Heart Foundation’s partnership with Simpson Associates to implement the Databricks Data Intelligence Platform has been a transformational journey. By addressing core pain points related to lack of a unified data architecture, British Heart Foundation has unlocked its data potential. The creation of the Integration and Reporting Platform (IRP) and the ongoing work on Enterprise Foundations (EF) demonstrates BHF’s commitment to building a unified and well governed ecosystem.  

The implementation of the Databricks Data Intelligence Platform on Microsoft Azure has led to the successful deployment of high-impact use cases that are directly supporting BHF’s mission, from optimising defibrillator placement and improving CPR training to accelerating research. BHF is now positioned with a scalable, secure, and powerful lakehouse platform to continue its life-saving work. 

Project stats 

Defib Distance Tool 

Reduced complex spatial analysis time from over 3 days on a laptop to less than 6 minutes on the new platform. This represents a reduction in computational load of almost 75%. 

RevivR App Analysis 

Over 124,000 people have completed BHF’s 15-minute digital CPR training. The data from the app helps strategically target training to combat the statistic that 23 million UK adults have never learned CPR. Learn more here.

English Prescribing Dataset (EPD) 

The Databricks platform successfully ingested and made accessible a 1.5 billion-row dataset of prescriptions dating back to 2014. This work transformed the analysis of a previously “bothersome” dataset into a process that can now be queried in minutes.