This opportunity is closed for applications

The deadline was Wednesday 30 December 2020

Health Education England

Using Machine Learning for predictive analytics across the NHS trainee pipeline (Alpha)

12 Incomplete applications

9 SME, 3 large

10 Completed applications

8 SME, 2 large

• Published: Wednesday 16 December 2020
• Deadline for asking questions: Wednesday 23 December 2020 at 11:59pm GMT
• Closing date for applications: Wednesday 30 December 2020 at 11:59pm GMT

Overview

• Summary of the work: HEE is seeking a partner with expertise in Machine Learning to assist in using predictive analytics to better understand our trainee doctor pipeline. This involves undertaking the full AI lifecyle, deploying models with demonstrable business value, and informing our approach to solving future ML problems.
• Latest start date: Monday 1 February 2021
• Expected contract length: 2 months
• Location: No specific location, for example they can work remotely
• Organisation the work is for: Health Education England
• Budget range: < £99,000 inc VAT

About the work

• Why the work is being done: This alpha project seeks to utilise machine learning (ML) to provide HEE with a better understanding of the NHS trainee (e.g. junior doctor) pipeline.; ; This work comes in response to the urgent workforce challenges outlined in the Long Term Plan and its ambition to address these by ‘making better use of data and digital technology’ as well as recommendations within the Topol Review to utilise AI to better understand the uniqueness of each individual.; ; This project aims to build on previous discovery and alpha work, to undertake the full AI lifecycle, to deliver accurate ML models which will help us predict and identify factors affecting the progression of our medical workforce through training (such as attrition and extensions to training), and ultimately inform our long-term approach to using ML to solve future problems for the organstion. ; ; HEE needs to extract new insight to better understand the reasons for people leaving, the pattern of attrition across the medical workforce, the factors causing gap in attainment levels, and related 'trainee pipline' priorities. Together, this work aims to enable improved, individualised support to trainees and direct development of tailored retention strategies across the NHS.; ; All work must complete by 31st March 2021.
• Problem to be solved: A number of closely related use cases have been identified below. Tackling these is in line with Gartner's recommendation of targetting a cluster of uses cases around common data.; ; - understand the reasons for people leaving their posts so we can better tailor retention strategies; - understand the potential impact when implementing programmes aimed at reducing burn out and subsequently attrition; - understand the pattern and causes of attrition across the medical workforce; - understanding factors affecting extensions to training ; - understanding reasons for different attainment between groups (inc. exam pass rates and ARCP outcomes) in order to inform appropriate interventions; - identifying what precedes a trainee experiencing difficulty in their training so that these early 'signals' can lead to provision of support; ; Objectives; Working in partnership with HEE to; • finesse our use cases; • build upon our existing ML use case assessment framework; • undertake the full AI lifecyle to design, build, test, deploy, and monitor models with demonstrable business value on our Azure platform; • develop an alpha process for utilising ML for additional use cases; • develop our staff skillset; • identify how HEE might to establish an ML capability to build and support ML products
• Who the users are and what they need to do: As a Medical Education Reform Programme lead; I need to understand the reasons for people leaving their posts ; So that I can better tailor retention strategies; ; As a Medical Education Reform Programme lead; I need to understand the potential impact when implementing programmes targetting improved retention; So that my programmes have a greater chance of success; ; As a Medical Education Reform Programme lead; I need to understand the pattern of attrition factors across different segments the medical workforce ; So that I can identify specialties where a positive impact of intervention is more likely; ; As the PGMDE portfolio lead; I need to be able to understand causes of extensions to training ; So that I can identify appropriate support and interventions for trainees; ; As the PGMDE portfolio lead; I need to examine factors causing gap in attainment levels between different groups; So that I can identify appropriat strategy; ; As a BI analyst/developer; I need to be able to develop and interpret ML models on our Azure platform against our datasets; So that I can use information in the reports and dashboards I build; ; Note: above represents a subset of user stories as per aims outlined elsewhere in this Description of Work
• Any work that’s already been done: Earlier this year, HEE ran a discovery and alpha to test whether we could deliver new insight on a low-cost ML platform.; ; We found that we could process more data, at greater speed, to find hidden patterns to both develop and test theories regarding the NHS workforce using our trainee datasets.; ; This work was completed on the Azure Machine Learning services platform where prototype models where built looking at trainee attrition for GP and Cardiology specialties.; ; These models were useful in informing what could potentially be achieved with more time and resource against a pool of related use cases.
• Existing team: Existing team are skilled in SQL Server and Tableau with some to no experience using ML within Azure. ; Some of project team have received training and certification using Azure ML area with 'real-world' experience has largely been gained via our previous work during discovery and alpha.; The team adopt an agile approach to existing workload.; Subject-matter expertise of the NHS trainee pipeline will from part of the internal project group .
• Current phase: Alpha

Work setup

• Address where the work will take place: Remotely.
• Working arrangements: We require individuals at appropriate times to work in partnership with our users, analysts, and developers. Preferably this will be via Microsoft Teams. This is so that skills and knowledge can be shared, user needs fully understood, and that HEE is able to support and iterate upon anything built in this alpha stage.

Additional information

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

• Essential skills and experience:
  • have experience designing and deploying ML services
  • have experience working with NHS-specific datasets
  • be able to work in partnership with HEE so that knowledge and skills canbe shared to support any alpha services that are developed
  • be able to provide specific and relevant examples of previous work
  • have experience delivery projects using an Agile approach
  • be able to utilise Azure Machine Learning for model development
• Nice-to-have skills and experience:
  • have knowledge of NHS trainee pathways
  • have experience designing and deploying ML services within the NHS
  • knowledge of NHS strategies including Topol Review, Long Term Plan, Interim People Plan
  • knowledge of GDS Service Standard
  • knowledge of NHS Digital Guidelines inc. Code of conduct for data-driven health and care technology

How suppliers will be evaluated

All suppliers will be asked to provide a written proposal.

• How many suppliers to evaluate: 6
• Proposal criteria:
  • Technical solution
  • Approach and methodology
  • Estimated timeframes for the work
  • Team structure (inc. no sub-contractors)
  • How the approach or solution meets our goals
  • how the approach or solution meets user needs
  • how they’ve identified risks and dependencies and offered approaches to manage them
  • how the approach can be flexible in the event of COVID disruption
• Cultural fit criteria:
  • work as a team with our organisation and other suppliers
  • be transparent and collaborative when making decisions
  • have a no-blame culture and encourage people to learn from their mistakes
  • take responsibility for their work
  • work in close collaboration to share knowledge and experience with other team members
  • can work with clients with low technical expertise
• Payment approach: Fixed price
• Additional assessment methods:
  • Case study
  • Work history
• Evaluation weighting:

  Technical competence

  60%

  Cultural fit

  20%

  Price

  20%

Questions asked by suppliers

• 1. Please clarify whether the previous Discovery and Alpha work has been undertaken by HEE or an external supplier? If an external supplier, please name the supplier.: The previous discovery and aplha was completed working in partnership with ICS.AI.; ; More details on this project acan be found here: https://www.digitalmarketplace.service.gov.uk/digital-outcomes-and-specialists/opportunities/11213
• 2. Could we please ask for an extension? Given Christmas leave, we would struggle with resources available to develop the responses but would be very much keen to bid, if we had 1st week in Jan to respond.: Unfortunately we are unable to extend. We have pursued this with the Crown Commercial Service but have been advised that 'requirements advertised on the digital marketplace relating to the Digital Outcomes and Specialists framework are fixed'.
• 3. Does this work need to finish this financial year?: Yes.
• 4. Is the start date fixed?: There may be some slight leeway on the start but given the relatvely short window for development work and the need to ensure there's enough time for input from HEE, it is expected that any changes to start date would be minimal.
• 5. Could you please clarify what you are looking for in relation to the question “be able to provide specific and relevant examples of previous work”.Other questions already ask for on experience of ML, NHS datasets, agile and GDS for example.: We are simply looking for some insight into what a supplier’s previous and relevant work entailed (i.e. specific examples) rather than a broad statement of experience in ML, NHS datasets, agile, GDS, etc.