NHS Digital

prj_3372 - NHS App Live Development and DevOps Delivery

Incomplete applications

12
Incomplete applications
8 SME, 4 large

Completed applications

18
Completed applications
9 SME, 9 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Tuesday 27 August 2019
Deadline for asking questions Tuesday 3 September 2019 at 11:59pm GMT
Closing date for applications Tuesday 10 September 2019 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work The Product Development Directorate, Citizen Health Technology Portfolio, at NHS Digital are seeking support for the Live Development and DevOps Delivery (including continuous improvement) of the NHS App. This work is to deliver digital transformation outcomes. Our preference is to use a Prime Contractor with one or more Sub-contractors.
Latest start date Friday 1 November 2019
Expected contract length Maximum of 17 months to 31 March 2021 (Post 31 March 2020 subject to funding approval)
Location Yorkshire and the Humber
Organisation the work is for NHS Digital
Budget range The Call-off contract will have an indicative maximum value of up to £13m over the life of the contract. The cost for the Initial SOW will be agreed at Stage 2 of the procurement process but will be Capped T&M linked to Outcome payments.

Charges for subsequent SOWs awarded under the Call-off contract will be agreed at the appropriate time. There is no commitment to work beyond the Initial SOW.

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done This is required to support the Live Development and DevOps Delivery of the NHS App.

Currently patients at 95% of GP surgeries in England can now use all the features of the NHS App. The NHS App offers a number of core user journeys which will continue to be improved over time. As and when new features or technical integrations are agreed with commissioner’s, clear delivery milestones will be allocated to each.
Problem to be solved The Buyer requires a supplier, with specialist skills in delivery and operation of large-scale public facing digital services, to support the DevOps and ongoing development of the NHS App. An agile iterative development model is being utilised so specific deliverables will be stated in each SOW. Suppliers will be expected to work as part of multi-supplier team.
Who the users are and what they need to do The NHS App provides a simple/secure way for people to access a range of NHS Services on their smartphone/tablet.

The NHS app enables patients to:
• check their symptoms using the health A-Z on the NHS Website
• find out what to do when you need help urgently
• register to be an organ donor
• choose how the NHS uses their data

If their GP practice is connected to the app, patients can register/verify their identity so they can:

• book/manage appointments at their GP surgery
• order their repeat prescriptions
• view their GP medical record securely
Early market engagement
Any work that’s already been done The NHS App is currently in the Public Beta phase and is being transitioned to live service. Currently patients at 95% of GP surgeries in England can now use all the features of the NHS App.
Existing team This work is currently delivered by two suppliers via separate DOS Call Off Contracts. The Buyer is looking to streamline the delivery approach for this work into one Call Off to deliver the next phase of the Programme requirements.

The Buyer would like to use a Prime Contractor with one or more Sub-contractors, but it is up to the bidder to describe their model for delivery.
Current phase Beta

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place Primarily based out of NHS Digital's offices in Leeds (currently Bridgewater Place). However, the Supplier may also be required to work in other NHS Digital offices in London, Exeter and could work in their own off-site offices if deemed appropriate.
Working arrangements We expect supplier staff to spend significant time on-site in Leeds. We are open to working remotely provided you can demonstrate a proven track record of doing highly collaborative design and development work with a remote team and can provide suitable tools to enable that.

Site visits and user research will need take place throughout England in a range of locations and care settings.

Travel to other NHS Digital offices may also be required. Only approved expenses supported by valid receipts in line with NHS Digital Travel policy will be paid.
Security clearance As a minimum, all supplier staff must have completed a Baseline Personal Security Standard (BPSS) check. Supplier staff with access to patient identifiable data/other sensitive personal information must have completed a security check. We would expect this to include any user researchers/data analysts/service designers and project managers.

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions Draft Order Form, the initial Statement of Work (SOW) and Draft Order Form/Call-off Terms and Conditions are available at the following link:
https://nhsdigital.bravosolution.co.uk/web/login.html

Bravo reference: pre_181 - NHS APP Live Development and DevOps Delivery.

To view the above you must be registered on NHS Digital's e-tendering portal. Suppliers not registered please register using the link above.

The Buyer reserves the right to award future SOWs under this Call-off Contract against all charging methods in the framework.

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Demonstrable capability in developing two or more service transformation projects for high volume, high availability digital services.
  • Evidence of delivery in accordance with the Government Digital Service Standard,
  • Evidence of working on native iOS and Android apps.
  • Demonstrable experience in successfully working in a multi-discipline, multi-vendor team environment.
  • Evidence of effective service design to inform/drive business change of service transformation for high volume/high availability digital services.
  • Resource/expertise to supply all team roles as detailed in the Government Digital Service Standard for each phase of delivery (Public Beta to Live).
  • Experience of implementing a DevOps model.
  • Evidence of effective knowledge transfer to build ‘in-house’ teams and sustainable in-house capability to support, operate and improve digital services.
Nice-to-have skills and experience
  • Have experience of successfully integrating digital services with Government platforms, for example GOV.UK Verify and GOV.UK Notify.
  • Have evidence of providing additional and/or value-added activities when acting in the role of prime contractor.
  • Have experience of working with Fast Healthcare Interoperability Resources (FHIR) or other healthcare standards.
  • Have experience of working with healthcare system suppliers in primary and/or secondary care.
  • Have evidence of delivering into health settings.

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 5
Proposal criteria
  • Delivery Approach and Capability
  • Resourcing Approach and Capability
  • Delivering to Government Digital Service Standard
  • Capability to Deliver within a Healthcare Environment
Cultural fit criteria
  • Experience of successful collaborative working alongside clients as part of a multi supplier team sharing knowledge within the team.
  • Detail how you will identify and keep the organisation informed of risks, dependencies, issues and other considerations relevant to planning.
  • Value for money.
Payment approach Capped time and materials
Assessment methods
  • Written proposal
  • Case study
  • Work history
  • Reference
  • Presentation
Evaluation weighting

Technical competence

60%

Cultural fit

10%

Price

30%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Please can you confirm if there will be a supplier briefing held for this opportunity? Yes, NHS Digital will be holding a supplier webinar on Monday 2 September at 2:00pm. Please register and/or log into Bravo for details.
2. Please can you confirm the Tech stack involved for both the App and DevOps? The specified technology the Supplier is required to use is stated within the draft Statement of Work that can be found on Bravo. All of the specified technologies outlined are currently deemed Essential for the delivery of the Releases. Where there is scope to improve tooling or technology as part of ongoing iteration this can be discussed and agreed.
3. Please ignore my last question about Tech stack, I have found the answer in the draft SoW. Apologies. No problem - please refer to additional clarification in response to question 2.
4. To get an understanding of scale, please can you let us know the number of developers currently engaged both from a development and DevOps perspective? The existing delivery approach utilises a four squad model with a mixture and roles and skills being deployed. It is anticipated that for the work being undertaken on the new contract specific roles will need to be filled and these are detailed in the draft Statement of Work that can be accessed via Bravo.
5. Who are the current suppliers delivering the services? Kainos and BJSS.
6. You mention you are working with two suppliers at the moment – can you share who they are please? Kainos and BJSS
7. Is there a mechanism by which "Primary Contractors" can be made aware of subcontractors who can provide a subset of these requirements? As sub-contractors do not need to be on the Digital Outcomes and Specialists 3 (DOS 3) framework we are not in a position to provide details of suitable organisations who could provide the capability; suppliers should utilise their own networking and supplier relationships. Bidding suppliers would need to put forward their proposal for how they will work with any other organisations to deliver this work, which will need to be outlined in more detail at stage 2.
8. Please can you confirm that the Bravo Portal is just for information, and that responses should be submitted via Digital Marketplace? Yes, at this stage, the Bravo portal is just to share additional information about the requirement.

All responses to the Shortlisting Criteria for this opportunity must be submitted via Digital Marketplace. Submissions to Bravo will not be accepted.

All questions about the requirement should also be submitted via Digital Marketplace, and not Bravo.
9. What is the size of the total team you are looking for and what is the split between Dev and DevOps? As potential suppliers, we are assessing the requirements in order to propose the right team but an indicative number from your end would be very useful. Please see the answer to question 4. Shortlisted suppliers invited to submit a tender as part of stage 2 will be requested to outline their resourcing approach and capability to meet the requirements of this contract.
10. You mention ''This work is currently delivered by two suppliers via separate DOS Call Off Contracts. The Buyer is looking to streamline the delivery approach for this work into one Call Off to deliver the next phase of the Programme requirements.'' Is this contract going to replace the existing contracts with Kainos and BJSS or will this add to the existing call-off? Yes, this will replace the existing contracts.
11. Are we required to only submit one response or if a subcontractor or being used do you require two responses, one from the prime and one from the subcontractor? Only the prime contractor is required to submit a response.
12. Please can you provide more details regarding what you consider to be high volume, high availability digital services? High volume refers to being capable to support multi-millions of users and transactions.
For availability 99.95% or above availability is required.
13. Please can you provide further details of what you mean by DevOps? By DevOps we refer to the culture and practice of having agile multidisciplinary teams working alongside operations teams to build and delivery digital services.
14. Why is the value of the contract so high? The value as stated on this market opportunity is an indicative maximum value for the term of the contract. This value is not guaranteed.
Individual Statements of Work will only be contracted for the duration and value of agreed funding that is available at the point of agreement.
15. Why are you proposing a prime/sub-contractor model? A prime/sub-contractor model has been chosen as a multi-vendor model will build resilience, diversity of skills, knowledge transfer, and retains some competitive pressure to keep quality high.
16. How much work do you expect could be completed remotely? Ideally teams will be collocated with our permanent staff on site in Leeds or London. However, we would consider remote teams if particular value for money could be demonstrated.
17. Would previous NHS Digital (or other scenarios) experience be an advantage over generic healthcare experience? All responses will be evaluated according to the evaluation criteria published.
18. Could NHS please confirm whether suppliers may present several versions of the sub/prime arrangement? E.g. can Supplier A and Supplier B each submit a Stage 1 response, and – if both are successful – Supplier A submits a Stage 2 response wherein it proposes to subcontract Supplier B, and Supplier B submits a Stage 2 response in which it proposes to subcontract Supplier A? Yes this would be acceptable e.g. one submission per supplier.

Multiple submissions from one supplier will not be accepted.
19. In question, ‘Evidence of working on native iOS and Android apps’, could the buyer please confirm that suppliers are permitted to provide two examples of app development work in the same response, in order to address the requirement for both iOS-native and Android-native experience? Yes, two examples will be acceptable.
20. In question, ‘Resource/expertise to supply all team roles as detailed in the Government Digital Service Standard for each phase of delivery (Public Beta to Live)’, could the buyer please clarify whether suppliers should provide an example and if so in what format? As the question does not lend itself to the usual preferred single-project format for DOS Stage 1 responses. If an example is required, could the buyer please advise how suppliers should provide evidence that will meet and exceed this requirement? As per the published evaluation criteria for stage 1 please provide either confirmation that you can supply all the team roles specified and how you would do this effectively, and/or provide an example of where you have done this previously.
21. In question, ‘Demonstrable capability in developing two or more service transformation projects for high volume, high availability digital services’, could the buyer please clarify a) precisely what is meant by ‘service transformation projects’ (i.e. performing a programme-wide business change initiative? Redesigning a discrete digital service? etc.) and b) whether in order to satisfy this requirement, suppliers should demonstrate having delivered two or more such projects ever, two or more projects concurrently, etc Please refer to question 12 for the definition of high volume and high availability.

Service transformation refers to transforming whole citizen-facing services or undertaking full department transformation rather than be focused on a programme or discrete digital service.

The examples provided do not have to have bee delivered concurrently.
22. Where there are a group of suppliers providing a proposal as a consortium, is the expectation from NHS that they will provide a single response to the evidence based questions? Only the prime contractor is required to submit a response.

Please note we are not looking for consortium or joint venture bids.
23. How do we formally instruct you of consortium members through DOS? in one of our answers? Only the prime contractor is required to submit a response. Bidding suppliers will need to put forward their proposal for how they will work with any other organisations to deliver this work, which will need to be outlined in more detail at stage 2.

Please note we are not looking for consortium or joint venture bids.
24. Often next phases of work go to the organisations who have been delivering previously - is there a particular skill set you are looking for that is new in this requirement? This procurement is a fair and transparent tendering process and aims to appoint the supplier that provides the best fit for the skills and experience as set-out in the digital marketplace opportunity.
25. How long do you envisage the handover period from the existing suppliers will be - to ensure no failure of the continuity? This is a key consideration and will be building in some overlap time for key members of the existing supply team to ensure that a full handover can be completed. We will agree the process for handover and timescales for this with the successful bidder.
26. It appears the past arrangement involved a separate Test organisation. Do you prefer the Test team to be independent, rather than part of the Development organisation? Albeit they would have to be sub to the prime contractor. It is expected that the development, test and assurance teams will operate as a single team. How this is delivered and by which specific suppliers within a prime/sub-contractor model should be outlined by suppliers within their bids.
27. Does the prime supplier need to be on the DOS3 Lot 1 framework to be eligible to bid for this work. Yes, this bid has been published under the DOS3 Lot 1 framework.
28. Is this a contract renewal and are the two incumbents set up to tender? We know they are on the DOS framework, we're asking if this is lined up for them. The existing contracts are due to come to their planned end date in the autumn. Both Kainos and BJSS will have the opportunity to bid for the new contract and will be evaluated in-line with the agreed evaluation methodology.
29. Given the criteria for a 3 'Exceeded' mark is 'Full answer plus additional relevant information provided', will the incumbent suppliers be allowed to use their previous work on this project as evidence of their capability? It is up to each potential bidder to use examples that they consider most appropriate in their tender response.
30. In question, ‘Have experience of successfully integrating digital services with Government platforms, for example GOV.UK Verify and GOV.UK Notify’, could the buyer please confirm whether suppliers should provide one example covering both platforms, whether it is suitable to use two separate examples in the same response, whether it is suitable to demonstrate use of one (i.e. either GOV.UK Verify or GOV.UK Notify), or – as these are examples – whether other similar Government platforms are suitable? The government platforms listed are examples. Suppliers can provide examples of integrating digital services with any government platform.
31. Will there be KT sessions by the current supplier to the new Contractor? Please refer to question 25.
32. Could the buyer please clarify what its aims are – i.e. what is it hoping to achieve – via the prime-/sub-contractor model, and whether those benefits would be available using a single supplier – as there are downsides to the prime/sub model (e.g. overhead/management costs) and a single supplier may be a better option, unless the buyer has specific requirements, not listed in the DOS ad, that can only be met using prime/sub. Will the Stage 2 evaluation criteria objectively award higher marks to consortium/joint/prime-sub bids? Any further guidance gratefully received. We are not mandating how suppliers propose to deliver, however our strategic aims of using a prime/subcontractor model are outlined in the answer to question 15.

At stage 2 we will be evaluating the Delivery Approach, Capability, as well as Resourcing Approach and Capability, and Cost. Higher marks will be given to the strongest proposal, which may or may not use a prime/subcontractor model. If the requirements can be met by a single supplier, then it would be considered and scored like any other.
33. In line with question 7, we are able to provide a sub-set, plus expertise, of the requested solution. I assume therefore that we can be made aware of the other participants who join the supplier briefing call so that we may be able to make contact and discuss partnering ? As sub-contractors do not need to be on the Digital Outcomes and Specialists 3 (DOS 3) framework we are not in a position to provide details of suitable organisations who could provide the capability; suppliers should utilise their own networking and supplier relationships.
34. Will you be sharing the presentation from the supplier briefing held for this opportunity on Monday 02 September? Yes, this has be published on NHS Digital's Bravo e-tendering portal.
35. Are you specifically looking for bids that use sub-contractors. Please see the answer to question 32.
36. What are the key metrics and performance indicators NHS is looking to achieve with the project and solution? Detailed key metrics and performance indicators will be agreed with the preferred supplier and formalised in the call-off contract. These will include high level metrics on users and downloads, operations and performance, service management, user satisfaction, and metrics specific to particular journeys within the app e.g. booked appointments.

Within the draft Statement of Work available to view on Bravo it has been stated that Service Level Agreements (SLAs) including availability and response times to incidents will be agreed according to phase. For availability 99.95% or above availability will be required.
37. Are you able to share the recording from the supplier briefing held for this opportunity on Monday 02 September? Unfortunately we had technical issues with the Skype recording and it is not available to share.

The slides from the call have been published on NHS Digital's Bravo e-tendering portal. All the questions and answers from the session have been published on Digital Marketplace.
38. Due to the nature of the app, would you be looking for all teams to be on-shore or will you be considering near-shore/off-shore options where there is good value for money? We will consider the merits of any proposed offshore arrangements during the Stage 2 evaluation. Any offshore work must be compliant with the Call Off Contract and no Patient Identifiable Data or Personal Confidential Data can be processed outside the UK.

Work will be primarily based at NHS Digital's Leeds office. The Supplier may also be required to work in London/Exeter and could work in their own offices if deemed appropriate. We are open to working remotely provided you can demonstrate a track record of highly collaborative design/development work with a remote team and suitable tools to enable that.