Care Quality Commission

Strategy for move to Testing Driven Development

Incomplete applications

Incomplete applications
9 SME, 3 large

Completed applications

Completed applications
11 SME, 5 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Friday 15 September 2017
Deadline for asking questions Friday 22 September 2017 at 11:59pm GMT
Closing date for applications Friday 29 September 2017 at 11:59pm GMT


Opportunity attribute name Opportunity attribute value
Summary of the work Deliverable to be procured:
1. Strategic overview - An overarching strategy that will define how CQC can adopt a test automation strategy at the data and application layers.
2. Implementation Design.
3. Non-functional Testing.
4. Requirements definition.
5. Resourcing requirements.
6. Policy and Training
7. Roadmap
8. Future proofing .
Latest start date Tuesday 10 October 2017
Expected contract length 6 months
Location London
Organisation the work is for Care Quality Commission
Budget range £70,000

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done This work will enable CQC to deliver new Digital services and intelligence led regulation with low risk and at pace. End of project testing in a waterfall delivery mechanism has proven to be unsuccessful and new approaches are required as CQC moves to an agile delivery model supported through test first strategies, continuous integration and rapid, automated deployments.

Specifically, this work is to define a test strategy and approach that supports continual change and release with automated regression testing through orchestrated services at the data and application layers
Problem to be solved Specifically, this work is to define a test strategy and approach that supports continual change and release with automated regression testing through orchestrated services, at the data and application layers.
Who the users are and what they need to do x
Early market engagement
Any work that’s already been done
Existing team CQC Digital team
Current phase Discovery

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place Care Quality Commission
151 Buckingham Palace Road
London SW1W 9SZ
Working arrangements To be agreed after proposal
Security clearance

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Experience, at scale, of applying testing services to support agile development, automated deployment and continuous integration across a range of services
  • Expertise in developing and delivering a test automation strategy to support large scale data processing and data science initiatives
  • Experience in assisting an organisation to move from waterfall development with quarterly deployments
  • Experience in managing legacy services interoperating with new services with differing deployment timelines in a bi-modal delivery model
  • Understanding of appropriate tooling, its configuration and implementation across a portfolio
  • Understanding of training and change management required to introduce new ways of working into an operational service
  • Solid understanding of orchestration and environment management in a mixed on premise and Cloud platform
Nice-to-have skills and experience

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 3
Proposal criteria
  • Technical solution
  • Approach and methodology
  • Estimated timeframes for the work
  • Team structure
  • Value for money
Cultural fit criteria
  • Work as a team with our organisation and other suppliers
  • Take responsibility for their work
  • Share knowledge and experience with other team members
Payment approach Capped time and materials
Assessment methods Written proposal
Evaluation weighting

Technical competence


Cultural fit




Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Hi, Could you please clarify whether you mean TDD or BDD? We are in the first instance looking for TDD. In the future we may wish to move further towards BDD but not immediately.
2. Do you have plans to migrate to new technologies We will be using new technologies but it is clear to us that some of our core legacy systems will be with us for a number of years even if we decide to migrate in the future. It is about creating framworks and working practices that allow them to be integrated into TDD
3. What is the current expected browser and device support Internet Explorer v11, Google Chrome. We utilise a number of different devices primarily lenovo thinkpad Laptops
4. What tools are currently used for development This would entirely depend on the system and/or software being developed. However as this is a strategic piece there is an opportunity to review and suggest changes to current working practices
5. Is there any continuous integration currently in place, if so, what is used We do not have a current established TDD envronments other than with individual products.
6. How long does testing take currently On the legacy systems e.g CRM testing can take 6-8 weeks for a significant release. For the smaller systems e.g ESB, Drupal Front end, these testng cycles are much shorter 1-3 weeks
7. How integrated are the test team with development team The test and development teams are quite well integrated for each individual system even where we have a combination of external suppliers and internal staff forming the test teams. We are not well integrated across systems
8. Do you employ contract testers on site Yes
9. Do you employ/engage any external testing services abroad No at present but we are looking to use one of our major 'international' suppliers in that way.
10. Are any non-technical staff heavily involved in testing, and if so, would they be expected to continue to be involved We employ user aceptance testers from our fronline staff for all our major testing cycles. They would obviously need to be suitably engaged in testing the functionality of new software
11. What unit/integration checks are currently being written/executed We create extensive unit, system and UAT scripts for the major legacy systems but where inetgration testing is required this is performed on a more ad hoc basis
12. How many testers are currently employed (non-contract) This varies, but up to 5 at peak times.
13. What current skills do any permanent testing staff possess We have staff who know our systems and develop testing scripts for the user acceptance testing. These staff have significant skills in this area. The system testing is generally performed by our suppliers
14. What is the current dev to test ratio For our core legacy systems it is between 50:50 and beginning to improve to 55 Dev: 45 test in terms of elapsed time
15. How is the test team structured There are a number of different structures currently for the different systems. Generally we have system test under the supplier and thus under Digital and then the UAT is under our bsuiness change function.
16. What operating systems / machines are the test team using day to day? All end users are on on Windows laptops. Obvious the systems run on a range of different technologies.
17. Has the team gone / undergoing agile training? All Digital teams have or are going on Agile training, as they work on Agile based developments
18. What tech / tools / languages are the test team using? This would entirely depend on the system and/or software being developed. However as this is a strategic piece there is an opportunity to review and suggest changes to current working practices.
19. Can you provide examples of the types of tests you imagine being automated? The areas where we would expect automation and TDD are in the front end Drupal developments, our ESB developments and Visual Studio work.
20. Are you looking to automate new tests going forward only, or to also automate your existing manual tests? We would be focused on automating new tests but may wish to automate some regression testing in order to deliver good TDD and continuous delpoyment.
21. What devices are currently available for testing use This would entirely depend on the system and/or software being developed. However as this is a strategic piece there is an opportunity to review and suggest changes to current working practices.
22. What is currently involved in a testable release This is completely dependent on the system. Suppliers and end users for system and UAT as a minimum
23. How are test environments currently set up On the legacy systems we have at least dev test and UAT environments populated with test data.
24. Are any architecture diagrams available to view We do not currently have diagrams that cover the development and test environments
25. What is the current process for making changes We currently havre a mixed economy. We have formal change request processes that go thorugh stages of prioritisation, analysis, development and test as well product managed services using Kanban. We are moving more and more to the latter.
26. With reference to questions on essential skill and experience, does CQC currently maintains any development and test environments on Cloud? If yes, which platform is used? The test environments in cloud are those supporting the Drupal developments. (See response to earlier question)
27. It is mentioned that there is need of defining test automated strategy at the data and application layers. Can you please provide some insight about number and type of applications currently being developed/maintained by CQC Accurate reporting of data and thus the impact of change on our reporting is critical to CQC. We have both a very wide range of MI and performance reporting from our Oracle warehouse, as well as a very wide range of risk reporting designed to support inspectors in their planning of inspections. This reporting combines our interenal data with a hundreds of externall acquired datasets to create indicators of quality fo the registered providers. It is important that we can be confident that change is not corrupting this reporting.
28. Does CQC Digital Team have any preference for Automation and CI-CD tools i.e. Open source or Commercial? There would be no preference. We would just want to select the best product. However we do utilise open source tools on some digital projects at present.
29. Is CQC open for other Agile methodologies for software development like ATDD (Acceptance Test Driven Development) and BDD(Behavior Driven Development) or the focus is on implementing developer TDD along with test automation to accelerate test cycles? CQC are positively moving towards Agile development wherever possible. We wish to move to TDD where practical
30. What are the agile related tool are you currently using? Jira for backlog management
31. Can you please elaborate the tools and technology currently used for testing of legacy system and new digital system both at data and application layer Our legacy system testing is performed in a very standard way with no automation. We have large test environments that hold test data and the test teams then run scripted tests in these environments. The scripts are cretaed prior to the testing in respnse the changes that are being put through.
32. In this exercise, what is the mix of applications that will be targeted. e.g. Mobile Apps, Websites, Backend systems, databases etc. As this ia a strategy piece of work we are looking for a review and then the development of options as to how we can integrate testing across all of our systems and applications. We have large core systems (e.g. Oracle Seibel CRM, warehouses with considerable MI and intelligence reporting off them (Oracle and SQL Server) ESB API connectivity (Mulesofft), a resource/inspection planning tool (CACI)and we are developing front end publically available applications (Drupal) na da large website for publication of the register and inspection reports. We expect the strategy to at least cover all these areas.
33. What are the main tasks/deliverables other than creating, writing and delivering a strategy? We would expect the deliverables from this work to cover:
An strategy setting out high level direction and aims.
Proposals for the techincal architectures and frameworks we would need to put in place to meet this strategy.
What resources would be required to support the testing. Initiation of policies required to ensure success.
A road map for moving to
Proposals that will help to future proof CQC against change in systems as we move forward with our Digital Strategy.
The version control and configuration managment that will need to be implemented to support the test strategy
34. Does this opportunity fall inside or outside IR35? This opportunity falls outside of IR35