HM Courts and Tribunals Service

HMCTS Crime Programme Test Services

Incomplete applications

18
Incomplete applications
15 SME, 3 large

Completed applications

22
Completed applications
14 SME, 8 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Thursday 19 September 2019
Deadline for asking questions Thursday 26 September 2019 at 11:59pm GMT
Closing date for applications Thursday 3 October 2019 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work To provide a flexible-team of c.18 experienced testers(including TPL)to:

• Oversee/ensure the quality of testing.
• Ensure features delivered and maintained to a high-level of quality/standards
• Develop/deliver effective test plan & associated metrics
• Provide appropriate test-reports as required.
• Maintain manual &automated test-suites including functional & non-functional scenarios
Latest start date Monday 20 January 2020
Expected contract length 24 months with option to extend for further 6 months
Location London
Organisation the work is for HM Courts and Tribunals Service
Budget range The Target Day rate range for Testers is between £675 - £750 per day, and under £990 for the Test Practice Lead.
Pricing will be evaluated out of 35% and pricing submission documents will be issued to suppliers who pass Round 1. Suppliers will be evaluated as per DOS criteria. The initial Statement-of-Work and Roadmap is available upon request. It is recommended that interested parties request this document.

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done To provide a flexible-team of c.18 experienced testers(including TPL)to:

• Oversee/ensure the quality of testing.
• Ensure features delivered and maintained to a high-level of quality/standards
• Develop/deliver effective test plan & associated metrics
• Provide appropriate test-reports as required.
• Maintain manual &automated test-suites including functional & non-functional scenarios
Problem to be solved The CJSCP Programme is designed to develop efficient, proportionate and accessible digital services to enable access to justice via appropriate modern channels. The Common Platform, which is a micro-services based architecture based on the JEE framework, provides service to CJS at all stages of the criminal justice process. To support this development an effective test function is required to develop and deliver an effective cross programme service (including test strategy, plan and associate monitoring / reporting metrics), enabling the programme to go live with confidence.
Who the users are and what they need to do 1) As a prosecutor, understand case evidence to ascertain likelihood of a successful prosecution.
2) As a defence practitioner, understand the evidence of a case to best advise my client.
3) As a HMCTS admin, manage the delivery of justice services so the process is delivered efficiently.
4) As a member of the judiciary, access all relevant information so that I can make a judgement
5) As a prosecuting agency, submit evidence to provide HMCTS with relevant information.
Early market engagement No
Any work that’s already been done The Crime Programme is established and has been running for a number of years.
We have multiple services already live on the platform including a citizen facing service in the form of online plea as well as a pre-charge capability for CPS and an initial case working function which enables single justice procedure cases to be handled within magistrates and crown court. The focus over the next few months is to deliver the full court capability which will enable the decommissioning of all current legacy systems that support the criminal justice service of today.
Existing team The current test team is led by a Test Practice Lead (TPL) with test resources embedded in existing digital delivery teams. Each team comprises of a delivery manager, a technical lead, a business analyst, user research/designer and frontend/backend developers. It is expected that the successful provider will provide a high level of delivery confidence through robust test execution, automation and output visibility and reporting against KPI’s, all executed through best practice capabilities across the function and supported by a comprehensive strategy and test plans by existing and future increments / features.
Current phase Beta

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place Primarily Southern House, Wellesley Grove, Croydon CR0 1XN. (Although some visits to other locations may be required.)
Working arrangements The team will be required to work onsite at the base location working a five(5)day week. The Service is being delivered against the GDS Service Manual using a Scaled Agile Framework (SAFe). SoWs will be issued to define the development outcomes of each project phase and typically span three to six-months.For any work performed at a location different to that of the Base-location and London(outside of the M25),all reasonable travel and expenses costs shall be met in accordance with the rates set out in the MOJ travel and subsistence policy.All expenses will require prior approval from HMCTS before being reimbursed.
Security clearance All supplier resources must Baseline Personnel Security Check (BPSS) clearance which must be dated within three months of the start date. See https://www.gov.uk/government/publications/government-baseline-personnel-security-standard for further guidance. The role of test practice lead will require SC security clearance

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions The initial Statement-of-Work(SOW) will be Time&Materials but the Buyer reserves the right,in the contract,to use alternative payment mechanisms such as Fixed-Price or Capped-T&M for future SOWs. Suppliers shall provide transparency to the Buyer on the rates paid to resources & any third-parties in the supply-chain on request.Expenses shall be submitted in line with the MOJ Standard T&Spolicy and agreed in advance of expenditure.

Liability-Insurance:minimum level of cover £5,000,000;
Professional-Indemnity:minimum level of cover £1,000,000

Suppliers must be able to provide resources within a 2-week notice period with the necessary skills to minimise impact on delivery.

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Test Practice Lead (TPL) and Testers to have demonstrable experience of coding automated checks in Java or JavaScript to a high standard, following typical design patterns and good practices.
  • TPL&Testers have demonstrable experience of producing and maintaining automated checks using RESTassured & Protractor, JMeter and Gatling (Performance) & OWASP ZAProxy (Security).
  • TPL&Testers have demonstrable experience of collaborating with developers across the programme to push automated checks down the Automation Pyramid.
  • TPL&Testers have demonstrable experience of operating within an Agile development methodology, specifically demonstrating experience of being able to work within any agile team and completing all testing-activities within sprint boundaries.
  • TPL&Testers have demonstrable experience of providing advice&guidance to teams on what,where&how to test features.Evidence should specifically include advice issues occurring using Event debugging tools,Dead Letter Queues,Monitoring,Alerting&aggregated logs in MicrosoftOMS
  • TPL&Testers have demonstrable experience of XP Practices – including Continuous Integration&infrastructure code tools(Jenkins and Ansible on Azure)including debugging issues, monitoring, alerting, log aggregation and Test-Driven Development (TDD), Test Design Improvement.
  • TPL&Testers have demonstrable experience of conducting non-functional testing and operational acceptance testing.
  • TPL&Testers have demonstrable experience of producing regular test reports on the state of the Application
  • TPL&Testers have demonstrable experience of being able to perform a Root Cause Analysis of Live features and identify potential bottlenecks in the application by reviewing application logs
  • TPL to have demonstrable experience of working on large scale programmes/project delivery/software development environment using complex integrated applications architectures including Java based microservice architecture.
  • TPL to have demonstrable experience of creating and implementing a Programme Test Strategies and test approaches to be adopted within the development teams.
  • TPL to have demonstrable experience of owning an operational acceptance testing approach, driving it through to completion and ensuring buy-in from all teams.
  • TPL to have demonstrable experience of being able to quickly analyse issues and suggest fixes with defined verification steps
  • TPL to have demonstrable experience of strong Client Facing skills, excellent written and verbal communication skills with the ability to communicate with technical and non-technical backgrounds and an ability to-understand-their-key-business-drivers
Nice-to-have skills and experience
  • Please describe your experience using the Digital by Default standard
  • Please describe your experience of testing CQRS and/or asynchronous systems
  • Please describe your experience using a SaaS offering for cross browser testing

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 5
Proposal criteria
  • Explain your overall Testing Approach,Delivery Methodology&key-artefacts to enable digital-teams&deliver capability within an agile-environment,how these complement Crime Programme’s requirement&how you’ll apply lessons-learnt from similar projects in the past to this programme(6%).
  • Explain how you’ll implement the first SOW;including how you’ll manage the transition-period(knowledge transfer,onboarding resources&maintaining velocity)to ensure you deliver the outputs in the first SOW&cite any risks you have identified(5%).
  • Explain how your TPL will work with your Testers,other Practice-Leads,delivery-teams&senior-stakeholders to develop&deliver a Test_Strategy&Plan with associated-metrics to ensure a high-quality-testing-service is achieved across program.Include a successful previous-engagemnt of your-proposed TPL_6%
  • Explain your non-functional testing approach,what kind of technical-checklist you put in place and what impact this had on easily signing off non-functional requirements on-schedule.Including delivering a test strategy&associated reporting&monitoring metrics(6%).
  • Describe how you will measure and manage quality,speed of delivery teams and individuals for continuous improvement throughout the contract including how you would meet the supplier targets for SOW001(6%).
  • Propose your approach for enabling digital teams, delivering capability and ensuring that knowledge transfer is conducted during handover of project outputs to the HMCTS BAU live/maintenance teams(6%).
  • Provide CVs_of_named-individuals in proposed-team(junior-to-senior)to meet initial-requirements showing_experience working in agile,multi-vendor teams&confirm the minimum-experience required for each role.Provide the same information if proposing an alternative,more flexible resourcing-approach(5%)
  • Provide three referenceable client focussed case studies where your company has provided the desired service capability; identify the proposed team member(s) from Q7 that were part of the delivery team(6%).
  • Explain how for contract-duration you will: • Retain key resources • Sustain pricing whilst retaining suitably experienced testers within changing-market conditions. • Maintain expected team velocity(6%)
  • Explain how you’ll flex your service-offering to meet HMCTS’ rapid demand/technology change&still provide high quality,appropriately-skilled testers in-line with the 2-week notice-period&what internal-tests/processing you’ll undertake to ensure developers match our requirements(5%)
Cultural fit criteria
  • Please describe your approach for working with multi-vendor teams across multiple projects at different locations(2%)
  • Please explain how you will ensure collaboration at all levels of the project and programme delivery between users, team members, and management(2%)
  • Please explain how you would maintain a no-blame culture(2%)
  • Explain how you will maximise the value added by co-locating with the other disciplines during the initial launch set up, mobilisation / transition period.(2%)
Payment approach Time and materials
Assessment methods
  • Written proposal
  • Case study
  • Work history
  • Reference
  • Presentation
Evaluation weighting

Technical competence

57%

Cultural fit

8%

Price

35%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Can you please provide the contact details where we have to request the initial SOW and Roadmap A draft SOW and roadmap can be obtained by emailing crime-commercial@justice.gov.uk
2. You describe a current Test team structure. Is that internal or 3rd party ? If the latter, are you able to share who the incumbent is ? And are they able to re-bid for this tender ? The current testing capability is provided by a 3rd party. Kainos Software Ltd are currently on the DOS framework and will be free to bid for this work at their discretion
3. Do you expect this to be assessed as outside IR35? The Crime Programme currently assess this opportunity to be a Services contract and outside of the scope of IR35. We are not in a position to speculate on any future assessments of IR35 which may be undertaken by HMRC.
4. Can you confirm if there is a testing team currently in place and will they be bidding for this work? There is a testing team currently on the Crime Programme as indicated in answer 2. The authority is not able to see on the CCS portal who is bidding until the competition closes.
5. There is a lot of detail around the evaluation marking (we assume this is at round 2) but very little around how the initial sift will be done to choose suppliers who will progress through to the evaluation phase. The initial application questions (17 in total) are all very technical/operational.....is there any weighting being applied to the 14 essential and 3 nice to have questions ? There is no weighting being applied to the essential and nice to have criteria in round 1. For the avoidance of doubt all weightings will be applied to round 2 as detailed in the published requirements.
6. Is the initial sift being undertaken by Civil Servants involved in the function/programme, or by a third party ? If the latter, is it possible to know who is providing this service ? The evaluation will be undertaken by members of the Crime Programme delivery team which will include Civil Servants.
7. Is the Authority looking for any new or additional services from the new suppliers though this procurement? The authority is seeking a test service based on the requirements outlined on the portal
8. Could you please confirm who is the current Incumbent? Kainos Software Ltd
9. To provide a flexible-team of c.18 experienced testers(including TPL)to' ...Could you define flexible-team? The test team should be able to span multiple projects and increment development across the programme rather than a static resource assigned to each team.
10. Can you provide an indication of the split of resources in terms of grade and skills that will be required to deliver the testing service? Taking into consideration the overview of the programme requirements, we would expect the bidders to present options for grade and skill split to meet our needs.
11. How many of the 18 team members currently form the core of the test team, i.e. are working continuously and are not part of any flex function? All current test resources form a core function, embedded within the teams and supporting the delivery of specific increments
12. Can you share your existing KPIs that relate to this service? Existing KPIS are based on Fulfilment, Delivery and Quality of Service.
13. What is the current process for resource demand management? (include timescales if possible). The supplier sources the required resources in accordance to demand and based on test experience and within a 2 week turn around
14. Are you using a third party development supplier or do you require development skills set in this team you are looking to procure? We use 3rd party development suppliers. This procurement is just for test resources
15. You have requested that "TPL & Testers have demonstrable experience of producing and maintaining automated checks using RESTassured & Protractor, JMeter and Gatling (Performance) & OWASP ZAProxy (Security)." Can you please confirm that you will expect this team to be responsible for Performance and Security Testing? Yes, we will expect them to be responsible for all testing including business acceptance, performance and security. In addition, providing assurance when releases are tested through the environment set and into live.
16. Do you expect this team to also be responsible for IT Health checks including penetration testing? No, we have a separate provider who undertakes ITHC and penetration testing
17. Please can you confirm the application to be tested? All capability and applications being built by the programme will require testing and are therefore part of this procurement exercise
18. Are other tools used to deliver the service currently, in addition to those mentioned in the technical competence questions? For example, Test Management and GUI Test Automation tooling? "There is no other ‘testing’ tools used except for what is listed in technical competence section.

JIRA is used for test management and defect tracking - This is common across all teams. For our GUI/E2E testing we have settled on Protractor. Our frontend applications are built using Angular and Protractor was the obvious framework of choice at that time. Tests execute via WebDriver"
19. What will be the earliest start date for this service? 20th January 2020 is the expected start date as stated in the requirement. It is unlikely that the future contract will start before that date.
20. Is there an existing set of automated regression test suites? If so, how large are they (e.g. no. of test scripts/steps), how long do they take to execute and how often are they executed? (Response Part 1)

Automated regression test suites have been built up over the past 3 years. We use the automated test pyramid approach and only cover happy path scenarios in our E2E test journeys. These tests take under an hour to run. Our backend API test suites are much larger and cover negative testing across all our microservices. These can take anything from 1 hour to 2 hours due to the nature of the architecture. - CQRS.
21. (Question 20 Continued )

Is there an existing set of automated regression test suites? If so, how large are they (e.g. no. of test scripts/steps), how long do they take to execute and how often are they executed?
(Response Part 2)

All tests are run against a feature branch prior to merging to master. Test execution also form part of the ‘definition of done’, therefore nothing can be considered for sign-off unless all automated tests have passed. We have setup parallel pipelines in our non-live test environments and the goal is to run all tests nightly.
22. Can you provide an indication of the number of existing digital delivery teams and frontend/backend developers that the embedded test team will be working with? There are 10 development teams that the testers will be working with. They are each of a standard size of approximately 4 back end and 2 front end developers.
23. Will the test team be involved with the direct support and analysis of issues through analysis of queues, error logs, alerts etc. on the live/production estate? Yes as appropriate.
24. Can you provide the number of applications currently supported by the existing digital delivery teams? The digital delivery teams are building and supporting 1 citizen facing on line plea application, with multiple applictations spread across the common platform which makes up the caseworking capability.
25. Can you describe your test environment strategy? Our current test strategy covers all elements of the testing lifecycle, from individual unit and programme integration testing through to external integration testing, business acceptance testing and into live.
26. The service request references to an existing TPL and Test Team. Will the service being requested be additional to or replace the existing resources? If replacement, will there be a requirement or option to take-on / TUPE existing resources? This procurement is being launched as a replacement to the current contract. There is no requirement to retain current resources, this would be a decision between any new and incumbent provider.
27. Is the minimum team 18 or a number below this? 18 is the maximum
28. Are there any TUPE implications when taking over the HMCTS Crime Programme Test Services? No TUPE implications have been identified.
29. Knowledge Transfer period:
Does HMCTS have a preference for how long the transition period (from incumbent supplier to the new supplier) should take?
Will the incoming Test Services provider be paid at the full contract rates, during the period of transition from the incumbent supplier?
The transfer from new to incumbent supplier is to take no longer than 4 weeks. The Programme will pay the incoming supplier directly for any services in the transition period.
30. Working Arrangements:
• Would it be possible for the supplier to base some of the test team onsite for only part of the 24 month contract (i.e. an initial learning period), then move these selected resources to a nearshore location for the remainder of the contract? (The remainder of the test team would be onsite for the full duration of the contract.)
No, the whole test team would need to be on site for the duration of the contract as they are a critical element of the delivery teams performance and are currently embedded across the teams. We do not procure nearshore or off shore for our services.
31. Scope of “Non-Functional Testing” and “Operational Acceptance Testing”:
• Please could you elaborate on what activities you expect to be in scope of Non-Functional Testing? (e.g. Performance testing, accessibility testing, security/penetration testing beyond OWASP ZA etc.)
• Please could you elaborate on what activities you expect to be in scope of Operational Acceptance Testing? (e.g. Deployment and backout testing, server monitoring testing etc.)
In scope for non-functional testing would be:
Performance testing (API and E2E performance).
Accessibility testing to make sure frontend apps meet the WCAG 2.1 AA standards.
Security testing at application level. - Infrastructure level is out of scope.
Cross browser testing.

In scope for operational acceptance testing would be:
Failover/resilience testing of clustered architecture. Both micro service and network failover.
Application recovery and database recovery testing from a specific point in time.
Testing the monitoring and alerting templates to ensure they are working correctly.
Testing deployment process.
32. Test Tools & Assets
• Are the non-functional test tools provided/already in place or is there an expectation that the supplier would need to provide specialist non-functional test tools & licences where required (e.g. load test tools)
• Are the test & defect management tools provided/already in place or is there an expectation that the supplier would need to provide these.
• Do HMCTS retain the IP to existing test assets (automation scripts, non-functional load test scripts and manual test scripts) and will these handed over during transition.
All non-functional test tools for performance, security and accessibility are open source. There is no need to provide specialist tools or licences for these. HMCTS also has a paid subscription for SauceLabs for cross browser testing. This comes at no extra cost to the supplier.

Test and defect management is handled within JIRA/Confluence and access will already be provided to all new starts on the program.

Yes, HMCTS hold the IP to all existing test assets and all be available during and after the transition.
33. Could you please provide any existing KPI's from the test delivery? Existing KPIS are based on Fulfilment, Delivery and Quality of Service.
34. Can you clarify the question about "Root Cause Analysis of Live Features"? Do you mean the RCA of feature defects or incidents discovered in Live? We would expect the test capability to participate in both the feature defect analysis as well as analysis of incidents discovered in live service.
35. Who is responsible for managing this work? The test practice lead which is part of this procurement is responsible for managing this work and the resources.
36. Are you looking for a response from a Consultancy who own a bench of Test Consultants or are you prepared to consider an approach based on interim Consultants available at the time of requirement? We are not prescribing the resources that the supplier provides to fill the roles, however the resource proposal needs to support the requirement for on boarding appropriately skilled and knowledgeable resources. This needs to be within the anticipated 2 week turn around timescale including resource replacements as required.