The Driver and Vehicle Licensing Agency (DVLA) – ITS Service Creation

PS-19-79 Test Engineering Service

Incomplete applications

14
Incomplete applications
11 SME, 3 large

Completed applications

26
Completed applications
18 SME, 8 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Thursday 2 May 2019
Deadline for asking questions Thursday 9 May 2019 at 11:59pm GMT
Closing date for applications Thursday 16 May 2019 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work DVLA requires a collaborative supplier able to provide a flexible capacity of highly skilled Test Engineers to support us in the delivery of software solutions, working closely with our Development Teams to, ensure solutions meet customer requirements, are defect free and robust.
Latest start date Monday 15 July 2019
Expected contract length 12 month contract with the option to extend for a further 12 months.
Location Wales
Organisation the work is for The Driver and Vehicle Licensing Agency (DVLA) – ITS Service Creation
Budget range The total value of all Statements of Work commissioned under the proposed call-off agreement will not exceed £2.0m.

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done DVLA are building new services, designing them from direct customer engagement, moving away from our legacy technology onto new, cloud-based platforms utilising modern technologies like containers and serverless.

DVLA needs Test Engineering Services to support its Digital Transformation Programme. This service needs to be able to provide the technical skills and experience to be held accountable for the design, creation, execution and analysis of functional and non-functional tests of components used by DVLA’s services.
Problem to be solved DVLA require the delivery of a Test Capability, provided as a service via individual Statements of Work (SOWs). Solutions delivered under SOWs need to ensure components being created or amended are fully tested as part of the development lifecycle. Capability provided under SOWs will need to have the ability to create and automate tests using a ruby/cucumber, create and improve acceptance criteria from user stories, analyse test results and advise software engineer on defects, possible causes and possible solutions. All test artefacts need to be supported by DVLA team and must therefore conform to DVLA standards.
Who the users are and what they need to do Business Stakeholders, Analysts, Software developers, testers and engineers.
Early market engagement There has been no early market engagement.
Any work that’s already been done
Existing team DVLA have in place a number of digital development teams working on a range of different projects and business as usual activities using Agile methodology.
Current phase Beta

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place The work will take place in DVLA, Morriston, Swansea.
Working arrangements The supplier is expected to work from the DVLA Swansea Offices.
Security clearance BPSS with Basic Disclosure (issued by Disclosure & Barring Service).

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions We currently see this role as sitting outside of IR35, as the Supplier would not hold a fixed office position and will not have direct day to day direction or supervision.

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Have demonstrable experience of supplying test services to large enterprise organisations.
  • Have demonstrable experience of providing test services in a multi-vendor landscape.
  • Have demonstrable experience of providing test staff at multiple skill levels and roles profiles. (SET, SDET, Analyst, Test Management)
  • Have demonstrable experience of providing test services with experience of working in an evolving team structure and changing requirements.
  • Have demonstrable experience of supplying a test service able to enforce best practice design processes.
Nice-to-have skills and experience

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 10
Proposal criteria
  • Explain your overall testing Approach and Delivery Methodology.
  • Describe how you will ensure that DVLA is supplied with high quality and appropriately skilled Testers at various skills levels.
  • Explain how you plan to retain key resources and how you can commit to meet DVLA's continuous need for test services.
  • Provide up to three referenceable client-focussed case studies where your organisation has provided the desired service capability.
Cultural fit criteria
  • Work as a team with our organisation and in a clear and transparent manner.
  • Take responsibility for their deliverables and work.
  • Be open and collaborative when making decisions.
  • Share knowledge and experience with other team members.
Payment approach Time and materials
Assessment methods
  • Written proposal
  • Case study
  • Work history
  • Reference
  • Presentation
Evaluation weighting

Technical competence

60%

Cultural fit

10%

Price

30%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Can you advise if you have a current dedicated supplier for testing services or whether this is provided by multiple vendors? There is currently one incumbent Supplier providing some of DVLA’s Testing Services.
2. Who is currently providing test services? DVLA’s Testing Services are currently provided by an incumbent Supplier and internal capability.
3. Please can you confirm if you will accept propositions from companies where workers will be based in Spain for the majority of the project? Please refer to DOS requirement - The supplier is expected to work from the DVLA Swansea Offices, this includes all resources that would be provided to meet the requirement.
4. Is there a current incumbent vendor? If yes, please could you tell us who the incumbent supplier is? DVLA can confirm that there is a current incumbent supplier.
5. How old BPSS & Basic Disclosure Certificate is acceptable by DVLA for suitable candidate? A current BPSS & Basic Disclosure Certificate will be required.
6. Will this opportunity be awarded to one supplier or will multiple partners be engaged? This will be awarded to one supplier.
7. Could you confirm if there is a current incumbent? DVLA can confirm that there is a current incumbent supplier.
8. Could you confirm who is evaluating? i.e. position, please. The evaluation panel will be comprised of DVLA staff within the IT Directorate.
9. Is there a current incumbent and if so will they be reapplying? DVLA can confirm that there is a current incumbent supplier who will alongside all other registered Suppliers be welcome to apply for this opportunity via the Digital Marketplace portal.
10. Is there any work already done in past or this is completely a fresh assignment? Any challenge which vendor needs to work on? This can be viewed as a new requirement. However, the supplier will be working alongside existing in-house teams with well-established delivery processes and ways of working and therefore will have to work to DVLA standards on testing.
11. Can you please give us the brief idea of the technology to be used in these projects? Test engineers will need to develop tests using the ruby/cucumber framework. Tests will be built into the CI/CD pipeline using Drone and Spinnaker. Developers will be using java, ruby on rails and react.js. Understanding of Docker and Kubernetes is required. Data is persisted using Postgres, mongodb and AWS S3. Working knowledge of Jira is required.
12. In case of On-boarding of new candidate, what is the lead time expected from vendor? Candidates are expected to be on-boarded within two weeks of each Statement of work.
13. Explain your overall testing Approach and Delivery Methodology – we are assuming that DVLA is looking for approach towards catering of both functional & non-functional requirements. Please confirm For this assignment Test Engineers will be working within squads developing tests as part of standard sprint cycles. DVLA are working towards ATDD, so tests start from story development via Gherkin acceptance criteria which are then automated using ruby/cucumber. Test code is managed in the same way as code under test and automation is driven via the CI CD process. This includes functional and non-functional tests (e.g. running OWASP/ZAP scans and JMeter performance tests) on deployment.
14. How many roles will need to be deployed on 15th July 2019? How many roles will be required within the first month? The supplier would need to provide this at the proposal stage.
15. Given that the project(s) are already in beta, are existing suppliers already testing and is the supplier just providing point test resources, or will the supplier be given SOWs with responsibility for all testing on particular projects? Supplier will be given SOW for a specific product being developed and tested within a discrete squad. Supplier may test alongside a DVLA Test Analyst.
16. We note the max £2m budget, which would buy a reasonable size test team for 12 months. Are you able to give an approximate idea of how many test teams (and their size) that you would require initially from mid-July? The supplier would need to provide at the proposal stage.
17. Do you an estimate of the numbers required and also the structure please? The supplier would need to provide at the proposal stage.
18. Is the use of contract resource permitted? The Supplier is required to provide appropriately skilled and BPSS cleared resource, how the Supplier sources this is at their discretion.
19. How does this link with the automation and Performance RFP’s released at the start of April 19? This is a separate requirement.
20. Does the entirety of the service need to be delivered onsite in Swansea or can elements be delivered from supplier delivery centres in the UK. Please refer to DOS requirement - The supplier is expected to work from the DVLA Swansea Offices, this includes all resources that would be provided to meet the requirement.
21. How is success of the engagement measured? Are there KPI’s? By delivery of SOWs for each discrete piece of work to be delivered.
22. What are the timeframes between resource requests and them being required onsite? Candidates are expected to be on-boarded within two weeks of each Statement of work.
23. Following on from our last question. Could you advise if content is missing from the portal as only a few questions have been published? No content is missing all answers to questions will be published.
24. Could the Authority confirm what "SET" stands for in the question – Have demonstrable experience of providing test staff at multiple skill levels and roles profiles. (SET, SDET, Analyst, Test Management)? SET is Software Engineer in Test, SDET is Software Development Engineer in Test.
25. Budget range – Is that for the first 12months or for the whole 12+12 period should the option be taken? The budget range is for the full 12 +12 month period.
26. Could the Authority provide the current team structure at DVLA? DVLA operates a squad structure with squads aligned to one or more products/components. The squad is led by an agile delivery manager (ADM) and includes a senior software engineer, software engineers and testers. The ADM works to a Technical Product Manager, under guidance of the Principal Agile Coach. Testers work to the ADMs but have to adhere to test standards set by the Principal Quality Engineer.
27. What is cloud technology stack at DVLA – AWS, Google or Azure? DVLA has a multi-cloud strategy, currently we predominantly use AWS and Azure.
28. What are the DVLA Standards for test assets? Standards will be provided in detail once on-boarded. For the purpose of bidding please assume they relate to the following areas;
1. Ruby version
2. Cucumber version
3. Docker images used for Drone builds
4. DVLA internal gems & looking at reusing best code already developed
5. Coding standards
6. Cucumber standards
7. Reporting
8. The kinds of testing performed and where, e.g. OWASP, Accessibility
9. The significant 3rd party gems in use
29. Any preference on Testing Tool set? (e.g. Micro Focus or JIRA with Zephyr) In most cases Jira. Zephyr is being phased out and other products tested.
30. What non-functional testing is within the scope? Some local performance, security vulnerability scanning, some operational readiness testing (tests of monitoring and alerting).
31. Does all of the Testing Services have to be undertaken onsite at DVLA or can the services be delivered using an onsite/offsite/offshore model? Please refer to DOS requirement - The supplier is expected to work from the DVLA Swansea Offices, this includes all resources that would be provided to meet the requirement.
32. Can we use Indian Offshore Test Engineers onsite at DVLA (after having gone through their BPSS check)? Please refer to DOS requirement - The supplier is expected to work from the DVLA Swansea Offices, this includes all resources that would be provided to meet the requirement.
33. What are the major technologies/systems that will be developed as part of DVLA’s Digital Transformation? DVLA are transforming our legacy estate, due to restrictions with word count for further detail please see the published DVLA IT Strategy.
34. What Tools do DVLA already have in place, or if none are there any preferences for the supplier to use for the following areas:
o Test Management/Defect Management/ALM
o Automated Regression Testing
o Load & Performance Testing
o Continuous Integration/Containers/DevOps
Test engineers will need to develop tests using the ruby/cucumber framework. Tests will be built into the CI/CD pipeline using Drone and Spinnaker. Developers will be using java, ruby on rails and react.js. Understanding of Docker and Kubernetes is required. Data is persisted using Postgres, mongodb and AWS S3. Working knowledge of Jira is required.
35. When completing the application on the portal, we only have viability of 5 questions is this correct? All answers to all questions will be published as soon as possible.
36. How many resources / how many Agile teams are you resourcing for? Is this number fluid or static throughout the engagement, and if expected to flex, do you have visibility of the minimum / maximum resource numbers? The Supplier will need to propose the resource to satisfy the requirements in each SOW. We expect this to be fluid throughout the contract and called off by separate SOW’s.
37. Can you advise the range of applications / technologies expected to be under test? Which cloud-technology is being utilised? Test engineers will need to develop tests using the ruby/cucumber framework. Tests will be built into the CI/CD pipeline using Drone and Spinnaker. Developers will be using java, ruby on rails and react.js. Understanding of Docker and Kubernetes is required. Data is persisted using Postgres, mongodb and AWS S3. Working knowledge of Jira is required.
38. Are we going to be testing in-sprint alongside the developers – true in-scrum tester-developers – or as a post-sprint separate testing activity? Supplier will be given a SOW for a specific product being developed and tested within a discrete squad. Supplier may test alongside a DVLA Test Analyst.
39. Which non-functional testing aspects are in scope? E.g.:
• Performance / load / stress / efficiency
• Security
• Accessibility / Usability / UX
• Compatibility
• Availability / reliability
• Operational Acceptance / maintainability / scalability
• Failover / Recovery
Some local performance, security vulnerability scanning, some operational readiness testing (tests of monitoring and alerting).
40. Is the Ruby/Cucumber framework mentioned in the tender already in place? Yes.
41. In terms of either test automation or performance testing, are there any tools, other frameworks or other solutions already in place? Test engineers will need to develop tests using the ruby/cucumber framework. Tests will be built into the CI/CD pipeline using Drone and Spinnaker. Developers will be using java, ruby on rails and react.js. Understanding of Docker and Kubernetes is required. Data is persisted using Postgres, mongodb and AWS S3. Working knowledge of Jira is required.
42. Are there any DevOps processes currently being followed? E.g. CI/CD, production monitoring, etc. Tests will be built into the CI/CD pipeline using Drone and Spinnaker.
43. What are the DVLA standards for test artefacts? Can we get early visibility of this please? Standards will be provided in detail once on-boarded. For the purpose of bidding please assume they relate to the following areas;
1. Ruby version
2. Cucumber version
3. Docker images used for Drone builds
4. DVLA internal gems & looking at reusing best code already developed
5. Coding standards
6. Cucumber standards
7. Reporting
8. The kinds of testing performed and where, e.g. OWASP, Accessibility
9. The significant 3rd party gems in use
44. When you state: “fully tested as part of the development lifecycle” will this be qualified in the definition of the scope of the statement of work? Yes, this will be detailed within each Statement of Work (SOW).
45. The requirements states, “All test artefacts need to be supported by DVLA team and must therefore conform to DVLA standards.” Will these need to comply with the DVLA’s internal testing quality engineering standards? Can there be more clarity on the scope of artefacts and ownership? Standards will be provided in detail once on-boarded. For the purpose of bidding please assume they relate to the following areas;
1. Ruby version
2. Cucumber version
3. Docker images used for Drone builds
4. DVLA internal gems & looking at reusing best code already developed
5. Coding standards
6. Cucumber standards
7. Reporting
8. The kinds of testing performed and where, e.g. OWASP, Accessibility
9. The significant 3rd party gems in use
46. Where would the staff of the winning supplier be located for the delivery of this work? Please refer to DOS requirement - The supplier is expected to work from the DVLA Swansea Offices, this includes all resources that would be provided to meet the requirement.
47. “Describe how you will ensure that DVLA is supplied with high quality and appropriately skilled Testers at various skills levels.” Can you offer more detail on the skill levels required? The skills levels will need to be appropriate to deliver the requirements of each agreed SOW and meet the essential skills criteria.
48. “Provide up to three referenceable client-focused case studies where your organisation has provided the desired service capability.” Are three different clients required? As long as these are three discrete and distinct case studies then the same or mixture of clients would be acceptable.
49. Is there another part to this tender? There only seem to be 5 question boxes relating to our experience, but nowhere to record responses for the other parts in the tender which are listed as “assessment methods”, including a “written proposal”, “case study” and “reference”. Is there a fault on the portal? Or will there be another stage to complete after this? The assessments methods required will be evaluated at the proposal stage.
50. How many roles will be required within the first month? Requirement is initially for 3 SDET and 3 Test Managers.
51. Are you able to give an approximate idea of how many test teams (and their size) that you would require initially from mid-July? Requirement is initially for 3 SDET and 3 Test Managers.
52. Can a more detailed overview of DVLA's resource ramp-up plan be provided, along with detail as to the initial engagement? Initial engagement would be through a SoW for approx. 6 to 12 weeks. It will be for the Supplier to determine the resource requirements to deliver the SoW but for the purposes of this proposal the equivalent DVLA team would initially be approx. 3 - 6 test roles (3 SDET and 3 Test Managers).
53. What is the roadmap for the project? Have you identified any underlying technology changes that we need to be aware of over the next 12-24 months that will affect any solution we implement? Several projects may come into scope for this contract. The only underlying technology change that is worth pointing out is a move towards a serverless architecture (e.g. AWS Lambda) which will sit alongside dockerised services running on Kubernetes.
54. What are the current interface interaction points? (API, Web, Desktop, SQL Database?) Most components will have a Restful API as the interface. All digital services are presented to customers via html5 interfaces which will consume underlying services via their APIs.
55. Will the short-listed price evaluation be on rate-card only? Yes price evaluation will be based on rate card. At proposal stage please provide a rate card as per the roles described within the requirement (3 X SDETs, 3 x Test Analysts, 1 x Test Manager). This is indicative of the types of roles required but number required will flex up/down throughout the project.