Greater London Authority (GLA)

GLA Test Management

Incomplete applications

13
Incomplete applications
9 SME, 4 large

Completed applications

32
Completed applications
26 SME, 6 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Thursday 1 June 2017
Deadline for asking questions Thursday 8 June 2017 at 11:59pm GMT
Closing date for applications Thursday 15 June 2017 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work The GLA has a requirement for a range of functional and non-functional testing. These tests will require junior and/or senior level resources. The types of testing includes unit, regression, smoke, A/B, performance and vulnerability. GLA is interested to hear of Supplier capability to make available user experience/research testing resources
Latest start date Monday 3 July 2017
Expected contract length
Location London
Organisation the work is for Greater London Authority (GLA)
Budget range

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done The GLA Technology Group manages a large range of technical and digital projects on an ongoing basis. These projects require a range of test resources on a regular basis. The nature of testing varies from simple functional testing to more complex and technical testing such as regression, performance and vulnerability testing.

The specific nature and functionality of project outputs and products tends to vary according to the business requirements but the technology “stack” used is relatively standardised to the following:-

Websites*

Drupal CMS (version 7)
PHP code
Ubuntu operating system
MySQL database
Cloud hosted mainly Amazon
*the primary website www.london.gov.uk
Problem to be solved The GLA anticipates that some resources will be needed more frequently and for longer durations than others. GLA principle officers will plan these requirements with suppliers in good time but we expect that any role / resource requested will be available within 2 weeks of a request

Roles required are Senior Tester, Junior Tester, Technical tester, Security Tester, User research/experience. The exact number of each resource is unknown at this time, so the GLA are looking for day rates for each resource and to create a call off contract to engage resources are required

Further information - davidbyrne1@tfl.gov.uk
Who the users are and what they need to do GLA Technology Group needs to perform testing on various systems and websites for new developments and changes to the software.
Early market engagement
Any work that’s already been done
Existing team GLA Technical Group developers and business represenatatives
Current phase Discovery

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place City Hall. Queens Walk, London
Working arrangements To be decided. It is anticipated that testing will be done on site, but there may be scope for remote testing
Security clearance

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Conduct unit testing
  • Conduct regression testing
  • Produce detaile dtest scripts
  • Produce test plans
  • Input to sprint planning and estimations
  • Train users
  • Run workshops
Nice-to-have skills and experience

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 5
Proposal criteria
  • Approach and methodology
  • Value for money
  • How the approach meets the GLA needs
Cultural fit criteria
  • Work as a team with the GLA Technology Group
  • Take responsibility for their work
  • Have a no-blame culture and encourage people to learn from their mistakes
  • Share knowledge and experience with other team members
Payment approach Time and materials
Assessment methods Written proposal
Evaluation weighting

Technical competence

60%

Cultural fit

10%

Price

30%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Do you have any existing test tools or would we be responsible for recommending tool sets as appropriate to budget and need? We currently make use of test tools such as Phantomjs and Selenium but would welcome the supplier recommending others as appropriate per project.
2. Do you anticipate automation testing as well as manual or is it up to us to decide upon project commencement? We very much anticipate automated testing as well as manual testing. Since this call off contract will apply to more than one project, the testing methodology will be decided on a project by project basis.
3. What is the budget and time frame for this role? The timeframe is the same as the duration of the contract i.e. 2 years. The budget has been published as £50,000
4. Do you have any roles descriptions that give a bit more of an in-depth description of the role titles provided in the details? Not at this stage unfortunately. We are primarily looking for suppliers to provide appropriately skilled and experienced resources to carry out the various types of testing set out. The following link may help: https://www.gov.uk/service-manual/technology/quality-assurance-testing-your-service-regularly
5. Are you considering these roles/the call off contract as outside of the Intermediaries Legislation (IR35)? Correct
6. Are there any specific testing tools that you will require experience of? Not specifically. We use Selenium and knowledge of that tool would be an advantage.
7. Will any of the resources require Security Clearance? Individuals who are SC or CTC cleared would be preferred but not essential.
8. With regards the budget being sent at £50,000 is that realistic? 2 years agreement, needing various skill levels? One resource for a year would exceed the budget and the tender mentions multiple resource required There will be no requirement for continual resource, this will be a call off contract where resources will be called on as and when required. The budget is now set at £100,000
9. When you ask about user training are you referring to training in the application or in the process of testing? Very much in the process of testing. One of the outcomes we seek is to up-skill GLA staff, particularly where they are product owners / managers in testing in an agile development environment
10. Is there a reason why system testing, system integration testing, accessibility, localization and operational acceptance testing are not listed as key testing types? The testing specified is not an exhaustive list and were quoted as examples. The type of testing will be specified when the supplier is notified of the requirement
11. Is any of the work delivered in a V-Model methodology? • Not specifically
12. Are you working in an agile methodology? If so is this Kanban, SCRUM or a derivative? Yes, we work using Agile methodologies and in particular, the best practice set by the Government Digital Service (GDS / gov.uk). We don’t exclusively use a particular implementation of Agile (e.g. Scrum) but incorporate the best elements of methodologies where they suit a particular project.
13. Can you further clarify that the budget for this work is £50K for 2 years work as this does not seem achievable for the scope listed? The budget has been revised and will now be £100k
14. Is the 100k flexible as this does not seem realistic for a 2 year term? Can you give us any more insight to what you think your 100k budget will pay for / service provide. Since this is a call off contract, the GLA wants to act flexibly to bring in resources on a short term basis as projects dictate. We don’t envisage that we will have test resources deployed at the GLA on a full time basis. We therefore currently think £100K is sufficient but will keep the contract costs under review as the contract progresses
15. Since the plan is to deploy testers on multiple projects, are we anticipating any lean period or 0% utilization between different projects? Could you share a high level resource deployment schedule? We don’t have a resource plan drawn up yet but there is the possibility of 0% testing resource utilisation at times.
16. Since the plan is to deploy testers on multiple projects, are we anticipating any lean period or 0% utilization between different projects? Could you share a high level resource deployment schedule? We don’t have a resource plan drawn up yet but there is the possibility of 0% testing resource utilisation at times.
17. Do you currently have an incumbent testing partner? If yes, then what are they key drivers to engage with a new partner? There is no incumbent supplier, but we have used external testing companies in the past.The GLA project teams will be the principal points of engagement by suppliers
18. Could you confirm if all tool licences cost will be borne by GLA? Yes, subject to approval by the GLA prior to purchase
19. For onsite requirements, will GLA provide laptops, PCs etc. or if the supplier has to arrange their own? The GLA can supply desktop PCs or laptops
20. Does the test team access QA environment residing within GLAs’ closed network? Yes
21. Is UAT, Accessibility & Automation testing also in scope of requirements? Yes
22. Performance Testing – is it only limited to running Load test & sharing reports or vendor is expected to do everything like monitoring, profiling, endurance test etc. This will be dependent on individual projects but Suppliers should expect to provide a full range of performance testing services
23. What is the anticipated scope of Non-functional testing? There may be an element of non-functional testing but the majority of testing is expected to be functional, security and performance testing.
24. Are you expecting to implement Continuous Integration? This is something we are aiming for but don’t yet have firm plans or timescale for.
25. Does the supplier setup complete test management for projects? The supplier will work with GLA officers and development partners but will be expected to be leading in setting up test management as required for a particular project.
26. Requirements says - It is anticipated that testing will be done on site, but there may be scope for remote testing. For remote testing would you consider near-shore or offshore or both? We work in an agile method and would prefer at least 50% testing to be carried out onsite. We would consider near/off-shore only if there was a compelling reason to do so.
27. Could you please elaborate what type of Workshops and training are expected to be run? These will primarily be on up-skilling GLA staff (especially product owners / product managers) on how testing should be performed methodically and in an agile environment. The rationale is that, particularly once a project is completed but smaller scale development work will continue, the product owners and managers should have a high level of self sufficiency in testing incremental developments ahead of releasing to production.
28. Who would be the users / attendees ? A variety of GLA staff at all levels of the organisation
29. What are your expectations from Unit test? We expect there to be unit testing as part of the development cycle for a project, coupled with regression testing ahead of a release to Production.
30. Will GLA engage in the screening part of the resources ? Are you expecting to see CVs and case studies as a part of proposal submission? The proposal submission is through the Digital Marketplace and with the limitation of 100 words, CVs will not be included. It may be a requirement for individual work packages
31. You mentioned that the testing will be on-site with an opportunity for remote testing. Is a proposal based mainly on remote resources a showstopper? Will the remote resources be able easily access the GLA's network to perform the necessary tests? Most of the GLA’s digital projects will be hosted in AWS and access will be provided if remote resources are agreed by the GLA.
32. What is the typical length of a project? Variable but tend to range from 3 – 12 months
33. If not following a specific Agile methodology, can you provide a breakdown of your sprint cycle? Nature of the sprint cycles tend to vary as dictated by the project but generally:

2 weeks in duration

Unit testing starts around a week into the sprint (we have nightly deploys into the test environment)

Regression testing and bug fixing takes place on the last three days of the sprint

Releases to production takes place on the last day of the sprint or 1-2 days into the next sprint

There are standups on a regular basis, often daily

Sprint planning takes place a week before the start of a sprint

Retros take place at the end of a sprint
34. What kind of projects are we talking about (desktop site, back-end, mobile, apps, ...)? Will the testing be specific to one type of projects? Currently, 90% of developments are websites (desktop and mobile) and internal / external facing web based products e.g. land planning systems. The GLA is keen to move into more app development work. The Supplier procured from this is unlikely to be required to carry out testing on back end development work but there may be an element of user experience testing for content editors, website administrators etc
35. You mentioned that the testing will be on-site with an opportunity for remote testing. Is a proposal based mainly on remote resources a showstopper? Will the remote resources be able easily access the GLA's network to perform the necessary tests? er. Not a showstopper but as previously set out, if the supplier is to provide remote resources, they will need to demonstrate a clear financial benefit to the GLA in so doing. We will also require a UK/ London based technical contact point for any escalations and for working togeth
36. In the additional document that you have sent you have mentioned that response to the RFP is to be sent by 15th June, where as on Digital Marketplace states that you got to evidence for initial shortlisting on 15th The GLA are expecting a response via the Digital Marketplace that demonstrates the supplier’s offering and which may be used to evaluate and award a contract, in accordance with the terms of the Digital Marketplace process. However, the GLA do reserve the right to contact any respondent for further clarifications as deemed necessary.
37. It has been stated that only 5 suppliers will be evaluated, on what basis would shortlist these 5 suppliers to evaluate? We will evaluate in accordance with the following criteria
Proposal criteria Approach and methodology
• Value for money
• How the approach meets the GLA needs
Cultural fit criteria • Work as a team with the GLA Technology Group
• Take responsibility for their work
• Have a no-blame culture and encourage people to learn from their mistakes
• Share knowledge and experience with other team members
38. Since the budget is less would you consider Hybrid Model of delivery (Onsite+Offshore) to give you the best cost optimisation. Also, it is easier to have a flex bench offshore that can be ramped up and down based on the requirement, almost immediately. What is the flexibility/ comfort of the team to work with an offshore QA team? We will evaluate any proposal based on onshore / offshore resourcing on its merits and in accordance with the published criteria. In particular, we would like to see how the supplier proposes to meet the Cultural fit criteria if proposing an off-shore team. For an offshore team, we will also scrutinise the value for money criteria.