Department for Work and Pensions

Cloud Native Architecture Migration

Incomplete applications

23
Incomplete applications
18 SME, 5 large

Completed applications

26
Completed applications
17 SME, 9 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Tuesday 7 May 2019
Deadline for asking questions Tuesday 14 May 2019 at 11:59pm GMT
Closing date for applications Tuesday 21 May 2019 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work Complete design of Cloud Native Architecture for existing service. Provide design documentation. Support governance processes. Build pipeline and tooling to allow secure operation of service and infrastructure. Support ITHC and remediation. Our estimates suggest a delivery team of 6 including 4.5 DevOps, 0.5 Tech-lead and 1 delivery management/BA.
Latest start date Monday 1 July 2019
Expected contract length Contract length: 2 years, with current expectation that work will last approximately 9-12 months.
Location London
Organisation the work is for Department for Work and Pensions
Budget range

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done We are continuing on the architectural journey for one of the department's strategic applications - the micro service based application is currently hosted in the cloud, the infrastructure is provisioned using infrastructure as code with deployments moving through an automated pipeline, we are now seeking to ensure we use our hosting services as economically as possible and to take more advantage of the features cloud computing afford, leveraging cloud native architecture and deployment patterns where appropriate.
Problem to be solved Current architecture is restricting our ability to make progress on developments to meet some non-functional requirements (NFRs) - such as zero downtime releases. We are also not using the cloud hosting platform as efficiently as we would like and do not yet have an optimal mechanism to allow auto scaling.
Who the users are and what they need to do The product owners and Senior Leadership Team for the service need to meet NFRs and cost management objectives.
Early market engagement A study and early proof of concept has indicated that the use of containers as a deployment platform is the preferred approach to adopting a Cloud Native Infrastructure.
Any work that’s already been done Work has started by a team from ThoughtWorks, evaluating existing container use within the project and other areas of the Department to assess strengths and weaknesses, work is prioritised to evaluate security monitoring tools and their suitability for inclusion in a pipeline that will deploy containers to run the microservices. This current work is expected to result in a proof of concept pipeline.
Existing team Working as a self managed delivery unit with product ownership provided from within the Department, the team will work alongside a team of Developers, QAs, Infrastructure, DevOps and SRE engineers who run the current platform. The team will be responsible for the development of the new process, the existing team will provide knowledge of the application and existing environment.
Current phase Not applicable

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place Caxton House, Tothill Street, London SW1H 9NA
Working arrangements On site in Caxton House - all equipment will be provided by DWP.
Security clearance Security Check clearance will be required for all team members.

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Please provide evidence of experience designing and implementing secure, resilient, highly available configurations of infrastructure and microservice based application components using containers and clustering technoligies within AWS.Weighting: 2%
  • Please supply evidence of experience of automation of cloud deployments and developing infrastructure as code for AWS using terraform. Weighting: 2%
  • Show evidence of experience creating testing frameworks to support cloud native deployments including simulation of failure scenarios. Weighting: 5%
  • Please detail your experience of architecture developments / migrations implemented on existing large microservice based applications. Weighting: 2%
  • Capacity to supply appropriate resources inc BA/Delivery management, DevOps/developers (>3 yrs’ experience), Terraform, docker, kubernetes, AWS, puppet, github, bash, python, java, Mongo, ActiveMQ, jepsen, microservice architectures, commodity cloud environments:5%
  • Show experience of working on significant digital projects within DWP or a comparable organisation within the last 3 years. Weighting: 2%
Nice-to-have skills and experience Show evidence of your ability to provide technical leadership and support governance process as a self managing delivery unit within a Government project. Weighting: 2%

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 3
Proposal criteria
  • Provide details of where your resources have delivered architectural change to a live cloud based system using agile methodologies. 10%
  • Experience working on the evolution of business critical cloud based services for a comparable customer. Please clarify how your resources added value, delivery focus and technical leadership. 10%
  • Projects involving deploying large microservice based apps using containers in AWS supporting zero downtime releases and optimising cloud resource utilisation. Technologies used, any security, other considerations influencing approach taken. 10%
  • Please describe your experience of and approach to ensuring technical aligninment of activities with other parallel work streams across multiple locations, ensuring quality standards are maintained. 5%
  • How have you ensured that you quickly gain a detailed technical understanding of complex systems and of the project requirements to allow rapid involvement in design and decision activities. 5%
Cultural fit criteria
  • Please provide two examples of similar projects that you have been involved in that demonstrate your ability to respond to and align with the culture of the project. 5%
  • Please describe the challenges presented by operating as a self-managed delivery unit within a broader agile based project team and your approach to ensuring success. 15%
Payment approach Capped time and materials
Assessment methods
  • Written proposal
  • Case study
  • Work history
  • Reference
  • Presentation
Evaluation weighting

Technical competence

60%

Cultural fit

20%

Price

20%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. How do we join the Q&A telephone conference? Meeting Details:
Dial-in number: 0330 336 6013
Guest/Participant Passcode: 768527#
To remain anonymous: When requested to say your name, wait for the tone then press then hash key without saying your name. You will then join the conference anonymously.
2. When (date/time) is the telephone conference pls? Please log in to view question and answer session details .
3. Do you have a budget in mind for this project? We will expect suppliers to quote competitive rates given the volume of work for the team of people as per the scope described.
4. Following Q&A #2, I have logged in to the Digital Marketplace but still don't see the date/time Scroll down the requirement page. Beneath the section titled "How suppliers will be evaluated" there is a section called "Question and answer session". If you are logged in, the details of the date and time for the call will be visible in this section. If you are not logged in, you will be prompted to log in to view the details.
5. When logged in, we have scrolled down beneath the "How suppliers will be evaluated" section, but there are no details of the date and time for the call. When is the the call please? Thanks. The Q&A session has now taken place. All questions and answers from the call will be made available via the digital marketplace.
The details of the Q&A session were made available in the section titled "Question and answer session" when logged in. If you are not logged in, the wording in the "Question and answer session" section states "Log in to view question and answer session details". You should click on the words which are hyperlinked to the log in page.
6. Clarification requested on scope - is it expected to include application changes as well as platform changes? The remit is restricted to the infrastructure platform, but where developments by the team will need application changes e.g. configuration, micro service deployment through the different types of environment, there will be a need to work with existing teams within the project. Teams will be made available where appropriate to enable this work.
7. Is the existing platform performing or is this requirement seeking to address performance issues? We don’t have current performance issues. We are supporting over 2m users in production and will be scaling to 8m users. Ongoing release assurance includes full scale performance tests before every release to ensure we test volumes a few months ahead of existing volumes. There is also regular 6-months ahead testing to predict bottle necks. This project is not correcting any existing performance problems.
8. Do you have a release schedule? We currently release every week. These alternate between a fortnightly technical release and a major release. Occasional major releases can be made on an ad hoc basis if needed. This work will help take us towards zero downtime releases, with functionality released to the business via toggles.
9. Do current releases include all micro services? Yes, major releases include all micro services. The evolvement aim is to only release micro services which have been changed. The current tooling is capable of this, but there are gaps to close. This work will help achieve that. This is not a fixed objective of this piece of work.
10. What are the languages of micro services? Micro services are java. Java is the only language of micro services.
11. What is the current hosting platform? The service is hosted in AWS.
12. What cloud native technologies are currently used? Cloud native technologies currently used are the core AWS services; e.g. EC2, VPCs, EBS. We are not currently using auto scaling and any containerisation.
13. Do you consume cloud native application services? We consume some AWS cloud native infrastructure services such as EC2 instances, EBS etc. We are not using application services or RDS.
14. Please can you confirm the languages of the micro services? Java.
15. What is the platform for downstream applications? The service has integration points with other platforms which the department hosts. The integration service is managed by the department.
16. Is there any tooling used for downstream applications e.g. ERP? The interfacing from the service is to other DWP internal systems e.g. CRM systems providing internal info. APIs are internal and custom written.
17. Is there any preferred CICD tooling used by DWP? We use Jenkins for CI/CD. GitHub is also mentioned in the requirements.
18. Kubernetes - is this already being used as a homebrew implementation or is this just the future direction? Containerisation platforms are still under review by this project and the wider DWP. Given the complex landscape, including security, and the fact that this is an area of rapidly developing new technologies, this project not actively using Kubernetes but it is an option for the future.
19. Is the expectation that the platform will be self-contained or will it be integrated with existing tools that have already been selected? The suppliers team will not be able to choose their own tools – this will be part of the existing platform. As this is an evolutionary project, key operational tools and processes will need to be respected, any changes suggested to help support changes introduced by this work would involve working with the DWP product owner to decide whether that provides appropriate value for project.
20. Please can you clarify security clearance and working location requirements? The working location is Caxton House, Tothill Street, London. Project is agile and thrives on collocated collaboration. The majority of people and teams are based within building and working closely together.
The expectation is that all staff, particularly all technical staff will need SC clearance. It will be helpful if suppliers already have this, however it is not a specified requirement that this must already be in place. The department will sponsor security check if needed, but team members must be eligible for SC.
21. Is it correct that no architects or security specialists are listed in the requirement? There is an embedded security team within the project who work alongside the rest of team in all aspects, so no security specialist will be required as part of this opportunity. All engineers will need to have a security-based mind-set as this is a large service with very sensitive data. The technical lead within the team would be expected to work together closely with our architects in DWP.
22. Re: CICD. Is the infrastructure deployed by pipeline and does the pipeline have tests wrapped round it? The infrastructure platform is not as mature in testing pipelines, but it is a major push for us at the moment. Configuration management around puppet is rapidly evolving in the area of testing automated by Jenkins. We are looking at similar arrangements for terra form. It would be of benefit if supplier has experience in these areas. For clarification, infrastructure does go through a Jenkins pipeline with tests, and these tests are evolving.
23. What level of testing is there of the application pipeline? In terms of scale: we have over 2 million lines of java code and 80 thousand automated tests covering java code. Each time there is a build in Jenkins, all tests are run every time. The build time for end to end test of an application is 20 minutes.
24. Team setup: why is DWP going outside of the team for this? In order to utilise the specialist skills in this area of technology, testing and automation that can be provided by suppliers. There is also a need to have a focussed product delivery unit working solely on this, without also managing BAU.
25. What are you considering around transition off the project at the end? The goal would be that as changes are delivered, the necessary tooling and knowledge will be handed over to the teams operating the service day to day.
26. Are there any operational systems that we need integrate with as part of release process e.g. service now? This is not applicable for this piece of work. Internally within the project we use Jira for majority of things - e.g. release and agile stories.
27. Is the expectation that team will work for fixed price outcome? Capped Time and materials was stated in the DOS opportunity.
28. Is the expected platform to be deployed on existing established AWS accounts or new accounts? The expectation will be that it will be into existing accounts, as that will make migration approach simpler.
29. Regarding CICD: will there be a move from Jenkins to Jenkins x through the lifecycle? As part of this piece of work, if the supplier team identifies improvements to or changes to CI/CD toolset, then that team would work with the DWP product owner to decide whether that provides appropriate value for project.
30. Is this platform independent from all other DWP platforms? Broadly yes. There are integrations to other systems in the department and dependencies on mutual understanding of changes and their impacts. However, this platform is sufficiently independent to move and make change at its own pace.
31. Do you have expectations of seniority in terms of roles and resources? Expectation of 3 years’ minimum experience for core technical people, and we would expect that the technical lead would be a more experienced individual.
32. Do you have a requirement to remain hybrid cloud independent or agnostic or would you be considering PaaS via AWS? The service is currently hosted in AWS with the infrastructure code creating and configuring AWS specific services. This work is expected to take us nearer to being able to deploy to any provider using a vendor agnostic platform.
33. Is the full team of 6 required to start straight away on day one or will the tech lead / senior architect do some design work first before scaling up? The goal is to progress rapidly through an initial engagement cycle through to the detailed planning and execution stages. We would expect that for the engagement phase the tech lead, delivery management and one devops resources would be a minimum. The engagement period should be for a maximum of 2 weeks.
34. You specify failure scenario's in the essential question on testing frameworks. Are there any specific or typical NFRs you are concerned about or could share? The key NFRs that we are concerned about during common cloud failure scenarios are availability and response times.
35. The questions and answers asked during the session are not available even when we log in. Are the questions and answers on this page the ones which were asked during the session? The opportunity was published on 7th, then cancelled and published again on 7th or 8th with a Q&A on the 8th. I'm not sure how many suppliers would have been able to respond so quickly and attend the call. All questions and answers from the call have been published on the requirements page in the section titled "Questions asked by suppliers".
36. You mention ThoughtWorks have done some initial work, are they set up to tender for this opportunity? Buyers do not have any visibility of which framework suppliers have applied to an opportunity until the closing date for applications has passed.

There are over 1,200 suppliers on the Digital Outcomes and Specialists framework. Opportunities are published on the Digital Outcomes and Specialists opportunities page. To be eligible to apply for an opportunity, it has to be in a category the supplier chose when they applied to the framework, and the supplier has to confirm they have all the all the essential skills and experience listed.

For further information on the Digital Outcomes and Specialists Framework please visit: https://www.gov.uk/government/collections/digital-marketplace-buyers-and-suppliers-information
37. “Please detail your experience of architecture developments / migrations implemented on existing large microservice based applications. Weighting: 5%'' – How would you define ''large'' in this context? To give an idea of the scale of the service that the successful bidder will be working on, it consists of approximately 120 different micro-services, there are a minimum of 2 of each of these micro-services running at any time. The service currently supports > 2million citizens (this will grow to > 8million) and provides the day to day work platform for > 8000 internal users. We are seeking teams with experience of working on the evolution of applications with significant levels of complexity and use.
38. ''Show evidence of experience creating testing frameworks to support cloud native deployments including simulation of failure scenarios.'' Usually it is the deployment of the applications that are tested using various frameworks, please could you elaborate on what ''cloud native deployments'' mean here?
Response part 1 of 2.
As the underlying infrastructure for this service is defined in infrastructure code and is deployed through a pipeline, we are seeking to develop the testing to ensure that all aspects of the service are behaving as expected and continue to allow us to meet our NFRs. This may include, for example, ensuring that all network and security configurations are correct or that all load balancing and scaling configurations are in place.
39. ''Show evidence of experience creating testing frameworks to support cloud native deployments including simulation of failure scenarios.'' Usually it is the deployment of the applications that are tested using various frameworks, please could you elaborate on what ''cloud native deployments'' mean here?
Response part 2 of 2.
The team will work with the existing teams responsible for both the automated and scheduled testing to identify tests that are appropriate to be applied to the infrastructure in isolation and those that would need to be applied to the application / infrastructure components combined.
40. What is the flexibility with the resource numbers? Do the 0.5 resources mean part-time or a mixed skill set resources? The Tech Lead is expected to be performing both a technical leadership function and hands on engineering work, hence 0.5 Tech Lead and 0.5 DevOps engineer is the same person rather than part-time roles. The suggested team break down is based on previous experiences of self-managed delivery units operating within the overall project. If you believe that a different team make up would be beneficial, you should provide background information to support your proposal.
41. How the resource numbers estimated? What assumptions have been made about the seniority of the resources? The resource numbers were estimated based on previous experiences of deploying self-managed delivery units within the overall project - this experience has shown that a team of this size is able to work effectively on work of this type. If you believe that a different team make up would be beneficial, you should provide background information to support your proposal. We stipulate >3 years’ experience using the core technologies for the devops roles with an expectation that the tech lead function would require a more senior individual.
42. The DOS framework has a restriction that we're able to provide only one example per bid question. For questions that have requested for experience in multiple technologies (such as Terraform, docker, kubernetes, AWS, puppet, github, bash, python, java, Mongo, ActiveMQ, jepsen, microservice architectures, commodity cloud environments), we will try as far as possible to quote one project example that encompasses all, but are you happy for us to quote more than one example to evidence all technologies requested? The question you identify is intended to confirm that you are able to deploy, in the required timeframe, individuals with appropriate skill levels in a range of appropriate technologies and / or an internal network to allow for access to further expertise. Drawing information from more than one project to evidence this capability is fine.