Start date: Wednesday 1 April 2020
Value: £598,400
Company size: SME
Office for National Statistics

PySpark and Agile Technical Support

8 Incomplete applications

7 SME, 1 large

9 Completed applications

8 SME, 1 large

Important dates

Monday 6 January 2020
Deadline for asking questions
Monday 13 January 2020 at 11:59pm GMT
Closing date for applications
Monday 20 January 2020 at 11:59pm GMT


Summary of the work
ONS requires a small team of PySpark (essential) & Amazon Web Services (desirable) software developers (x3) and an Agile Project Manager (x1) to join the Trade and Investment Development Project team.
Latest start date
Wednesday 1 April 2020
Expected contract length
1 year plus 1 year optional extension
Organisation the work is for
Office for National Statistics
Budget range
Maximum day rate of £680 (plus VAT) for each team member.

Total contract value for 2020/21 must not exceed £830K.

About the work

Why the work is being done
With the UK due to leave the European Union in 2020, the focus on trade and investment statistics has never been higher. The ONS Trade and Investment Development Project was established in 2016 in order to meet the user needs of key stakeholders across government, particularly Department for International Trade (DIT). At the heart of this project is the need to develop and transition both new and existing processing systems on the office's strategic technology.

Throughout 2020/21 ONS requires the services of a supplier who can provide a small team with expert technical skills in Cloudera and Amazon Web Services software development and Agile Project Management. This team will be integrated into an existing project team and work through an established product backlog to deliver system capability to a business as usual team who are responsible for the publication of new and improved trade and investment statistics for our users.

This small team should consist of:

3 x Cloudera (essential) & AWS (desirable) software developers
1 x Agile Project Manager
Problem to be solved
With the UK leaving the EU in 2020, the government will be negotiating new bilateral trading partnerships across the globe. In addition to this, wider industry has a need to understand and monitor the levels and impact on current UK trade and investment.

In order to meet these objectives effectively, we require more diverse and granular data and statistics and therefore the systems and capability to deliver these.

The ONS strategic platform for statistical production is based on both PySpark by Cloudera and Amazon Web Services technology. Within Economic Statistics Group, where this project is based, there is currently a shortage of staff with expert skills required to take forward the development, test and production of statistical pipelines at pace.
Who the users are and what they need to do
As a development manager, I need to ensure the project team has expert skills in PySpark and Amazon Web Services software development and Agile Project Management, so that we can develop, test and productionise statistical transformation pipelines at pace.

As a business as usual team member, I need to be able to utilise newly developed statistical pipelines to analyse and publish new trade and investment statistics, so that my users have the right insight to meet their needs.

As a user of trade and investment statistics, I need to be able to have access to the latest trade and investment data from ONS in a timely manner, so that I can understand the current picture and use this to inform future policy direction.
Early market engagement
Not applicable
Any work that’s already been done
The Trade and Investment Development Project has been in-flight since 2016 and produced a range of system, data and analysis outputs to date.

The project has a well-established team comprising of Agile Project Managers, Product Owners, Technical Leads and specialist delivery teams all working from a defined product backlog. It has developed a Cloudera cluster, which is highly performant and requires programmes written in PySpark. Data has already been loaded onto this cluster. An efficient development cycle involving test and continuous integration already exists.

It is governed by the wider Economic Statistics Transformation Programme (ESTP) via project and programme boards.
Existing team
Rob Pike - Deputy Director, Global Trade & Investment (Project Executive)

Luke James - Assistant Deputy Director, Global Trade & Investment (Senior Project Manager)

Mel Gore - Global Trade & Investment (Product Manager)

Leigh Skuse - Global Trade & Investment (Agile Project Manager)

Remainder of the project team comprises of 5 distinct specialist delivery teams (circa 5 - 8 ppl) ranging across data analysis, methodological design and system development.
Current phase

Work setup

Address where the work will take place
Office for National Statistics
Government Buildings
Cardiff Road
NP10 8XG
Working arrangements
The supplier is expected to work alongside the existing project team, on site in Newport.

The project is governed using agile principles and practices and therefore the supplier is expected to take a full role in these (incl. working in sprints, presenting at show and tells, contributing to retrospectives).

The supplier will also be expected to work openly, taking part in team events where necessary and use digital collaborative tools to engage with the wider project team.
Security clearance
Security Clearance (SC) is required for the 3 x PySpark & AWS developers

Additional information

Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Essential skills and experience
  • Developers – strong experience in development of relational database systems using the Cloudera stack (particularly Pyspark, Hive and Impala SQL) 20%
  • Developers – able to produce technical documentation to support the development of production systems 5%
  • Developers – experienced in development of Extract Transform and Load (ETL) pipelines 10%
  • Developers – experience of developing continuous integration solutions (particularly in Jenkins) 3%
  • Developers - experience in version control and software review using GIT 2%
  • Agile Project Manager – strong experience of leading large multi-functional project teams within a statistical environment 20%
  • Agile Project Manager – expert at planning complex cross-organisational project deliverables, including the management of stakeholders and dependencies 10%
  • Agile Project Manager – expert in managing risk and removing blockers which could impede delivery 5%
  • All team member – proficient in the use of Jira for work item management 5%
  • All team members – experience of working across the full development lifecycle 5%
  • All team members - awarness of good data security procedures and adherence to ONS secure information management policy 5%
  • All team members - ability to work collaboratively with members in other technical roles including Business Analysts and Methodologists 10%
Nice-to-have skills and experience
  • Developers – experience of data management 40%
  • Agile Project Manager – certified Scrum Master (or similar) 40%
  • Agile Project Manager – Prince2 Practitioner (or similar) 20%

How suppliers will be evaluated

All suppliers will be asked to provide a written proposal.

How many suppliers to evaluate
Proposal criteria
  • How the approach or solution meets user needs
  • How the approach or solution meets your organisation’s policy or goal
  • Value for money
Cultural fit criteria
  • Be comfortable working at pace
  • Works to overcome blockers without allowing them to derail progress
  • Work as a team with our organisation and other suppliers
  • Be transparent and collaborative
  • Coach other team members including sharing technical knowledge and experience
  • Works closely with stakeholders and takes collaborative decisions
Payment approach
Time and materials
Additional assessment methods
Work history
Evaluation weighting

Technical competence


Cultural fit




Questions asked by suppliers

1. Is there an incumbent?
Yes, contract expires 31 March 2020.
2. Do you expect pre-clearance on the security side, or are you able to put people through SC?
Pre-clearance would be preferable and is expected however we are able to put people through SC if they meet all other requirements but do not have SC clearance in place.
3. Can the candidates work remotely (full time or several days per week)?
Candidates will be expected to work on site at ONS in Newport, South Wales for the vast majority of time. There is flexibility to allow candidates to work remotely however this is expected to be around 1 day per week on average.
4. Can you elaborate "experience of data management" requirement?
For this requirement, we would expect candidates to be proficient in the management of data including (but not limited to)

Building ETL pipelines,
Version control of data assets configurations and code,
Tracking and updating of metadata.
5. What's the procurement timeline?
Following closure of the first stage expression of interest on 20.01.20 the responses will be sifted and those shortlisted will be invited to submit a proposal via our Intend portal (second stage) circa 27.01.20 closing 04.02.20 with award decision due around 21.02.20
6. Can you sponsor SC for right candidates?
ONS can sponsor and cover the initial baseline cost. Costs incurred over and above this should be covered by the service supplier.
7. Does the rate include agency fees?
8. Is there an incumbent supplier of this service at present?
9. Will SC clearance need to be in place already or would you be willing/able to support this process?
Pre-clearance would be preferable and is expected however we are able to put people through SC if they meet all other requirements but do not have SC clearance in place.
10. You need 3 devs, but this opportunity allows 100 words max against their skills, which is normally used for ONE person. Do you want us to enter just one of them here? Wouldn't it be easier if you split this one in 4 different opportunities instead?
Whilst the brief explains the current team is made up of 3 x Developers and 1 x Agile Project Manager, this is an outcome specification. Please answer in terms of your business rather than specific individuals.
11. Does this statement refer to one year or two years?: "Total contract value for 2020/21 must not exceed £830K."
Refers to one year 01.04.2020 - 31.03.2021.
12. Is the contract deemed to be outside IR35?
The IR35 assessment tool can only be conducted once bids that include details of individuals have been sifted/selected.