This opportunity is closed for applications

The deadline was Friday 24 July 2020
UK Centre for Ecology & Hydrology

PL077-20: NERC Data Labs Phase 3

6 Incomplete applications

4 SME, 2 large

3 Completed applications

2 SME, 1 large

Important dates

Published
Friday 10 July 2020
Deadline for asking questions
Friday 17 July 2020 at 11:59pm GMT
Closing date for applications
Friday 24 July 2020 at 11:59pm GMT

Overview

Summary of the work
Provide a flexible team comprising Business Analyst, implementation team, Test Lead and Project Coordinator to support UKCEH's existing Data Labs leadership and implementation teams.
Latest start date
Monday 20 July 2020
Expected contract length
12 months with an optional extension of up to 12 months
Location
South West England
Organisation the work is for
UK Centre for Ecology & Hydrology
Budget range
The estimated value of this opportunity for the initial period is a maximum of £350,000.00 excluding VAT.

Should the optional extension be utilised, the total value shall not exceed £700,000.00 excluding VAT.

About the work

Why the work is being done
The goal of Data Lab is to extend NERC's Environmental Data Service offering through utilisation of existing NERC Data Centres as data analytics service providers; allowing researches to have accessible, secure, flexible, cloud-based research environments, combining dynamic big data resources, flexible analytical capabilities, and opportunities to collaborate.

Phase 3 of the project allows Data Lab to scale out, in storage and compute resources (on-premise and hybrid-cloud) to meet the operational need of our research communities. As Data Lab expands, so the need to manage scale and complexity of users, content and services will increase, requiring simplified administration and robust data discovery.
Problem to be solved
In supporting the anticipated operational scale of Data Labs (in terms of both infrastructure and complexity of content) five key deliverables must be realised:

1. A platform management interface allowing administration of users, infrastructure and data assets.
2. Access to large, performant and highly scalable storage supporting big data analytics.
3. A federated Data Labs asset discovery service.
4. Implement a job scheduling interface enabling access to HPC resources in JASMIN’s managed-cloud infrastructure.
5. The ability to work beyond the confines of Data Labs current hosting environment (CEDA's JASMIN) if an activity demands it.
Who the users are and what they need to do
1) As a Researcher/Scientist, I need to undertake computationally intensive modelling, simulation and data analysis activites, and/or access research outputs (data, visualisations, etc.) that are computationally demanding to produce, and beyond the capabilities of my personal laptop, workstation or research centre's own infrastructure.
2) As a scientist, I may need to collaborate with other scientists, make use of previous assets, or make my assets discoverable for future researchers, through a federated Data Labs asset discovery service.
3) As a system administrator, I need to be able to manage a growing infrastructure and user base through simple administrative interfaces.
Early market engagement
n/a
Any work that’s already been done
Data Lab phase 1 (2017) delivered a cloud-based solution (hosted on JASMIN) allowing scientists to access an intuitive self-service, highly scalable data management and computational analytics capability. This phase focussed on delivering on-demand Jupyter and Zeppelin analysis environments and RShiny-based web applications as demonstrators of the Data Labs approach, underpinned by dedicated private storage supporting parallel data processing by Apache Spark and Dask.

Data Lab phase 2 (2019) facilitated increased research collaboration and enhanced lab security through project-based environments. The range of data analysis and publication services also increased to include RStudio.
Existing team
The project team includes representatives from the UK Centre for Ecology & Hydrology (UKCEH), STFC's Centre for Environmental Data Analysis (CEDA) and end user research scientists from academia, NERC Research Centres, and wider UKRI Councils.

The Supplier will primarily work alongside UKCEH's project leadership and implementation team with significant collaboration also anticipated with infrastructure managers at CEDA.

Liaison and engagement with a core group of Data Labs Champions across NERC and UKRI more widely is also anticipated, facilitated by UKCEH's project leadership team.
Current phase
Beta

Work setup

Address where the work will take place
The majority of work will be undertaken from the suppliers own premises, through remote connectivity and collaboration with the UKCEH team (and other stakeholders as required). UKCEH have well established remote collaboration tooling covering project communication/reporting and technical development activites. The successful supplier will be expected to align with UKCEH's approach.

When conditions permit travel to UKCEH research sites (Lancaster, Wallingford) and CEDA (STFC Harwell) are anticipated on an ad-hoc, part-time basis. Occassional travel to other project partner locations may be required in support of community engagement and solution roll-out activities.
Working arrangements
The Supplier will largely work remotely and report daily to the project implementation team.

UKCEH advocate the Scrum agile project management approach. The successful supplier will be expected to engage in UKCEH's Scrum methodology, including participation in its range of project execution ceremonies.

A summary of progress, supplier team utilisation and budget tracking will be provided as a written report at the end of each implementation sprint by the supplier.

The Supplier will provide consistent representatives in key roles throughout the project's lifetime, although flexibility within the wider team reflecting needs of the project as it progresses may be necessary.
Security clearance
Security clearance is not required

Additional information

Additional terms and conditions
The supplier will incorporate flexibility into the timetable to enable End User involvement to assist in the development of user stories, acceptance testing, and UKCEH-led engagement activities.

The Supplier will accommodate breaks in the timetable for customer led end user engagement, decision making and governance.

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Essential skills and experience
  • Management of large scale open source projects
  • Experience of developing self-service data analytics platforms in both academic and commercial research environments, providing cloud-hosted compute and storage access
  • Experience of cloud-based technologies and the CNCF landscape; including OpenStack, Teraform, Kubernetes, Docker, Auth0, Vault, GlusterFS, and MinIO
  • Experience of parallel data processing: Dask and Apache Spark
  • Experience of execuable notebook technologies: RStudio, Jupyter and Zeppelin
  • Experience in web-based data exploration tools: RShiny
  • Experience of developing cloud-based APIs in relevant technologies: GraphQL
  • Appreciation of common research tools (e.g. R, Matlab, IDL, Python, Java Applications, Audio and visualisation techniques), data formats (e.g. NetCDF, CSV, GRIB, Binary, WAV) and processing needs
  • Software testing (automated and manual testing) including use of Selenium, Swagger, Postman
  • Experience in user experience user interface implementation
  • Experience in metadata design and standards implementation: Dublin Core, DCAT, Schema.org; and application to FAIR data principles
  • Experience in implementing data catalogue solutions including relational and NoSQL-based approaches: PostgreSQL, SQLlite, MongoDB, ElasticSearch
  • Experience of providing software solutions that meet complex and varied scientific requirements for exploration, manipulation and visualisation of data
  • Large data volume software solutions for batch and parallel processing
  • Leading and working in teams using Scrum and other Agile techniques
  • Cooperating and collaborating with 3rd party academic research organisations such as STFC (JASMIN) and NERC
Nice-to-have skills and experience
  • Have a strong scientific or technical background (e.g. a Scientific degree or higher)
  • Engagement with user communities
  • Understanding of business process planning and requirements to support provisioning, management and administration of a Data Lab
  • Experience of delivery of agile projects in a mixed supplier / client team

How suppliers will be evaluated

All suppliers will be asked to provide a written proposal.

How many suppliers to evaluate
3
Proposal criteria
  • The proposed technical solution
  • The proposed approach and methodology
  • How the approach or solution meets user needs
  • Team structure and organisational makeup
  • Value for money of the proposed solution
  • How they have identified risks and dependencies and offered approaches to manage them
  • Flexibility to delivery the proposed approach to ensure end user engagement
Cultural fit criteria
  • Work as a team with members of the project and end users to produce the best outcome
  • Transparent and collaborative
  • Collaborating and delivering value in an environmental science research environment
  • Good record keeping and tracking of progress
  • Be comfortable discussing complex and varied end user requirements with scientists and technicians and confident in managing discussion and keeping focus
  • Contribute to the ongoing development and improvement of the Data Lab
  • Take responsibility for their work
  • Share knowledge and experience with other team members
  • Challenge the status quo
  • Can work with clients with low technical expertise
Payment approach
Capped time and materials
Additional assessment methods
  • Case study
  • Presentation
Evaluation weighting

Technical competence

60%

Cultural fit

20%

Price

20%

Questions asked by suppliers

1. The deadline for applications is Friday 24 July 2020, but the latest start date is listed as Monday 20 July 2020. Should the start date be the 20th of June?
This is an error on the form. The preferred start date is as soon as possible, at the latest the end of July.