SurfaceNet Collate Platform
5 SME, 2 large
3 SME, 5 large
Monday 26 February 2018
Deadline for asking questions
Monday 5 March 2018 at 11:59pm GMT
Closing date for applications
Monday 12 March 2018 at 11:59pm GMT
Summary of the work
Using AWS the project will deliver a range of functionality including:
ETL data flow engine with flexible rules driven quality control processes
Data migration / conversion of existing data
Metadata querying and production e.g. calibration and quality flags
Delivering visualised data to consumers in near real time
Early life support
Latest start date
Wednesday 30 May 2018
Expected contract length
South West England
Organisation the work is for
In the region of £1,000,000.00
About the work
Why the work is being done
The current on premise system for the collection & processing of surface sensor data is reaching its end of life. The Met Office are looking to leverage Amazon cloud platform technology to provide a flexible, scalable and resilient data processing platform to meet customer requirements around timeliness and quality of data.
Observations data is at the heart of everything that the Met Office undertakes including climate research and the initiation and validation of forecasting models. Raw observations contribute to the national archive, scientific research and international exchange of data and improved access to this data is a key project driver.
Problem to be solved
The project needs to be able to collect and disseminate observations data to allow existing systems to be retired while also meeting rules around quality and timeliness.
Automated tools to improve UX and deliver efficiencies.
Ingest raw data from a range of sources.
Transform data through calibration and a flexible data QC rules engine.
Load data for consumption in agreed API formats
Handle comprehensive Metadata.
Meet NFRs such as security, timeliness, reliability, and WMO/CAA etc. standards.
Ensure Continuity of existing services and data.
Use AWS tool sets where possible
Who the users are and what they need to do
Direct consumers of the data will include:
Quality control team.
Each grouping will have some bespoke requirements and may have slightly different perspectives on the same core requirement or element this will drive a significant degree of user flexibility to be incorporated into the system design.
Early market engagement
Any work that’s already been done
A detailed requirements capture process has informed a high level system architecture. The concept has also undergone peer review from industry experts. Rules around processing metrological data are well defined with definition from the WMO and others.
A Proof of Concept has been undertaken with AWS IOT team to demonstrate core functionality.
The project sits as part of a wider program of work including a new data gateway and is underpinned by the subject matter experts within the observations department.
Initial hardware testing has been undertaken on integration of the field infrastructure with the AWS platform.
The Met Office team includes:
Hardware Test Engineer.
Met Office developer staff will also be assigned to ensure transfer of skills during development to ensure future operating capability.
Timely and appropriately skilled resource to complement and support the supplier team as and when required will be made available in order to remove impediments or provide domain knowledge.
Address where the work will take place
Met Office, Exeter, EX1 3PB. The majority of work is expected to take place on site to ensure robust knowledge transfer.
The supplier will deliver through Agile methods and will be able to work closely with Met Office teams. It should be noted that the project is required to meet a series of stringent regulatory requirements. The supplier will be expected to use the Met Office’s own systems such as JIRA (Confluence, Stories on Board) etc.
Work on the Met Office premise or its IT environment will require SC.
Additional terms and conditions
It is the Met Offices intention to hold the full IPR for all systems components and at the completion of the project all source code, business rules and documentation should be provided to the Met Office.
Skills and experience
Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.
Essential skills and experience
- Experience/evidence of working on AWS utilising PaaS based tools and technologies to implement modular solutions to provide ETL functionality.
- Experience of working alongside a customer team to deliver an integrated platform (ensuring knowledge transfer).
- Ability to use Agile methodologies while complying with mandated international (meteorological) standards (provided).
- Experience of dealing with data with prescriptive data processing and timeliness rules.
- Ability to surge resources to support agreed project plan.
- Experience of working with and designing scalable APIs for a wide variety of end users.
- Utilise the most up to date IoT toolsets and understanding of IoT Security Foundation best practice.
- Digital design skills with a focus on user experience.
- Experience using Continuous Integration development approaches.
Nice-to-have skills and experience
- Experience of delivering projects within the environmental monitoring sector ideally from a dispersed network of remote monitoring solutions.
- A proven partnership with Amazon delivering AWS cloud hosted solutions, for large organisations.
- Proven experience of working with government sector organisations to deliver projects with complex regulatory requirements including international standards and governing bodies.
How suppliers will be evaluated
How many suppliers to evaluate
- Track record in similar applications
- Experience with working with customer teams and integrating with existing platforms
- Approach and methodology
- How data latency can be managed effectively
- Key risks and dependencies of your proposed strategy and mitigations you would put in place to manage them
- Proposed Team structure and capabilities on a phase by phase basis
- Organisational capacity to provide additional or flexible capability aligned to the scheduled deliverable
- How knowledge will be imparted to ensure the MO can manage operation of the system into the future including examples of this from previous projects
- Value for money
Cultural fit criteria
- Alignment to the GDS digital principles with a ‘user first’ focus Embedded agile alignment
- A culture of openness and collaboration, a willingness and understanding of the principles involved in being a ‘strategic partner’ rather than a supplier
- A flexible approach to communication, especially in the promotion of agile principles
- A willingness to support co-location and face to face working
- A cohesive approach to working with internal capabilities
- Evidence of thought leadership in the creative and strategic space
- Focus on open knowledge transfer to ensure future Met Office internal support
Capped time and materials
- Written proposal
- Case study
Questions asked by suppliers
1. Reference: "The majority of work is expected to take place on site to ensure robust knowledge transfer". Can suppliers propose a hybrid approach in which the majority of the work takes place at the premises of the supplier and supplier's staff meet frequently with the Met Office's project team?
The majority of the work is to take place on the MO premises in Exeter to allow pair programing and robust knowledge transfer.
We expect the supplier to be embedded with our staff.
2. Apart from AWS, what other technology stack does the project wish to use e.g. Java, Python?
The core technologies we use are:
Java (in AWS Lambdas)
Spring Boot for micro services
AngularJS for web applications
Python is also used when appropriate.