Foreign and Commonwealth Office Open Source Unit

FCO cloud analysis platform development

10 Incomplete applications

7 SME, 3 large

14 Completed applications

7 SME, 7 large

Published: Wednesday 5 September 2018
Deadline for asking questions: Wednesday 12 September 2018 at 11:59pm GMT
Closing date for applications: Wednesday 19 September 2018 at 11:59pm GMT

Overview

Summary of the work: Building a secure cloud data portal, drawing data from a range of international open sources and providing a suite of user-friendly analysis tools for users.
Latest start date: Monday 22 October 2018
Expected contract length: Until 31/3/2019
Location: London
Organisation the work is for: Foreign and Commonwealth Office Open Source Unit
Budget range: £300-400k based on 3/4 fulltime individuals for the duration of the project.

About the work

Why the work is being done: The Open Source Unit's aim is to best use open data to help the Foreign Office deliver across the whole spectrum of its policy priorities (encompassing many spheres, such as: crisis response, consular services, security and sustainability).
To enable this, we aim to construct a data platform that gives staff and collaborators easy access to a rich selection of open data, and a suite of powerful, user-friendly analysis tools they can use to unlock the most pertinent insights.
The work must be completed by 31st March 2019.
Problem to be solved: Useful data is widely dispersed and cannot be accessed from a single point. Standards vary (e.g. different date formats and regional breakdowns), making it difficult to combine data from different sources.
Data is in different formats (html, csv, APIs, etc.) requires different processing to make it useable on a common platform.
Data is of different types: numeric, text, satellite imagery, metadata, etc.
Specialist tools/code (R/Python) are currently required to perform analysis – we need to abstract this complexity to open analysis to a broader group of users.
The platform must be secure and meet the NCSC’s requirements for official documents.
Who the users are and what they need to do: As an economic researcher, I need to combine and visualise data, for specific regions and time periods, from different sources (e.g. World Bank Open Data, and page view data from the Wikipedia API), so I can make informed inferences when making policy recommendations. I also need to locate pertinent insights from large collections of documents to create briefings, and identify/monitor patterns in time series data which imply that the probability of an event has increased. I need to collaborate effectively with colleagues while completing these tasks.
Early market engagement: OSU have developed a prototype component of the platform, focussing on text analysis. Early feedback has been positive, but a larger scale user testing exercise would be required. Prior development, we conducted a small user research exercise, the main conclusions of which were:
People currently spend too much time traversing various different data sources – highlighting the need for more centralisation
Use of numerical analysis is currently very limited – more access to tools that enable this would be beneficial.
Any work that’s already been done: OSU have developed a prototype text analysis component of the platform. It uses Spacy to perform named entity recognition and tag documents, which were then indexed in Elasticsearch. The tool has a simple front-end, built with Vue.js, which allows users to explore the documents via geographical and metric based interface.
OSU have also harnessed the open source collaboration product, Nextcloud, for inter-team working. We are interested in the possibility of using Nextcloud as an ecosystem for the wider platform, and adding analytical capability by developing additional Nextcloud apps.
Existing team: OSU has three data scientists who would be involved with the data science aspects of the project e.g. machine learning / data wrangling. At this stage we do not expect any additional suppliers to be involved with the project.
Current phase: Alpha

Work setup

Address where the work will take place: Foreign & Commonwealth Office, King Charles Street, London, SW1A 2AH
Working arrangements: We would like the supplier’s team to be present in our offices at least three days per week to facilitate effective communication with the FCO staff also involved.
Security clearance: Existing SC clearance required for all individuals involved.

Additional information

Additional terms and conditions: Standard Contract

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Essential skills and experience: Demonstrate experience of developing user-friendly, visually engaging cloud appications

Demonstrate experience of interacting with Elasticsearch

Demonstrate experience of capturing, standardising and storing data from varied sources in a suitable container (e.g. MongoDB etc.)

Demonstrate experience of developing data-pipelines, e.g. to provide updates from APIs

Demonstrate experience of developing attractive, user-friendly UIs with (Vue/React/AngularJS), including data visualisations and geospacial representations

Demonstrate experience of working in an agile team to deliver software based on user requirements

Demonstrate experience of producing user documentation and maintenance documentation

Demonstrate experience of government cyber security requirements for official data

Demonstrate experience of developing secure user authorisation and user interaction masking

Demonstrate experience of producing documentation to evidence security credentials
Nice-to-have skills and experience

How suppliers will be evaluated

How many suppliers to evaluate

Proposal criteria

Technical solution
Approach and methodology
How the approach or solution meets user needs
•How the approach or solution meets your organisation’s policy or goal

Cultural fit criteria

Work as a team with our organisation
Be transparent and collaborative when making decisions
Have a no-blame culture and encourage people to learn from their mistakes
Take responsibility for their work
Share knowledge and experience with other team members
Challenge the status quo
Be comfortable standing up for their discipline
Can work with clients with low technical expertise

Payment approach

Fixed price

Assessment methods

Written proposal
Presentation

Evaluation weighting

Technical competence

50%

Cultural fit

15%

Price

35%

Questions asked by suppliers

1. Please can you clarify E10 – producing documentation to evidence security credentials. Is this about verifying our people have SC clearance or about security from a software development point of view?: This is about producing documentation that demonstrates the steps taken to ensure that any software/applications developed are sufficiently secure. This will help the FCO’s IT security accreditors sign-off any software/applications before they are launched.
2. You have requested existing security clearance will need to be in place which may limit the number of suppliers who can bid for this opportunity. Would the FCO be willing to sponsor individuals through the SC process?: Existing security clearance is required. Due to the timescales involved in obtaining SC, it is unlikely that any suppliers would be able to complete this process in time for the desired contract start date
3. Please can you confirm if FCO will sponsor individuals who don't yet have SC clearance?: The timescales involved with obtaining new SC clearance are likely not compatible with the timescales of this project.
4. The requirement states: "Existing SC clearance required for all individuals involved." To our understanding such clearance can be only obtained with the support of contractor. Would you be willing provide such support for proposed team?: The timescales involved with obtaining new SC clearance are likely not compatible with the timescales of this project.
5. Please can you clarify : 1) What is the coding technology of choice for the alpha? 2) Is the ambition to expand on the prototype code or to start fresh with the lessons learned? Thank you: 1) Our existing work has been largely based in vue.js, incorporating Elasticsearch
2) We are open minded about how this work is taken forwards. Ideally we would like to build upon existing work, but could be persuaded otherwise if a more optimal solution was convincingly put forwards.
6. Could the Foreign and Commonwealth Office Open Source Unit please confirm whether an incumbent team is in place delivering these services?: There is no incumbent team. In Jan – Mar this year we built a prototype of the wider proposed system with the aid of two contractors who departed after that time.
7. The pricing basis is fixed price. If we respond with a proposal to carry out the work using agile methodology will you consider an approach which includes fixing time and input, therefore flexing scope?: We would be prepared to consider a more flexible approach if the benefits of doing so could be demonstrated convincingly.
8. Will FCO sponsor SC Clearance for members of the team who do not already have it?: The timescales involved with obtaining new SC clearance are likely not compatible with the timescales of this project.
9. What supplier team profile do you envisage, to complement the existing team?: We do not wish to be too prescriptive about this. We are open to suggestions put forwards by suppliers and will select a profile based on merit.
10. Will the supplier be responsible for the delivery management of the team and the outcomes required?: We would encourage suppliers to be active in delivery management.
11. Does the FCO have a preferred cloud platform provider for hosting the portal?: We have previously worked with AWS, but we are impartial on the choice of provider. It is essential that the solution meets to security standards specified by NCSC for official data.
12. Does the FCO have a preferred cloud platform provider for hosting the portal?: We have previously worked with AWS, but we are impartial on the choice of provider. It is essential that the chosen platform meets to security standards specified by NCSC for official data.
13. Please can you clarify: 1) What is the public cloud platform of choice for the alpha? Elasticsearch is mentioned which would indicate Amazon Web Services? 2) For alpha, is the scope to refine the text based analysis capability and add numerical analysis or to expand and encompass more data types as described in the problem statement?: We have previously worked with AWS, but we are impartial on the choice of provider. It is essential that the chosen platform meets to security standards specified by NCSC for official data. Elasticsearch is open source and not exclusive to AWS.

By the end of March 2019, we wish to: refine the text analysis; add numerical analysis capabilities; and, resource permitting, explore incorporating other types of data (e.g. satellite imagery).
14. Please can you clarify what you mean by 'user interaction masking'?: If possible we would like the interactions of users (e.g. queries, search terms etc. that they send to the platform) to be obfuscated from third parties.
15. For the question “Demonstrate experience of developing secure user authorisation and user interaction masking”, could you clarify and elaborate what you mean by “user interaction masking”? In the question “Demonstrate experience of developing secure user authorisation and user interaction masking”, are “secure user authorisation” and “user interaction masking” linked or independent aspects of experience? If they are linked, could you clarify how? Could you clarify further what you mean by “producing documentation to evidence security credentials”? Should this be “evidence security controls”, and if not what meaning is intended?: If possible we would like the interactions of users (e.g. queries, search terms etc. that they send to the platform) to be obfuscated from third parties.

User authorisation (ensuring only verified users can access the platform) and interaction masking (clarified above) are two separate tasks.

By “producing documentation to evidence security credentials” we mean that we hope that the supplier will be able to produce documentation that demonstrates the steps taken to ensure that any software/applications developed are sufficiently secure. This will help the FCO’s IT security accreditors sign-off any software/applications before they are launched.
16. Can you be clear about what you mean by "user interaction masking"?: If possible we would like the interactions of users (e.g. queries, search terms etc. that they send to the platform) to be obfuscated from third parties.
17. Is the full scope of what is described – end-user friendly loading, joining, analysis, visualisation, and abstraction of R/Python models for a wide variety of data sources in structure and unstructured forms – expected to be delivered within the budget range and timescale indicated?: We do expect the full functionality of the system to be delivered on budget within the timescale indicated. If it later transpires that this is not feasible, we would address this by focussing on a smaller sub-set of data sources (while leaving the door open to add more at a later stage).
18. SC clearance for all individuals or exceptionally with FCO?: OK for SC clearance to be with other Government Departments (e.g. Home Office)
19. Is SC eligible acceptable?: No, need the SC, we can sponsor but it could take months.
20. Processing and data - particular open source tools we would use?: We are open minded that other coding would work.
21. Do GDS assessment for the project?: Not putting it through a formal GDS process.
22. View to open the portal to external users?: FCO primary users but broaden to OGDs but currently for FCO users.
23. With respect to authentication/access, are there services we want to use?: We don't know, as could be loosely locked down e.g. FCO IP addresses or may need a full authentication on a user basis but not finalised. Team coming in will help shape this and preference for open source but not committed if a commercial option is the best solution.
24. Are you aware of any rules-based access requirements or will there be different levels of access?: All users can use all parts of the portal. But customise the interface for communities to have an interface - allow users to define their experience.
25. Is it about gathering and curating data or will it be used for identifying patterns?: More former and a bit of the latter. Have a monitoring capacity but not convinced by adding analytics/predictions.
26. Geospatial in document, is that from a visualisation perspective and have geo-spatial hotspots of the data collated?: Partly that, implementing hot-spots analysis and interpret open data feeds to provide a geographical interface. But also adding a geographical search tool, so addition of these in the collation.
27. Some of the data sets - anything that we should be aware of now and are we working on smaller subsets of the data in the alpha stage?: Some data is small, some large (40GB). Would like to harness GDELT data but we have questions over building our own databases or query with Google BigQuery - so flexible as to the best solution.
28. So architectural questions on whether using DB or APIs. Target infrastructure - using cloud hosting or FCO Services?: In the first instance using AWS rather than internal structures as our work is exploratory. So not until it is fully mature will we move to FCO Services.
29. Security considerations, with regards to the SC clearance if use AWS.: There is a NCSC OFFICIAL level template for AWS and this is our intent. At this stage only talking about Open Source data, but wish to have it technically secure to OFFICIAL.
30. Is it the aggregation of the data that requires SC clearance?: Yes, and as well as access to our facilities which makes it practical too.
31. Would you like to have people onsite?: It is helpful to have people present, but we would consider teams wanting to work predominantly work remotely. Requirement is to have people present at least once a week.
32. When it gets around to following best practice and testing?: Yes, we want it scalable, robust and resilient. Want advice for this.
33. Is there a system we need to work with to get it into production?: The team will work independently to IT teams (but collaboration with IT security people), and instead will work with us.
34. World Bank Data is a data source, are we collaborating with them already?: No, it is a collection of data in the public domain which will be helpful to the FCO policy priority areas.

One aspect we want to mention - a previous piece of work was done using the VUE framework. Is this the most suitable framework to use moving forward or is ANGULAR/REACT better?
35. Is that due to personal preference?: Yes, the contractor liked it.
36. Is there an investment for re-working the front end?: We have a prototype of the front end which we aren't committed to. An example to recreate and would have recyclable components (e.g. queries for elastic search).
37. Please can you clarify Q10, are you referring to the management of security credentials within a system, or the provision of documentation for security assurance purposes?: This is about producing documentation that demonstrates the steps taken to ensure that any software/applications developed are sufficiently secure. This will help the FCO’s IT security accreditors sign-off any software/applications before they are launched.
38. Can you be clear about what you mean by "user interaction masking"?: If possible we would like the interactions of users (e.g. queries, search terms etc. that they send to the platform) to be obfuscated from third parties.
39. Is the full scope of what is described – end-user friendly loading, joining, analysis, visualisation, and abstraction of R/Python models for a wide variety of data sources in structure and unstructured forms – expected to be delivered within the budget range and timescale indicated?: We do expect the full functionality of the system to be delivered on budget within the timescale indicated. If it later transpires that this is not feasible, we would address this by focussing on a smaller sub-set of data sources (while leaving the door open to add more at a later stage).

Can we store analytics cookies on your device?

Awarded to Scott Logic Limited