Office for National Statistics

PU-17/0128 - Data Source Collection and Assessment

Incomplete applications

0
Incomplete applications

Completed applications

9
Completed applications
4 SME, 5 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Friday 26 August 2016
Deadline for asking questions Friday 2 September 2016 at 11:59pm GMT
Closing date for applications Friday 9 September 2016 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work ONS are looking for a dynamic, highly skilled, collaborative supplier to work on the Alpha of its non-survey collection (part of the Data Collection Platform). We need a supplier capable of delivering significant technical change at pace in a data focused environment using Agile techniques
Latest start date 30/09/2016
Expected contract length A minimum of 3 months.
Location Wales
Organisation the work is for Office for National Statistics
Budget range n/a

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done ONS is creating a small number of platforms that will be shared across the organisation (and in some cases across Government) to deliver various data journeys across the statistical value chain (based on the Generic Statistical Business Process Model - GSBPM). This work concerns part of the collection platform and will enable the secure loading of data from (ultimately) a variety of non-survey (i.e., not interactively collected by ONS) external sources.
Problem to be solved Create the alpha (prototype) of various products that constitute the non-survey data collection elements of the overall Collection Platform. Three products are involved: (1) Import External Data - import data source and execute malware checks,(2) Data Source Assessment - apply rule based structural data assessment and (3) Data Exchange - load and transform data for further processing. These are products within the ONS EA framework. The alpha stage is as defined within the GDS Service Design Manual.
Who the users are and what they need to do As a(n) external supplier I need to electronically pass data for an approved data source and signed off data journey So that the data source can be made available for pre-processing and assessments. As a(n) internal user I need my data source and metadata to be loaded into an accessible repository if it does not contain malware So that the pre-processing stage can continue. As a(n) internal user I need to be able to confirm that the data are structurally correct So that downstream processing can proceed. Other user stories have been identified.
Early market engagement n/a
Any work that’s already been done High level discovery has been completed which identified high level user needs, Epic stories, EA Product/Platform mappings and high level Service Definitions (which contain the high level user stories referred to above). Of course during the alpha phase more detail will be inked in with a focus on the minimum requirement. The Business Analyst involved in the discovery will carry forward into the alpha phase also.
Existing team The supplier will be collaborating with multiple existing (and to be created) multi-disciplinary product teams. The supplier will provide User Research resources as well as technology development capability Each team will include a Delivery Manager, Product Owner, Technical Lead, Business Analyst and Statistical Methodologist. The teams are made up of 3rd party delivery partners and ONS permanent staff.
Current phase Discovery

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place The work will be based in ONS' Newport (South Wales) Office - there is a possibility some travel will be required to ONS' Titchfield site, Fareham, Hampshire.
Working arrangements The team must work alongside existing teams building products to strategic drivers and objectives. They will collaborate with other teams across the service and beyond and will invest time and effort into knowledge sharing, coaching and mentoring across individuals in order to ensure increased capability of ONS people. They will be co-located with technical and business colleagues, operate using Agile tools and techniques, actively participating in Agile ceremonies including service show and tells. They will be expected to be on-site 5 days per week.
Security clearance All team members must be in possession of a current Disclosure Scotland clearance prior to starting contractual work.

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions Framework Ts and Cs will apply

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Expertise in solutions that utilise SOA and micro services - 3 points.
  • Expertise in development/delivery using Cloud Foundry (Python, Scala, Java) - 3 points
  • Expertise in the design and delivery of Restful API's (inc: security functional and non-functional requirements) and use of appropriate tooling such as Swagger/RAML. - 2 points
  • Expert in cloud technologies: design, planning. security, integration, whilst remaining Cloud provider agnostic - 2 points
  • Experience of effective identification and use of open source software components - 2 points
  • Experience in relational and non-relational databases in multiple formats - 2 points
  • Experience in using automated (testing) environments and continuous integration, delivery & (potentially) deployment, particularly GitHub and Jenkins - 2 points
  • Expertise in research and analysis of user needs - including those related to Assisted Digital and accesibility - and translating them into product needs - 2 points
  • Expertise in Agile delivery (e.g., Scrum, Kanban etc.) as outlined in the Government Service Design Manual using BDD and TDD - 3 points
  • Experience of delivery in Data Engineering and Data Management Services, especially in a sensitive data environment - 3 points
  • Proven track record of delivery and integration of services in 'brownfield' & 'greenfield' estates - 2 points
  • Expertise in coaching/mentoring less skilled people, leaving a legacy - 2 points
  • Expertise in Full Cloudera (Apache Spark etc.) Stack - 2 points
  • Expertise in workflow and integration solutions especially involving API gateways - 2 points
Nice-to-have skills and experience
  • Experience of working across government agencies - 1 point
  • Security knowledge of AWS, Azure or locally deployed on VMWare infrastructure - 1 point.
  • Experience of creating solutions using 'R' - 1 point

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 3
Proposal criteria
  • Technical Solutions = 5 points
  • Approach & Methodology = 5 points
  • Approach the approach or solution meets user needs = 5 points
  • How the approach or solution meets the organisation policy or goal = 2 points
  • Estimated timeframes = 2 points
  • Identify risks and dependencies, approaches to manage them = 2 points
  • Team Structure = 2 points
  • Value for Money = 2 points
Cultural fit criteria
  • Work as a team with our organisation and other suppliers - 5 point
  • Transparent & Collaborative when making decisions - 5 points
  • Have the user and customer needs at the core of decision making - 5 points
  • No blame culture and encourage people to learn from their mistakes - 5 points
Payment approach Time and materials
Assessment methods
  • Written proposal
  • Case study
  • Work history
  • Presentation
Evaluation weighting

Technical competence

60%

Cultural fit

20%

Price

20%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. If the work starts on 30 September what is the planned completion date? The intention is for the alpha phase to last 3 months subject to progress.
2. Can the analysis aspects of this work be done offsite with assistance from online collaboration tools? Naturally all workshops would still occur onsite. ONS' experience has been that this approach has not worked well in the past and with agile collaborative working the intention is for the teams to be fully onsite.
3. Following on from question 2 (about undertaking analysis offsite), the benefits of this approach would be a lower cost to the tax payer as expenses would be reduced and increase competition as more partners could supply. Staff undertaking the work would also benefit as they could spend more time with families and relaxed staff benefits ONS. There are numerous examples of this model working with different vendors and customers. If references could be provided would ONS reconsider its position about 100% onsite work? Is there some other objective such as staff training inside ONS? Since this is an alpha the expectation would be onsite work for the period of the engagement.
4. Could you please share the number of data journeys defined so far with examples Have you identified the list of non-survey data sources. Please share the indicative data volumes from each source, data formats, data ingestion frequencies per day including one-time data load and delta / change data volume. Do you have any existing EDI solutions in place. Could you please share the list of selected products/tools (including the hardware platform and software stack) to be used for data ingestion. Since this will be an alpha the focus will be on limited number of data sources to demonstrate the capabilities of a platform and its ability to meet early user needs. Work is progressing to identify and make available a range of data sources.
Terms of data volumes etc. work needs to be done on this during the alpha phase, part of the alpha will be to identify and agree potential solutions (especially the Cloudera technology stack) working with the ONS architect. The key aspect is the solution needs to be able to scale as more data are made available.
5. Is there any purchased data source from external agencies and do you have any license agreement in place with them including supplier access to these data sources Could you please share scope of of the high level discovery phase which has been completed and user needs Do you have the enterprise data hub / data lake to land, stage, process and repository in place along with the target state architecture A number of questions there – the data sources and any licence questions will be dealt with by ONS.
The scope of the high level discovery was the end to end data journey of administrative data into our processing platforms. We are building the data management platform (hence the reference to Cloudera etc.) and this alpha will contribute to the evolution of that platform and interact extensively with it.
6. Have you done any assessment of Data Security, Data Risk, Privacy, compliance with GDPR etc. for non-survey data elements We understand that we will be collaborating with multiple existing and new multi-disciplinary product teams. Could you please provide list of existing team with details Are you expecting the supplier to form teams comprising of Delivery Manager, Product Owner, Technical Lead, Business Analyst and Statistical Analyst roles Are you considering the engagement to follow T&M basis or a Fixed price model. The security requirements for each data source will be different for each data source and that will be agreed with the supplier by ONS. We have a good view of the requirements for current sources and we recognise that this will evolve as more data sources are made available. A discovery piece is about to begin that will greater insight into the overall security needs in this area.
The reference to other teams reflects the fact that ONS is creating a number of shared platforms comprising different products which are being developed by agile teams.
7. Have you done any assessment of Data Security, Data Risk, Privacy, compliance with GDPR etc. for non-survey data elements We understand that we will be collaborating with multiple existing and new multi-disciplinary product teams. Could you please provide list of existing team with details Are you expecting the supplier to form teams comprising of Delivery Manager, Product Owner, Technical Lead, Business Analyst and Statistical Analyst roles Are you considering the engagement to follow T&M basis or a Fixed price model. contd. This particular product team will be building part of the collection platform so will need to interact with other teams developing other parts of that platform as well as potentially developing elements of the data management platform.
ONS will provide Product Owner, Business Analyst and Architect roles, the supplier would be expected to provide the other roles (i.e., User Researcher, UX, Technical Lead, Software Developer etc.). We are recruiting Delivery Managers but there may be a short term requirement for this role as part of the team that the supplier fields.