This opportunity is closed for applications

The deadline was Monday 27 January 2020

Home Office- Data Services and Analytics.

Data Engineering- Data Services and Analytics

22 Incomplete applications

16 SME, 6 large

18 Completed applications

13 SME, 5 large

• Published: Monday 13 January 2020
• Deadline for asking questions: Monday 20 January 2020 at 11:59pm GMT
• Closing date for applications: Monday 27 January 2020 at 11:59pm GMT

Overview

• Summary of the work: To provide a Data Engineering team to;; ; Analyse new data sources, modelling and mapping data to corporate POLE model.; ; Create routines to transpose data using "on-premises" and Cloud based ETL platforms.; ; Fix issues and bugs.; ; Migrate current "on-premises" platform to Cloud based capability.; ; Undertake service optimisation and enhancement.
• Latest start date: Sunday 1 March 2020
• Location: London
• Organisation the work is for: Home Office- Data Services and Analytics.
• Budget range: Any submitted day rates within the contract must not exceed £1,000 (exc VAT).; The buyer does not anticipate SFIA level 6 or 7 resources to be used for delivery of this contract. Should the parties exceptionally agree any SFIA level 6 or 7 resources for limited use (e.g Client workshops), the day rates for such resources will remain under the cap of £1,000 (exc VAT).; ; Responses must include maximum billed day rate in SFIA rate card submission form.

About the work

• Why the work is being done: Data Services and Analytics is a centralised service within the Home Office's Digital Data and Technology's Enterprise Services, and is at the forefront of efforts to re-imagine the departments relationship with data. It leads the way in professionalising our approach to the acquisition and analysis of data, promoting dynamic insight-led and evidence based decision making tools and techniques. The Home Office Data Engineering Team is an operationally critical capability delivering regular ingests of multiple data sources to downstream products and services. ; As the front end capability in DSA it is highly visible and the one service upon which all other downstream services rely.
• Problem to be solved: Data Services and Analytics is in the process of implementing a corporate technology strategy that will see existing 'on premises' platforms migrate to the Cloud. The key challenge is maintaining continuity of service whilst designing, implementing and operating new bulk and fast data pipelines to serve downstream consumers and provide enhanced capability.; New capability is being built around AWS Cloud services and bespoke java based platforms. The Home Office is therefore looking for a supplier with knowledge of transitioning to and operating ETL in commodity Cloud-based environments whilst being able to support existing pipelines based primarily on SAS-Data Integration Studio and ActiveEon.
• Who the users are and what they need to do: The users are either:; ; Data Scientists and Data Analysts within the Home Office who explore and use data to provide analysis, insight, reporting and data products for their customers, to support business operations, policy making and decision making.; ; OR; ; End Business Users elsewhere in Government who use DSA Data Products.
• Any work that’s already been done: Development of the current 'on premises' capability to ingest and process data sets to business requirements.; ; Construction of a new Cloud based common landing zone for all data consumed by the DSA.; ; Orchestration, prioritisation and automation of data ingest.; ; Development of a new Cloud based "fast data" streaming capability performing matching, ETL and data transformation for consumption.
• Existing team: The existing Team comprises eleven data engineers made up of three Home Office Permanent Civil Servants, two Graduates supplied to the Home Office through an Academy programme and six from a contracting partner.
• Current phase: Beta

Work setup

• Address where the work will take place: The work will be carried out from a Home Office location primarily in Croydon, as required on other Home Office and Supplier sites.; Occasional working from home may be permitted with prior agreement from the Data Manager and in line with current Home Office policy.
• Working arrangements: The supplier is expected to provide a team of specialists to deliver the outcomes. The Supplier must be flexible and make working with a mixed team across multiple sites effective. Typically we expect resources to be on site 5 days a week.; ; The team will work from the principal location as required Monday to Friday 0900-1700, although some out of hours support may be required occasionally.; ; Travel to regional sites may be required, expenses incurred from travel outside of the M25/Greater London area will be subject to Home Office Travel (no Subsistence) Policy.
• Security clearance: All resources must hold, or be prepared to undergo, SC security clearance to operate on the project. (with willingness to acquire resources with NPPV3 and DV clearance).

Additional information

• Additional terms and conditions: The work will be progressed by a number of Statements of Work which may be fixed price, T and M or Capped T and M as appropriate.; ; Standard DOS framework and call off terms and conditions will apply.; ; Resources cannot charge for more than 100 working days in any consecutive six months or more than 200 working days in any consecutive year.

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

• Essential skills and experience:
  • Have comprehensive experience of ETL data processing across multiple datasets including expertise in Scheduling, Orchestration, Prioritisation and automation of data pipelines.
  • Have expertise in the use of Structured Query language (SQL).
  • Be able to demonstrate experience in change data capture approaches to simplifying data movements within data pipelines.
  • Expertise in the development, troubleshooting and maintenance of Python code.
  • Demonstrable experience in loading and managing large volumes of data including both structured/unstructured sources in many formats including (but not exclusively) JSON, XML, SQL Server, Oracle, Access and MySQL.
  • Demonstrable experience of enabling data governance criteria for Data access, deletion/removal and general compliance.
  • Experience in loading streamed data sources and related Cloud based technologies.
  • Expertise in the development, troubleshooting and maintenance of Java Code (required to support core platforms).
  • Experience in developing and maintaining ETL with SAS Data integration Studio tooling.
• Nice-to-have skills and experience:
  • Maintain and develop Visual Basic scripting.
  • Maintain and use ActiveEon (Scheduler).
  • Use Postgres databases.
  • Demonstrate capability in Pivotal GreenPlum MPP database.
  • Manage data in an AWS Cloud environment.
  • Demonstrate experience of operating in the Public Sector environment.
  • Demonstrate expertise in the bulk and incremental data load to Cloud native services including simple cloud storage and structured/unstructured data repositories (Graph DB, RElational DB, Data warehousing, Data indexing/search).
  • SC clearance already in place.
  • Have experience in Kafka message queuing systems.
  • Have experience in containerization technologies such as Docker.

How suppliers will be evaluated

All suppliers will be asked to provide a written proposal.

• How many suppliers to evaluate: 4
• Proposal criteria:
  • Approach and methodology
  • Technical skills and experience
  • Service Model
  • Mobilisation Plan
  • A plan for optimising resources.
  • Identification of risks and dependencies.
  • Value for money
• Cultural fit criteria:
  • Work collaboratively with all teams in DSA to ensure smooth handoffs in an environment of continuous improvements and changes.
  • Ability to function effectively and collaborate in a multi-supplier environmant.
  • Work successfully in complex Public Sector organisations.
  • Approach to proactive issue management, problem resolution and improving ways of working.
  • Focus on achieving value for money in all activities.
  • Blend rapidly and efficiently into existing methodology and constantly deliver continual improvements to raise delivery standard.
• Payment approach: Capped time and materials
• Additional assessment methods:
  • Case study
  • Work history
  • Reference
  • Presentation
• Evaluation weighting:

  Technical competence

  60%

  Cultural fit

  5%

  Price

  35%

Questions asked by suppliers

• 1. Would you support SC applications for staff who don't yet have SC clearance but capable of achieving SC status?: Yes, while candidates who are already SC cleared will be simpler and easier to onboard, we do actively do this for candidates as you describe so we would support their applications.
• 2. Can you please confirm the expected or approximate contract length?: The contract will be awarded as a standard DOS Framework contract with an expiry date two years from issue, and a six month extension provision, though the actual contract will be managed via individual Statements of Work of varying length depending upon the project phase.
• 3. Can you provide more detail in respect to the quantity/size of data (total size and no of records) and how it is currently federated (is it stored in multiple datasets/databases or amalgamated across a single dataset/database? - Looking at the information suggests it's multiple datasets/databases but would be good to confirm.: The total data in our analytical data store is approximately 40Tbytes, and comprises billions of records. We currently store data in multiple schemes in a Pivotal Greenplum database and retain source format, history and latest state. We are however looking to move both storage and compute capability to AWS.
• 4. 1. Can you advise what type of data is contained in the current analytics platform? (for example, is it technical/systems data or Home Office specific data that might contain sensitive/private such as asylum applications, immigration data etc?2. Are the systems that produce the data feeds that link to the analytics platform moving to the cloud along with the analytics platform or are these systems staying on prem?3. How fast is the internet connection at the premises where the analytics platform is hosted/data is stored?: 1. The data stored in our platforms is from multiple sources and includes:; a. Commercially available data e.g. Dunn and Bradstreet and GB.; b. Data from immigration and border systems (sensitive to highly ; sensitive).; c. Data from policing and intel functions (sensitive to highly sensitive); ; 2. Some source systems are moving into the Cloud, others will remain; within the Home Office estate.; 3. The current internet connection to the on premises capability; is 2 x 10Gbits links.
• 5. Can you please advise if this requirement is to solely cover the source provided by the contracting partner or will it cover all of the existing team?Are you able to advise who the contracting partner is?: Existing permanent resources in the team will remain in post. The requirement is to provide replacement for the supplier provided expert resources needed to run the team at current capacity, with current capability, and help to transition to Cloud operations, and we cannot disclose whom that incumbent partner is.
• 6. Are you able to forecast the approximate expected size/number of resources required, based on the Statements of Work?: The current team stands at 11 total (3 full time Civil Servants, 2 Graduates and 6 supplier provided) and we expect this will need to flex by potentially a couple over the next year, but as we are only at the design stage of our bulk data pipeline replacement this is difficult to quantify exactly.
• 7. Was an Alpha phase carried out and if so, was it carried out by an incumbent or third party provider, separate to the existing team?: All Alpha phase work was carried out by the current incumbent team which is a mixed economy team of permanent Civil Servants, Graduates and contracted resource.
• 8. 1. How many people do you expect us to supply for this project?2. What's the procurement timeline?3.Do nice-to-haves bear the same weight as essential skills?4. Any remote work flexibility?: 1. We expect the current supplier team to be stable over at least the 1st year; 2. Questions can be asked to Jan 20th, last date for submission of; applications is Jan 27th. Whilst the intention is to commence; service on March 2nd this will depend upon on-boarding requirements.; 3. No. Essential skills are exactly that, essential. The nice to haves are also; exactly that, nice to have, and will form a part of the evaluation.; 4. There is currently very limited opportunities for remote working so ; bidders should assume that resources will be operating from Home; Office locations.