Awarded to Zaizi Limited

Start date: Monday 31 January 2022
Value: £86,675
Company size: SME
The National Archives

AWS Data Transformation Pipeline Service for Court Judgment documents

1 Incomplete application

1 SME, 0 large

9 Completed applications

6 SME, 3 large

Important dates

Published
Monday 6 December 2021
Deadline for asking questions
Monday 13 December 2021 at 11:59pm GMT
Closing date for applications
Monday 20 December 2021 at 11:59pm GMT

Overview

Off-payroll (IR35) determination
Supply of resource: the off-payroll rules will apply to any workers engaged through a qualifying intermediary, such as their own limited company
Summary of the work
Design, implement, test and document an AWS data-transformation infrastructure and pipeline service to securely connect our systems. The service will retrieve, transform and send data according to business rules. Initial delivery is an MVP Beta for one data type (Court Judgments in MS Word), to be operational by April 2022.
Latest start date
Monday 17 January 2022
Expected contract length
Three months with the possibility of extension to a maximum of 6 months.
Location
No specific location, for example they can work remotely
Organisation the work is for
The National Archives
Budget range
Up to £90k to include travel and all other expenses.
We have checked the role requirements (to the best of our knowledge) using the assessment tool found at https://www.gov.uk/guidance/check-employment-status-for-tax and the determination for the role(s) as advertised is that the intermediaries legislation does not apply to this engagement (out of scope).

About the work

Why the work is being done
To support delivery of a new, cloud-based Data Transformation Service aiming to deliver secure, robust, automated and extensible ‘pipelines’ for routing data between internal systems and services. Specified data transformations and tools will be embedded into the pipeline as needed.
This outcome initially supports an organisational commitment to receive, process and publish Court Judgment documents by April 2022. The supplier will deliver a Minimum Viable Product (MVP) as defined by The National Archives (TNA).
Court judgments are sent to TNA as MS Word documents. The MVP Data Transformation Service must securely retrieve these from a specified cloud location, transform them to XML / LegalDocML using a parser application (created by another TNA project) and route them to our publication service.
The MVP should be designed with a view to future extensibility. Further refinements on the product roadmap include filtering data based on business rules, handling additional document types, applying additional and more sophisticated transformations and routing data to and from other TNA services.
Problem to be solved
Specify, design and deliver an event-driven, automated workflow for Court Judgment documents as a step towards creating a fully featured Data Transformation Service for TNA.
The MVP should identify receipt of a new Court Judgment by our Transfer service; retrieve and process the document using custom-built parser software to create an XML version; repackage the data; and deliver it to our Publication service. The Data Transformation Service must operate securely across TNA’s cloud environments and service boundaries.
The MVP will handle only the most common form of Court Judgments, which are individual Word documents intended for publication. Error handling is required, e.g. to identify more complex Judgments and route these for manual processing.
This MVP service should be designed with a view to adding further scenarios in the future.
The supplier will work closely with TNA staff to ensure their design and deployment is consistent with our architecture and technology stack. The service should initially be deployed on its own AWS infrastructure at TNA, with the potential to share an environment with other cloud-based services in the future.
The service must meet NCSC standards for system security and GDS service standards. Any web components must meet WCAG 2.1 Accessibility standards.
Who the users are and what they need to do
Internal stakeholders of the Data Transformation Service are TNA Service Owners responsible for the operation of other services (initially, Transfer and Publication) that either produce or consume data and rely on the Data Transformation Service to supply that data in a format suitable for use.
External indirect stakeholders of the Data Transformation service are Court Service staff, who expect that Court Judgments sent to TNA will be published in line with a one-hour target. This requires a robust, automated pipeline to identify when a new judgement has been received, process it accurately and make it available for publication quickly.
Future stakeholders will include Service Owners for our Preservation, Cataloguing and Access services, which will need to send and receive data and other types of public records.
Early market engagement
We have researched available data transformation tools and identified Talend Open Studio as suitable for our requirements for creating transformation scenarios (packaged as executable jar files) for both the immediate need and our longer-term roadmap.
Any work that’s already been done
We have carried out a Discovery phase and are currently concluding the Alpha phase for the Transformation Engine to evaluate feasibility and identify our preferred technology stack. As part of the Alpha phase, prototypes have been implemented using event-based consumers written in Scala, a cloud-based Apache Kafka queue and transformations designed using Talend Open Studio.
A separate project is currently creating a parser to convert Court Judgments received as MS Word documents to XML in LegalDocML format. The parser code will be executed by the Transformation Service.
Existing team
The supplier’s team will deliver the Data Transformation Service. The National Archives staff will include a Product Manager with an understanding of user and stakeholder needs and an Agile Delivery Manager. The team that produced the Alpha prototype (Data Engineer, Technical Architect and Senior Developer) will be available and must be consulted about the supplier’s proposed approach and fit with our technology stack (e.g. Configuration of AWS environments, use of messaging for communication between services, proposals for the data pipeline). Internal users will be available to test that the service routes data correctly and processes it accurately.
Current phase
Alpha

Work setup

Address where the work will take place
Supplier to work remotely. Occasional attendance on-site at The National Archives, Kew, Richmond, Surrey TW9 4DU may be required.
Working arrangements
The National Archives’ staff will be available during UK 9am-5pm. The supplier’s team is expected to work during these hours with scope for flexibility with mutual agreement.
The supplier will provide their own equipment and technology and will be given access to our organisational GitHub, Jira and Slack resources as appropriate. Regular meetings will take place via Microsoft Teams with Slack available.
The supplier will apply an Agile approach to scope, plan, and deliver work incrementally, with regular communication. The supplier will conduct regular, frequent product review or ‘show and tell’ sessions to demonstrate progress and seek feedback.
Security clearance
Baseline clearance will be required (BPSS)

Additional information

Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Essential skills and experience
  • Strong experience in building and configuring flexible data pipeline services
  • Excellent experience of deploying and optimising operational data processing services in AWS
  • Excellent experience of building, configuring, and securing AWS environments, including infrastructure configuration management (e.g. Terraform)
  • Experience of building automated event-based architectures using message queuing technologies
  • Experience of delivering Continuous Integration/Continuous Deployment pipelines including integration testing methodologies
  • Track record of delivering highly secure services and meeting OWASP and CIS web application security best practices
  • An iterative, Agile approach to delivery
  • Ability to produce clear technical documentation
Nice-to-have skills and experience
  • Track record of delivering cloud services that conform to the Government Service Standard
  • Knowledge of meeting GDS service assessment criteria

How suppliers will be evaluated

All suppliers will be asked to provide a written proposal.

How many suppliers to evaluate
4
Proposal criteria
  • Experience of building and configuring flexible data pipeline services in AWS.
  • Evidence of how highly secure services would be delivered to meet OWASP and CIS application security best practices
  • Experience of producing clear technical documentation and ensuring effective handover to enable the service to be maintained operationally
  • Team structure, including the relevance of the team members' skills and experience.
Cultural fit criteria
  • Communicate clearly and openly, including working collaboratively with our product and delivery managers, technical specialists, and users to ensure the product meets the needs of its users
  • Work in an open and transparent way, communicating actively, sharing work in progress, and involving others as you go
  • Explain how you will engage and use your expertise to work effectively and constructively challenge our team
  • Describe how you will support positive working relationships throughout the life of the contract
Payment approach
Capped time and materials
Additional assessment methods
Presentation
Evaluation weighting

Technical competence

70%

Cultural fit

10%

Price

20%

Questions asked by suppliers

1. How did you arrive the budget of 90k ?
We estimated the amount of work we think is necessary to deliver the outcome, taking into account past experience, the length of the engagement, the progress we have made so far with court judgments, the in-house team contribution, our experience of the market in terms of day rates, and a contingency factor.