Awarded to Profusion Media

Start date: Wednesday 18 May 2022
Value: £158,000
Company size: SME
InnovateUK

DDaT22027 - Janus Analytics Data Platform Design, Build & Deployment Project

12 Incomplete applications

11 SME, 1 large

5 Completed applications

4 SME, 1 large

Important dates

Published
Thursday 3 March 2022
Deadline for asking questions
Thursday 10 March 2022 at 11:59pm GMT
Closing date for applications
Thursday 17 March 2022 at 11:59pm GMT

Overview

Off-payroll (IR35) determination
Contracted out service: the off-payroll rules do not apply
Summary of the work
We have recognised the need to improve our Data Analytics Platform to enable us to progress our strategic objectives. We wish to design and deploy a consolidated data analytics platform which will hold both internal and external data and enable us to leverage both in our analytics and reporting outputs.
Latest start date
Friday 20 May 2022
Expected contract length
3-4 Months
Location
No specific location, for example they can work remotely
Organisation the work is for
InnovateUK
Budget range
Total costs should not exceed £200,000 exclusive of VAT.

About the work

Why the work is being done
Innovate UK is progressing on its journey to becoming a data-driven organisation. We have recognised the need to improve our Data Analytics Platform to enable us to progress our strategic objectives. Specifically we wish to design and deploy a consolidated data analytics platform which will hold both internal and external data and enable us to leverage both in our analytics and reporting outputs.
Problem to be solved
This is a new platform intended to enable interactive data mining and to enable AI and ML data functions to be applied against it in real time. To achive this we need the platform constructed and deployed. This will require a number of current data sources and integration tools to be interfaced with this new platform to ensure it retains current and accurate data from the internal systems sources. Also external data feeds from a variety of providers will be integrated using our existing SIL (message bus) capabilities.
Who the users are and what they need to do
This is a technical delivery platform with a wide array of users and requirements. These are stated in a separate presentation.
The analytics platform will be hosted in AWS in line with our strategy and other InnovateUK internal systems and is likely to be based on RDS PostgreSQL. The platform is intended to be flexibly scalable, resilient and highly performant as it will serve interactive Power-BI reporting systems across the organisation. Additional capabilities to utilise other analytics tools and systems in future is in scope of this project as well as integration of current deployed BI reports currently hosted on other and legacy platforms.

In addition we require an AWS data lake to be provisioned with supporting EC2 nodes to run python ETL processes against that datalake. Further expansion to include apache spark and apache kafka are considered likely in subsequent phases (not in scope for initial platform delivery) however note should be made of this and provision in design made accordingly.
Early market engagement
N/A
Any work that’s already been done
Discovery has already been done – this is an alpha phase project.
A full presentation on the requirement, strategy, architecture and desired outcome is available separately.
Existing team
This project will require close working with the Innovate UK data team, helping to develop capability within the Innovate UK team, as well as delivering the core deliverables of the tender. The main objective of this project is to design, build and deploy a fully operational data platform suitable for intensive query use from a large number of Power-BI dashboard users across the organisation. Additional data loads and automated analytics processes are required to interact with this data based on Python, and a variety of other AWS analytics tools including but not limited to SageMaker.
Current phase
Alpha

Work setup

Address where the work will take place
Hybrid working will be used. The activity is expected to be performed using hybrid working – mostly remote with some in-office time at our Swindon offices for meetings and design discussion workshops for example.
Working arrangements
This project will require close working with the Innovate UK data team, helping to develop capability within the Innovate UK team, as well as delivering the core deliverables of the tender.
Security clearance
BPSS required

Additional information

Additional terms and conditions
N/A

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Essential skills and experience
  • Demonstrate your teams experience and understanding of up to date industry developments regarding data management, use, and analysis.
  • Detail with an example where you have worked with AWS Aurora Postgres.
  • Detail using an example your experience of RDS.
  • Detail using an example your experience of MySQL
  • Detail using an example your experience of using ETL Tools.
  • Detail using an example your experience in Talend.
  • Detail using an example your experience of WS02
  • Detail using an example your experience of Kafka, Spark, Python and Java.
  • Detail, using an example your application experience
Nice-to-have skills and experience
  • Detail your experience of current data manipulation development knowledge
  • Detail your experience of AWS database portfolio more broadly, AWS analytics and data lake services, ETL tools including some of Knime,
  • Detail your experience of Talend, Alteryx, WS02 others. You will also have extensive experience of use of BI tools including Power-BI, Tableau, and other tools. Python, Java

How suppliers will be evaluated

All suppliers will be asked to provide a written proposal.

How many suppliers to evaluate
5
Proposal criteria
  • Describe the approach and methodology you will apply to Alpha, including how priorities will be determined and emerging findings presented
  • Describe how the approach will achieve the objectives and intended outcomes
  • How the structure of the team and how their technical-skills and experience will be used to deliver-the-need, including estimated-resource dedicated to the project linked to the overall cost
  • A project plan with proposed timelines for mobilisation and delivery
  • Identify the risks and dependencies you associate with the project and provide mitigation actions to manage them
  • Value for money, including how you will take previous learning to deliver better value for this project
Cultural fit criteria
  • Demonstrate how your proposed team will work successfully with UKRI as part of a remotely distributed multi-disciplinary team.
  • Demonstrate how your proposed team will deliver in an open, collaborative, agile way. Detail your plan for sharing knowledge and experience
  • Demonstrate how you will work with stakeholders at all levels, within the programme and across the wider UKRI landscape.
  • Detail how you will work within given constraints (including but not limited to technology, policy, regulatory, financial, legal, ethical, social, user constraints).
Payment approach
Capped time and materials
Additional assessment methods
Presentation
Evaluation weighting

Technical competence

65%

Cultural fit

10%

Price

25%

Questions asked by suppliers

1. Was the Discovery exercise conducted by an incumbent supplier or in-house?
Discovery exercise was conducted in-house and led by Innovate' s Head of Data Management, MI & Analytics.
2. Who is the incumbent supplier who has conducted Discovery phase?
There is no incumbent supplier, internal team and lead conducted discovery.
3. Would it be possible to outline how you are using WS02 Products/Services? Is this for API management, Identity and Access Management, Digital Asset Governance and/or using Ballerina?
API and ESP services - intending to replace it - need to examine it closely and then replace.
4. Can small business bid for this project?
Yes, small businesses can bid for this requirement
5. Please can we enquire about who completed the Discovery phase. Was this Innovate UK or a third party?
As per question 2 answer, discovery was completed in house by Innovate
6. Was Discovery conducted by the internal team? If not, who was the supplier involved?
As per question 2 answer, discovery was completed in house by Innovate
7. Who is currently delivering the Discovery for this?
As per question 2 answer, discovery was completed in house by Innovate
8. What was your reasoning behind choosing technologies such as AWS Aurora Postgress, WS02, Talend, Spark?
Some are legacy we are migrating from, aws aurora postgres, spark, kafka, we are moving to - latter is strategic - others needed to understand to help with migration (talend, ws02 legacy for example)
9. We don’t specifically have WSO2 experience, however, we leverage an associate network who have a wide range of different technology experience so we can bring in that skillset for this project. Would this be considered as meeting the essential skills for this submission?
Yes if you can do it within budget and maintain quality and time to deliver
10. The requirement mentions that a Discovery has already been done, and a full presentation on the requirement, strategy, architecture and desired outcome is available separately. How do we obtain a copy of this presentation?
Outcomes from previous stages will be shared with shortlisted suppliers
11. Why are all three of postgres, MySQL, and RDS being used? Are there multiple legacy systems and data that will be accessed?
MySQL is legacy, rds postgres is target future platform - postgres flavour of rds will be used - both mentioned for completeness.
12. Is there an incumbent and are they eligible to bid?
There is no incumbent supplier.
13. Which supplier delivered the Discovery phase of the project?
As per question 2 answer, discovery was completed in house by Innovate
14. Can you clarify/correct question 9
Give an example project where you have used some or all of the relevant skills and experience to deliver a data infrastructure solution
15. Is Knime experience a ‘must’ requirement for your new Analytics Platform?
It is part of our current platform but not a must have - desirable to understand as many tools involved as possible (talend, ws02, alteryx, knime, spark, kafka, plus any others in that space)
16. Are you considering other ETL technologies to replace Knime?
Yes
17. Is the Knime integration productionalised?
Yes - but has limited deployed scale in our use case
18. Are you using any streaming technologies at a production level?If yes, what is the high-level use case for streaming data? And are you currently using live dashboards?
Not yet - data doesn’t really lend itself to streamed processing. yes we are using live dashboards based on powerbi and some tableau
19. Are you using licenced versions of Talend Cloud in your solution?
To be confirmed during project - we are licensed and have production services and adminstrator workspaces live with talend
20. Do you have SMEs within your business already utlising Apache Spark?
Yes - limited use at present - expecting to increase this
21. Does your team have experience with Databricks?
Not known
22. To what extent do you implement DevOPs code management procedures?
Handled by another team - we have a devops service delivery team that handles this - we anticipate some change to that to increase our hands-on devops capability in this project