Environment Agency

Environment Agency - Data Profiling and Data Quality Testing Project

Incomplete applications

17
Incomplete applications
14 SME, 3 large

Completed applications

27
Completed applications
22 SME, 5 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Tuesday 12 February 2019
Deadline for asking questions Tuesday 19 February 2019 at 11:59pm GMT
Closing date for applications Tuesday 26 February 2019 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work We want to undertake data profiling, data quality analysis and reporting work to:

1. Understand the data and potential quality issues
2. Identify potential quality rules and standards
3. Time allowing create a process to guide future data quality analysis work with external suppliers
Latest start date Monday 11 March 2019
Expected contract length Approx. 6 weeks but TBC with Supplier
Location No specific location, eg they can work remotely
Organisation the work is for Environment Agency
Budget range Maximum budget of £50,000.

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done The Environment Agency (EA) are developing a Data Quality Service to help the management of its data quality. The goal is to make it quicker and easier to find out exactly the quality of the EAs data.

The EA are tendering for an external consultant(s) to help create a process to commission Data Profiling and Data Quality Testing work.

The three data sets being used in this project are: Complex spatial, Complex non-spatial and Simple non-spatial data.

This work needs to be complete by the end of the financial year 2018/19.
Problem to be solved Where we identify we cannot meet the need internally we want to gain experience of working with a supplier to outsource analysis of data and data quality. This will free up resource internally for decision making and continuous improvement work. The supplier will profile data, test the quality of the data and explore the best way to communicate or visualise the results.

Using experience of this work, we need to create a process to follow for commissioning Data Quality work, with specifications, guidance, timeframes and cost estimations, varying by the different types of data that were assessed.
Who the users are and what they need to do As a Data Custodian I need to understand the make-up and quality of my data, and use the experience of working with a supplier to understand the specification and information needed for commissioning future pieces of data quality analysis work.

So that I can find the right improvements needed within the business to improve the fitness for purpose of the data in the most effective way.
Early market engagement
Any work that’s already been done This is the second year of a three year project. The first year focused on gap analysis, requirements and scoping. This work highlighted the need for a different way to undertake data quality work.

Currently we rely on internal resource and skills to undertake data quality work. The tools currently available for data quality work are not designed for this use and require technical knowledge and skills our staff do not possess.

We need to make the best use of our staff’s knowledge and develop a Data Quality Service providing support for the gaps in technical ability and availability.
Existing team The supplier will be working with the Environment Agency (EA) only, no other suppliers are involved. Other EA staff working on the project include a Senior Advisor in the Data Integrity & Flood Data team and Technical Specialists of the datasets involved.
Current phase Not applicable

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place We expect the work to be carried out offsite. There is a requirement for regular meetings to provide updates on how the work is progressing. These meetings would be a mixture of phone calls, WebEx and potentially face to face at the Environment Agency's head office in central Bristol.
Working arrangements The number of days per week expected to be worked by the successful supplier will be based on number of consultant(s) allocated to the task by the supplier and how long they believe the work will take to complete.

We would expect this work to be carried out offsite, with regular progress meetings on the phone or via WebEx.

Occasional face-to-face meetings may be required in central Bristol.
Security clearance

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Data quality management knowledge and experience
  • Data Profiling knowledge and experience
  • Data Standards knowledge and experience
  • Process building knowledge
  • Excellent Communication skills (written, graphical and verbal)
  • Ability to work collaboratively and share knowledge
Nice-to-have skills and experience ISO8000:150 accreditation would be an advantage, but not essential.

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 5
Proposal criteria
  • Approach and methodology
  • How the approach or solution meets The Environment Agency's needs
  • Value for money
Cultural fit criteria
  • Work as a team with our organisation and other suppliers
  • Be transparent and collaborative when making decisions
  • Take responsibility for their work
Payment approach Time and materials
Assessment methods
  • Written proposal
  • Work history
Evaluation weighting

Technical competence

50%

Cultural fit

5%

Price

45%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Could I please ask if year 1 of the project was delivered by a third party supplier or internal resources? The gap analysis was done in collaboration with an external supplier and then taken on by internal resource
2. What DQ tools does EA use at the moment? We make use of Microsoft Office, FME, RadiusStudio and Informatica tools and some others to a greater or lesser degree to undertake data quality work at present. But the supplier should choose the most suitable tool for the requirements and concentrate on the deliverables.
3. What are EA's preferred DQ technologies, in particular for testing DQ? We make use of Microsoft Office, FME, RadiusStudio and Informatica tools and some others to a greater or lesser degree to undertake data quality work at present. But the supplier should choose the most suitable tool for the requirements and concentrate on the deliverables.
4. Please could EA expand on their interpretation of ‘complex’ and ‘simple’ data: does this, for instance, refer in any way to unstructured data? This project does not involve unstructured data. Complexity relates to larger datasets with spatial data quality considerations, versus smaller scale non spatial datasets
5. What are the data sources in scope? We cannot send actual examples, DOS doesn’t allow for uploading data.  Complex spatial refers to a dataset with large amounts of structured data with many attributes, and many spatial and non-spatial data quality considerations. Complex non-spatial means a large non spatial structured dataset with many attributes and linkages, and simple non-spatial will be a small non-spatial structured dataset.
6. Does EA expect the supplier to provide DQ tooling? We expect the supplier to use the appropriate tooling to answer our requirement, but not embed the tooling within the Environment Agency
7. What onward systems/applications will use the improved data? In summary, the requirement is to profile the data, and form draft data quality rules, and then test the data against those rules. This work will then help us define the best way to take this kind of work externally in the future.  The next steps the EA takes to improve the data quality are out of the scope of this project.
8. Is the goal to put this into a repeatable process for new source data to be assessed in a similar manner, or are there ad-hoc analysis to be carried out? We would like to define a process to take similar requirements for data quality work externally based on the experiences of working with a supplier on data quality profiling and testing.
9. What are the tools and processes in use today, and what are the skill gaps expressed? We make use of Microsoft Office, FME, RadiusStudio and Informatica tools to a greater or lesser degree to undertake data quality work at present. We sometimes do not have theresources, data quality knowledge or technical skills to undertake data quality analysis and monitoring work needed
10. May we have some representative examples of the data in scope: Complex spatial, Complex non-spatial and Simple non-spatial data? We cannot send actual examples, DOS doesn’t allow for uploading data.  Complex spatial refers to large amounts of structured data with many attributes, and many spatial and non-spatial data quality considerations. Complex non-spatial means large non spatial structured datasets with many attributes and linkages, and simple non-spatial are small non-spatial structured datasets.
11. Does the gap analysis already done include discovery of where the data quality issues are entering the system? Is it possible to share the gap analysis? We cannot upload/share anything through the DOS platform.  The Environment Agency is a large and complex organisation with many different remits. The gap analysis was performed on the approach the Environment Agency takes to data quality management compared to an idealised data quality management framework rather than on a single system. This work is targeting one of the gaps identified.
12. Is the data migrateable? If it is, then does EA have the facility for it to be migrated to an environment within EA's security domain? This is out of the scope of this project.  The data will be made available under conditional licence to the supplier to complete the work offsite.
13. Is the data migrateable to an environment outside EA’s security domain? i.e to be able to run DQ processes outside the EA network? The data will be made available under conditional licence to the supplier to complete the work offsite.
14. Is the successful supplier precluded from a possible future Data Quality Service? No, but it is important the work be completed so the process to take future work to external suppliers does not include bias towards any particular future suppliers
15. Can you advise on the data sets involved? We cannot send actual examples, DOS doesn’t allow for uploading data.  Complex spatial refers to a dataset with large amounts of structured data with many attributes, and many spatial and non-spatial data quality considerations. Complex non-spatial means a large non spatial structured dataset with many attributes and linkages, and simple non-spatial will be a small non-spatial structured dataset.
16. Hi, The work summary includes a task to “Identify potential quality rules and standards”. This does not specify a standard, but Is accreditation against ISO8000:150 an end goal of the project? No, accreditation against ISO8000:150 is not an end goal of the project. The standards mentioned here are potential data standards and will be derived from the profiling of the data
17. Is it possible to provide a bid timeline and next steps following the DOS PQQ submission on the 26th February? Suppliers will be sifted down to 5 based on the skills, experience and fit responses. Those 5 will then be invited to submit their quote prices. A winning supplier will be evaluated using the weightings quoted. We expect to ask for quotes by the 29th Feb with winning supplier identified early in the week of 4th Feb
18. How many people do you envisage as being part of the project team for the duration of the contract? Both in terms of Environment Agency staff and the successful company staff. Within the Environment Agency there will be the Project Manager and the option of talking to a technical lead for the datasets involved. We envisage a team from the successful supplier, the number which will vary according to their assessment of the best way to achieve the requirements of the project within the timeframe and budget.