Oxford City Council (as lead organisation for 14 councils in total)

Learn how to help customers avoid calling councils unnecessarily through the use of chatbots and AI

Incomplete applications

29
Incomplete applications
24 SME, 5 large

Completed applications

44
Completed applications
38 SME, 6 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Monday 10 December 2018
Deadline for asking questions Monday 17 December 2018 at 11:59pm GMT
Closing date for applications Monday 24 December 2018 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Summary of the work Design and assist with user research into how to help customers avoid calling councils unnecessarily through the use of chatbots and AI. Also to generate business cases for four service areas selected and background research on potential platforms to use
Latest start date Monday 7 January 2019
Expected contract length Project must complete by 29 March 2019
Location No specific location, eg they can work remotely
Organisation the work is for Oxford City Council (as lead organisation for 14 councils in total)
Budget range This project is funded from the Local Digital Fund and as such has a maximum spend of £72,500

About the work

About the work
Opportunity attribute name Opportunity attribute value
Why the work is being done This project aims to deliver;
• a methodology for evaluating the suitability or otherwise of a particular application for developing into a chatbot or AI product
• a research base to assist local authorities developing their individual business cases to save time and resource rather than duplicating research
• a summary of the potential technology solutions that are available with their individual advantages and disadvantages of each
• a set of case studies drawn from participating councils user research
Problem to be solved There are problems for local authorities looking to adopt chatbot and AI solutions;
• There is no shared understanding of the technology and its advantages/disadvantages as a solution to key issues.
• There are a variety of products and platforms that already exist on the market, but no objective evaluation of each to identify the most appropriate to use.
• Without a sound evidence base, there is a risk of inappropriate chatbot/AI projects being pursued that have poor prospects of meeting their objectives, wasting further time and resource

Full information at https://localdigitalchatbots.github.io/full-application/
Who the users are and what they need to do The work will cover four service areas; planning advice, highways defect reporting, waste and recycling enquiries, revenues and benefits enquiries;

As a resident I need to make a report easily so that I can have the council resolve the issue
As a resident I need to enquire about a service and if it applies to me so that I know what steps to take next
As a service delivery staff member I need to deal with only complex queries so that my time is better spent on people that need the most help
Early market engagement No substantive work has been undertaken to date

Outline conversations around project scope were undertaken in October 2018, as outlined in our blog post https://digital.oxford.gov.uk/blog/2018/11/15/collaboration-is-difficult-let-s-do-more-of-it
Any work that’s already been done No user research work has been undertaken, although some councils have started to investigate the potential of chatbots and AI to deliver simple service solutions
Existing team Oxford City Council’s Digital Development Manager will lead the project on a part-time basis. There is no dedicated team for this project. Resources will be drawn from participating councils to undertake user research, under the guidance of the successful supplier.

There will be no other suppliers engaged on this work
Current phase Discovery

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place Oxford City Council is the lead authority with a base in Oxford

As the partner councils are spread across a wide geographical area, the successful supplier will be invited to devise an appropriate working arrangement to provide support at specific locations.

The councils are; Hertsmere Borough Council, Cheltenham Borough Council, Doncaster Council, Preston City Council, North East Derbyshire District Council, Bolsover District Council, Calderdale Council, Rotherham Metropolitan Borough Council, Bromsgrove District Council, Redditch Borough Council, Surrey County Council, Adur and Worthing Councils
Working arrangements This will require a mix of desktop research and practical assistance to councils engaging in user research for the first time.
The large number of councils and their distribution will make collaboration on project delivery more challenging.

We will use collaborative tools such as Google Docs for creating and sharing information, and Google Hangouts for our meetings.
We will adopt an ‘open first’ approach to how we work to help others learn from our experience.

An initial face-to-face kick-off meeting with all councils involved is required as soon as the contract starts to clarify goals and expectations.
Security clearance None required

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Have good experience of designing and running user research projects
  • Have technical knowledge of the AI/Chatbot market
  • Be experienced in coaching or mentoring staff with limited or no user research skills
  • Understand how to meet the needs of a diverse range of users
  • Work to user-centered, iterative, Agile principles
  • Be able to work remotely with teams to deliver a successful project
  • Have experience designing user research for a wide range of services
Nice-to-have skills and experience Experience of delivering an alpha project incorporating a chatbot or AI solution

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many suppliers to evaluate 4
Proposal criteria
  • Clarity of the approach
  • How the approach or solution meets user needs
  • How they’ve identified risks and dependencies and offered approaches to manage them
  • Team structure, including skills, experiences and relevance of individuals
  • Understanding of user needs
  • Ability to meet the time-frame for the work
Cultural fit criteria
  • Committed to working to the Digital Declaration
  • Work as a team with our partners
  • Be transparent and collaborative when making decisions
  • Have a no-blame culture and encourage people to learn from their mistakes
  • Take responsibility for their work
  • Share knowledge and experience with other team members
  • Work openly
Payment approach Fixed price
Assessment methods Written proposal
Evaluation weighting

Technical competence

60%

Cultural fit

7%

Price

33%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. Is there a requirement to submit a full proposal to support the answers provided for the essential and nice to have skills for the 24th December deadline? Yes, we will need the skills/experience of all suppliers applying so that we can score them against our evaluation criteria. More information helps us score appropriately based on the information we have received.
2. A question was asked about a full proposal submission until 24th of December. Please confirm us if for this phase we need to answer to the questions in the marketplace (6) or we need to propose a more detailed technical offer, as well. Thanks Applications are open until 24 December. Complete applications, answering all the points, are required by this date in order to be properly considered
3. Please can you confirm if the stated budget is inclusive or exclusive of VAT? The council is able to reclaim VAT so it is helpful if suppliers do not include this in their financial proposal.
4. Will expensive costs incurred by the supplier travelling to other council locations be separately covered outside of the stated budget? There is no additional budget to the capped sum indicated in the opportunity. Suppliers must ensure that their likely costs are all included within their proposed sum.
5. Please can you confirm your evaluation/award timescales, specifically: (a) when would shortlisted suppliers be notified?
(b) what is the submission deadline for proposals from the shortlisted suppliers?
(c) how long will the Council require to evaluate proposals and make an award decision?

There would appear to be insufficient time for these stages to be completed if Mon 7th Jan is the latest start date for the contract.
(a) Shortlisting is scheduled to take place on 27 December with suppliers notified the next day
(b) Suppliers are not required to submit any further proposals after shortlisting as full evaluation will be based on proposals and supporting materials received by the closing date on 24 December
(c) As there will be a maximum of four suppliers shortlisted we are confident this can take place and award by 4th Jan. If there is slippage we will consider starting the contract later
6. The online submission only appears to allow bidders to submit responses to the "essential" and "nice to have" skills and experience. If we are required to submit responses summarising our approach, timelines, team and commercials, where do we do this? If suppliers want additional materials considered we are happy to consider these if URLs are supplied that can be easily viewed the the assessment team. We have used this approach in the opportunity to link to our full application text to give suppliers more background.

These would need to be relevant and required to demonstrate their approach to the project, and not general marketing materials
7. What format would you like the proposal in?
Are there specific sections you want to see, such as: costs, timescales, previous experience, etc.?
We don't have any specific requirements for any proposals suppliers want to put together. Our advice would be to use an approach that best sets out your skills/experience and the approach you would use to deliver this project given the challenges. A breakdown of costs would be useful to include along with a preliminary roadmap for work/deliverables
8. Do you expect the user research sessions to happen in person, remotely, or through a combination of both? This was raised at today's Q&A session.
We have four areas for user research, with a lead council assigned to each one and 1-3 other councils associated with each. This provides opportunities to carry out on site research directly for/alongside the lead councils.
We think a mixed approach that utilises staff in some councils through direct help, and also a remote approach, would be pragmatic. We do not expect suppliers to undertake all research across all councils themselves given the geographic challenges and time/budget available.
9. How involved will the other councils be and what will their roles be? We have four areas of service research planned, with a lead council assigned to each. Other councils will align themselves to one area of research to create clusters around a particular topic. We are currently setting areas of interest for each council
We see the lead council working with their allied councils in working out how to progress their research.
We would also like to bring everyone together at the earliest stage to discuss with the supplier how we can best ensure everyone can have a role and participate.
10. Who do you see as the core team? As answered above, the core team will be the representatives from the four lead councils. Key personnel from each will need to be assigned to the work required.
In addition, we want to work openly and collaboratively to give everyone a chance to take a role and participate, so will be open to suggestions from suppliers on how we can make this happen
11. What is the scope of the training aspect? Some training (Agile for Teams) will be provided to participating councils through MCHLG in late January.
There is a mixed ability in relation to user research across councils, so some training here would be valued
12. Is the total budget of 72500 inclusive or exclusive of VAT? (The same question has been asked earlier but the answer is not clear) Apologies. The total budget is exclusive of VAT
13. Please clarify your responses to questions 5 and 6. The opportunity description implies that a written proposal including Price, Team etc will be required from the shortlisted suppliers. However, there is no where to supply this information within the form, and you have stated that no further information will be submitted by the shortlisted suppliers once they have been shortlisted. Is this an error, or do you want us to include a URL to our proposal in one of the (unrelated) criteria boxes? Yes please, provide a URL to your proposal
14. Why did the council specifically select the 5 areas for research (Question 3) for AI/ Chat bots over other areas? Is there data to show these benefits in more detail and perhaps any other opportunities you may have excluded and why? The areas of research were chosen by participating councils as those in which they were experiencing the most customer queries and considered there to be a potential to resolve this. We do not have any data to support these being beneficial areas for chatbots, but would value research carried out to show if they are or not.
15. To better refine our methodology: Are the populations of the authorities included distinctly different from a digital maturity/ digital infrastructure perspective? Are there groups of councils with more/less digitally aware or enabled citizens which should be considered in the methodology? We have not carried out that assessment of the participating authorities. There are national tools (such as http://heatmap.thetechpartnership.com) to assess the digital capabilities of particular areas of population
16. What are the constraints and considerations around vulnerable customers which should be considered in our approach? We would expect any participating customers to have consented to their participation and for any information relating them to be anonymised. Suppliers with good user research experience will be able to advise us as to potential issues to watch for.
17. Do you have an existing partner with whom you have worked with for AI/Chatbot projects to date? If so, has this been a successful partnership in terms of what has been achieved? No. The participating councils are all looking for advice and guidance on how to start considering the use of chatbots. The outputs of the project are expected to be shared widely with the local government sector for other councils in a similar position.
18. It is a little unclear what suppliers need to do to apply for this opportunity. Typically all a supplier would do at this stage is answer the 100 word questions but you seem to have indicated that you want full proposals by the same date as the 100 word answers. Please can you confirm this is the case and what should be covered in the proposal? Usually the proposals would only be submitted at the next stage, after shortlisting has taken place so this is quite unusual. Can you clarify to ensure a consistency across supplier responses please? Yes, we were advised in the supplier Q&A that the approach we have chosen is not the normal approach - our apologies. We would ask suppliers to include their proposals so that we can assess and award the contract at as early a date as possible given the short timescale afforded the project. As indicated in the answers above, include a URL to your proposal in one of your answers
19. You have suggested that a full proposal for the project and costs should be supplied by 24th December. Please can you provide an email address that we can submit our response to as we will have a proposal document and cost spreadsheet to send over wont be able to do this using a url? The suppliers we have spoken to have indicated that supplying a URL to make their proposal and costing available is a feasible approach, so we would ask for all suppliers to do this.
20. How was the Supplier Q&A session advertised? Details of the supplier Q&A were made available via the Digital Marketplace opportunity (as per their guidance)
21. are you following the digital outcomes process as explained here: https://www.gov.uk/guidance/ways-to-assess-digital-outcomes-and-specialists-suppliers? We have been made aware that the normal process is to shortlist on skills and experience and then require proposals, so in this regard we have selected a different approach to help with short timescales. Otherwise we have followed all aspects of Digital Marketplace process.
22. Will the research take Oxford City Council's current platform as the base? Or will a common platform be considered as well? Oxford City Council does not have a chatbot platform. The idea of a common platform is an objective we would like to explore as far as a discovery project will allow.
23. We did not attend the Q&A session, could we get information on the content please ? This was a two-hour session and so notes were not taken. A brief overview is available at https://localdigitalchatbots.github.io/2018/12/14/running-a-supplier-q-a-session/
24. It seems to say in Q1 & 2 that bidders are to submit a full proposal but there's no apparent way to submit documents other than the text responses on the Digital Marketplace portal. Is there any other component required for submission at this stage ? Please see answers to similar questions above
25. It seems to say in Q1 & 2 that bidders are to submit a full proposal but there's no apparent way to submit documents other than the text responses on the Digital Marketplace portal. Is there any other component required for submission at this stage ? Please see responses above
26. Does this published opportunity meet the DOS requirements? **** IMPORTANT CLARIFICATION *******

It has become clear from supplier questions that we have inadvertently departed from the expected process in requiring proposals from everyone rather than just shortlisted suppliers. We don't want this to disadvantage smaller suppliers so are making a change

To fix things we are proposing to move back to shortlisting on skills/experience after the closing date, then requesting written proposals from the 4 shortlisted suppliers.

We will allow a further week on the published timetable for suppliers to develop proposals and similarly push back the award/start dates.

Please amend your submissions if you need to.