The Financial Conduct Authority (FCA)

Lead Data Engineer, Analytics and Information Management (AIM) Project (Ref: CON-18-038)

Incomplete applications

5
Incomplete applications
3 SME, 2 large

Completed applications

6
Completed applications
4 SME, 2 large
Important dates
Opportunity attribute name Opportunity attribute value
Published Friday 2 March 2018
Deadline for asking questions Tuesday 6 March 2018 at 11:59pm GMT
Closing date for applications Friday 9 March 2018 at 11:59pm GMT

Overview

Overview
Opportunity attribute name Opportunity attribute value
Specialist role Developer
Summary of the work The FCA require a Lead Data Engineer to set best practice across the developing Analytics and Information Management (AIM) platform and provide mentoring and skills transfers to other less experienced Data Engineers.
Latest start date Friday 1 June 2018
Expected contract length
Location London
Organisation the work is for The Financial Conduct Authority (FCA)
Maximum day rate

About the work

About the work
Opportunity attribute name Opportunity attribute value
Early market engagement
Who the specialist will work with The Lead Data Engineer will be joining a dedicated Scrum Team responsible for delivering the data platform. The Scrum Team is co-located and currently consists of a Scrum Master, 5 Data Engineers, a Data Architect and a Solution Architect. The Scrum team is part of a broader project team and also works closely with experts from source (data) systems, as well as users of the initial deliverables. The intention is to continue to expand the team organically, bringing in knowledge and experience, whilst maintaining agile values and allowing the team to self organise.
What the specialist will work on The Lead Data Engineer will lead the team in:
• Building data and Metadata processing using established frameworks.
• Performing testing to validate that data is correctly processed and metadata is correctly maintained.
• Designing and modelling, ensuring the alignment of semantic models to the FCA Conceptual Models.
• Ensuring/demonstrating efficient code execution in the target environment.
• Configuring the code orchestration, ensuring monitoring and error handling function in accordance with the FCA designs.
• Managing developed artefacts into the FCA repository.

Work setup

Work setup
Opportunity attribute name Opportunity attribute value
Address where the work will take place 25 The North Colonnade,
Canary Wharf,
London
EC14 5HS
Working arrangements The specialist will be expected to work on-site at the FCA's head office five days a week - though working from home may be permitted where possible. Being on-site will allow a greater level of collaboration with colleagues and allow the individual to attend face-to-face meetings as required.
Expenses will only be granted in extraordinary circumstance. As no travel is expected as part of this role, we do not anticipate granting any expenses.
Security clearance

Additional information

Additional information
Opportunity attribute name Opportunity attribute value
Additional terms and conditions

Skills and experience

Buyers will use the essential and nice-to-have skills and experience to help them evaluate suppliers’ technical competence.

Skills and experience
Opportunity attribute name Opportunity attribute value
Essential skills and experience
  • Experience with AWS.
  • Experience with Cloudrea Hadoop Distribution (CDH 5).
  • Implementing data ingestion/presentation/semantics data layers as guided by BA/Data Governance/Conceptual Data models.
  • Using data ingestion/data transformation frameworks and assisting in the development of new frameworks/or framework variants.
  • Skills in data modelling (both structured and unstructured data).
  • Skills in metadata repositories and technologies (Hive / Cloudera Navigator).
  • Skills in data acquisition (landing, ingestion and metadata) of various data types including Salesforce, XML and Relational data.
  • Skills in data manipulation: Java, Scala, Python executing within a
  • Spark environment, orchestrated by Oozie/Talend.
  • Experience with HDFS/Hive/AWS/Sentry Policies/Oozie/ZooKeeper.
  • Experience with Big Data self-describing formats (JSON/Parquet/ORC etc., and when to use them).
  • Experience with Spark clusters, both permanent (Cloudera) and transitory clusters Altus/EMR.
  • Data presentation via Sentry/Impala to visualisation technologies e.g. Tableau / SpotFire.
Nice-to-have skills and experience
  • Data management and analytics.
  • Data discovery processes.
  • Familiar with Scrum and working in an Agile team.
  • Generation of data catalogues/models from the metadata repository and exposing that to various user communities.
  • Skills in Search technologies (Drill/Lucene/Solr).
  • Talend.
  • Developing technical specifications corresponding with architecture, based on business and functional analysis.
  • Capturing and enhancing metadata and using the metadata to generate a Data Catalogue (and the mechanism to expose that to the user community).
  • Definition of Security Policies for the data implemented and the implementation of those policies in Sentry providing Authorisation controls at the required granularity.
  • Experience in large scale distributed processing architectures e.g. enterprising caching, low latency data driven platforms (Enterprise Data Warehouse preferably).
  • Expertise in fault tolerant systems in AWS including Clustering & multi AZ deployments.
  • Working knowledge of setting up & running Big Data clusters (e.g. Hadoop, Spark, Impala) supported by various Databases, object stores (S3) on AWS.
  • Understanding of configuration management and Devops technologies (e.g. GitLab /Jenkins ).
  • BI/Data Prep/Visualisation/ETL tools; Zeppelin, SpotFire, Tableau, StreamSets for large scale dashboards, analysis and reporting.
  • Knowledge of the various flavours of SQL.

How suppliers will be evaluated

How suppliers will be evaluated
Opportunity attribute name Opportunity attribute value
How many specialists to evaluate 10
Cultural fit criteria
  • Work as a team with our organisation and other suppliers
  • Be transparent and collaborative when making decisions
  • Have a no-blame culture and encourage people to learn from their mistakes
  • Take responsibility for their work
  • Share knowledge and experience with other team members
  • Challenge the status quo
  • Be comfortable standing up for their discipline
Assessment methods
  • Work history
  • Interview
Evaluation weighting

Technical competence

50%

Cultural fit

20%

Price

30%

Questions asked by suppliers

Questions asked by suppliers
Supplier question Buyer answer
1. When is the contract expected to finish, and what is the IR35 status? The Call-Off Contract term will be one year.
All candidates should fall outside of IR35.
2. How long is the assignment? The Call-Off Contract term will be one year.
3. Can you confirm that SC clearance or any other type of clearance is not required? SC clearance is not required but would be beneficial to have. No other clearance is required.
4. Can you please confirm the IR35 status for this piece of work All candidates should fall outside of IR35.
5. May i know the exact contract length? and if there is any Cap on budget for the role? The Call-Off Contract period will be one year.
Whilst there is no cap on the day rate (and by extension budget) for this requirement, the FCA would not expect day rates to exceed £1000. All candidates will be assessed based on the value for money they are deemed to present against the requirements.
6. What is the duration of this contract? And, what is the day rate? The Call-Off Contract period will be one year.
Whilst there is no cap on the day rate (and by extension budget) for this requirement, the FCA would not expect day rates to exceed £1000. All candidates will be assessed based on the value for money they are deemed to present against the requirements.
7. Please could you confirm the max day rate, expected contract length and IR35 status of the role? The Call-Off Contract period will be one year.
Whilst there is no cap on the day rate (and by extension budget) for this requirement, the FCA would not expect day rates to exceed £1000. All candidates will be assessed based on the value for money they are deemed to present against the requirements.
All candidates must fall outside of IR35.
8. What is the rate per day that you are expecting to pay for this specialist?Please include whether inclusive of exclusive of VAT Whilst there is no cap on the day rate (and by extension budget) for this requirement, the FCA would not expect day rates to exceed £1000 (exclusive of VAT). All candidates will be assessed based on the value for money they are deemed to present against the requirements.
9. The latest start date has been confirmed as June 2018, can you confirm the earliest start date? Ideally, we would like the desired candidate to start as soon as possible. In reality - taking into account the procurement process and on-boarding - at earliest this would be at some point during the week commencing 19 March 2018.
10. Our specialist's total rate is higher than the framework limit set. Is there a mechanism to include his total rate? (it is lower than the guideline ceiling) The upper limit of £1000 per day is the maximum the FCA will consider but should not be used as a guideline. We would expect candidates to be priced between £650-£1000 (ex. VAT). Though this may seem like a large range, we do not want to limit our options based solely on price. There is no mechanism for suppliers to charge above the framework limit; doing so would make the call-off contract non-compliant.