AI Product Analyst

AI Product Analyst

Our client is building cutting-edge AI products for their customers and for colleagues across the group, in tax, HR, health and safety, and care. We can see that people use them; what we cannot yet see with any precision is how well they perform - for which clients, in which sectors, on which kinds of question - and therefore where to invest next.

This role answers that. You will own product analytics for our AI products: measuring performance across the dimensions that matter, building the reporting and dashboards the team and the business work from, and drawing out the insight that sits behind the figures. The purpose is to give the business a reliable evidence base for its decisions about where to invest.

You will also own the evaluation programme. Where analytics establishes how the product performs in the hands of real users, evaluation establishes whether a change is good enough to release. You will own the datasets the product is tested against and the analysis of what the results mean, working closely with our AI Domain & Workflow Specialist.

This is a fantastic opportunity to join a respected, fast-growing AI division based full-time out of our London office. You'll work alongside a highly technical engineering team to shape what ships, rather than just describing it after the fact.

Day-to-Day Responsibilities

Product Performance & Analytics: Measure how our AI products perform across retrieval quality, correctness of output, and user engagement.

Observability Management: Own production quality observability, tracking metrics like thumbs-down rates, regeneration rates, task abandonment, and usage drift.

Dashboard Engineering: Build and maintain the daily analysis and dashboards the AI team relies on, working with engineers to instrument and capture missing data points.

Strategic Insight & Segmentation: Interrogate data using cohort comparison, trend analysis, and segmentation (by client type, subject matter, and question type) to identify high-material patterns and outliers.

Evaluation Programme Ownership: Define what gets tested, analyse evaluation outputs for regressions, and collaborate with engineers to integrate testing harnesses into the build pipeline.

Golden Dataset Curation: Build, version, and maintain curated sets of questions and known-good answers, running elicitation sessions with subject-matter experts to capture professional edge cases.

Cross-Functional Reporting: Translate complex findings into actionable recommendations for the Product Manager and Director to shape priorities and inform the broader roadmap.