SUMM

James Lowe introduces himself as head of AI engineering at the Incubator for AI, a UK government team focused on delivering public good using AI.
The incubator was established by 10 Downing Street and delivers a broad range of AI products, from frontline services to supporting the prime minister.
AI product management is becoming increasingly important due to the reduced cost and increased feasibility of building products with AI.
Product management involves balancing business viability, technology feasibility, and user desirability.
The introduction of AI complicates each of these areas, raising new questions around experimentation, evaluation, user experience, and the probabilistic nature of outcomes.
An AI product manager must have data and AI proficiency, understand the significance of evaluation, and deal with uncertainties unique to AI.
The role of AI product manager is seen as a mindset rather than a fixed position, with value in combining product and engineering expertise into fewer individuals.

The incubator's Consult project involved analyzing large-scale public consultations, which can take months and cost millions.
Initial efforts focused on building a product with existing NLP techniques, but results were inconsistent and did not meet strict legal standards.
The team shifted to first resolving AI capability uncertainties, collecting real and synthetic data, and developing the “themefinder” package.
Outputs using this prioritized AI approach matched human accuracy, were 1,000 times faster, and 400 times cheaper.
Early evaluation helped identify key points for human-in-the-loop involvement and led to a product design different from the initial vision.
The main lesson: resolve AI uncertainties early through evaluation and real-user testing.

The Minute transcription tool aimed to streamline secure AI transcription and summarization for government use.
Available off-the-shelf solutions existed, so the focus was on user experience and feature experimentation.
The team rapidly developed and tested multiple AI-powered features, leveraging quick development and low attachment to features using AI coding assistants.
User feedback indicated the product was initially overwhelming and complex, with many features unused.
By focusing on a narrower user group (probation services), the team removed unnecessary features, merged overlapping ones, and delivered a simpler, more effective product.
The main lesson: experiment extensively with features, then cut back to what works best for users.

Redbox was initially designed to digitize the workflow of government ministers carrying paperwork in physical red boxes.
User testing revealed the key desired feature was the ability to securely chat with large language models using government data.
The project pivoted to focus on secure AI chat, attracting thousands of users in weeks within the cabinet office.
The team identified an opportunity to integrate other tools and datasets into the Redbox chat interface, further increasing its value.
Subsequent market changes (Microsoft releasing free enterprise AI chat, new standardization protocols) forced another pivot, moving the focus from Redbox itself to making the tools accessible in any enterprise environment.
Redbox remains useful, but the speed of change requires continual reevaluation and rapid pivots.
The main lesson: the AI landscape evolves rapidly, so teams must be prepared to pivot faster and more frequently than before.

Lesson zero: An AI product manager (or mindset) with technical expertise is essential.
Lesson one: Evaluate AI early to resolve uncertainties with real-user testing.
Lesson two: Go wide with features, then streamline after testing.
Lesson three: Be ready to pivot quickly due to the fast-changing AI landscape.
While some elements are familiar from traditional product management, AI brings heightened experimentation, evaluation needs, and rapid change, making these lessons especially critical for success.

Why your product needs an AI product manager, and why it should be you — James Lowe, i.AI