Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

Introduction to Prompt Tax 00:00

• Andrew Thompson introduces the concept of "prompt tax" as a challenge for those building AI systems.
• The talk is structured into four sections: the pain of progress, shipping at the frontier, battle-tested tactics, and considerations for managing prompt tax.

The Pain of Progress 00:12

• Rapid advancements in AI models create a sense of excitement and challenge for developers.
• Developing with new AI functionalities can lead to unintended consequences, much like learning to ride a bike.
• There is a constant tension between leveraging new opportunities and managing the risks of regressions or unexpected behaviors.

Shipping at the Frontier 01:51

• Thompson discusses his company, Orbital, which automates real estate due diligence to expedite property transactions.
• The agentic software significantly reduces the time lawyers spend on manual document analysis.
• Since launching their first product, Orbital Co-Pilot, the company has scaled from zero revenue to multiple seven figures in annual recurring revenue within 18 months.

Evolution of AI Models 06:06

• Orbital's product has evolved from using GPT-3.5 to more advanced models, with a significant increase in token consumption from less than 1 billion to nearly 20 billion per month.
• Key decisions included optimizing for prompting over fine-tuning, leveraging domain expert knowledge, and relying on subjective feedback over rigorous eval systems.

Battle-Tested Tactics 11:56

• The team learned to simplify command instructions for new AI models, focusing on clear objectives instead of detailed tasks.
• Feature flags, commonly used in software development, are also applicable when rolling out new AI models to manage risk and user anxiety.
• A mantra of betting on future AI capabilities helps guide product development.

Feedback Loops and Risk Management 17:07

• Strong feedback mechanisms are essential for rapid adjustments to the product based on user experiences.
• An example is provided where feedback is instantly evaluated and incorporated back into the system for continuous improvement.

Challenges and Future Considerations 20:29

• As AI technology evolves, product teams face the challenge of unpredictability in capabilities and user needs.
• There is speculation on whether an evaluation system could help manage the complexities introduced by rapid advancements in AI models.

Conclusion and Call to Action 23:39

• Thompson emphasizes the importance of shipping products quickly to take advantage of new AI capabilities while managing uncertainties.
• He encourages the AI engineering community to share additional tactics and collaborate on future innovations, highlighting the need to stay at the forefront of AI development.