AI is Eating Search

Origin Story and AI's Impact on Web Interaction 00:15

  • Scrunch AI was founded after recognizing the disruptive potential of LLMs on how people interact with businesses online.
  • Early on, businesses mainly wanted chatbot plugins, but the founders realized most consumers don't want to engage with site-specific chatbots.
  • The core insight was that AI is fundamentally altering online discovery and customer journeys, not just traditional SEO.

Monitoring AI Search and Content Discovery 06:52

  • Scrunch monitors how major AI platforms (ChatGPT, Gemini, Perplexity, Claude, Meta AI) surface and reference businesses in their search and conversational interfaces.
  • Focus is on platforms with significant user adoption; the coverage will expand as new platforms emerge.
  • Many brands are more concerned with AI search features (search box enabled) because user actions with commercial intent increasingly involve search, which pulls real-time or recent web content.
  • For pre-training crawlers, brands can control what is exposed, particularly avoiding including sensitive or short-lived promotion info.

Web Content, Paywalls, and the AI Content Ecosystem 09:53

  • There is growing debate and experimentation around compensating content creators for AI's use of their data (e.g., Cloudflare's paywall for bots).
  • Businesses must balance content restrictions with their need for exposure and traffic via AI/search engines.
  • New browsing paradigms (AI browsers, agent-based retrieval) are emerging, making it harder to entirely block LLMs from consuming web content.
  • The shift is not just AI replacing search but AI potentially replacing much of traditional web browsing for many user intents.

How AI Search Works & Optimization Challenges 16:04

  • Most AI search platforms still use traditional ranking algorithms (BM25, PageRank, etc.) before applying AI-based re-ranking and summarization.
  • Traditional SEO practices continue to matter, but effective re-ranking depends on having descriptive, relevant page metadata and content.
  • Structurally clear, informative pages perform better in AI-generated responses than those optimized just for human clickthrough or visuals.

Strategies for Content Exposure and Personalization 19:59

  • Businesses must decide how much product/context detail to publish for AI consumption—more detail improves discovery and inference by AI models.
  • AI search does not yet pass user query data to destination sites, mainly for privacy reasons, but inbound traffic analysis can guide content development.
  • Real-world example: Monitoring what topics or products inbound AI-driven visitors engage with helps refine content for future queries.

Hallucinations, Content Clarity, and SEO Evolution 22:42

  • AI models sometimes hallucinate features or information; this can reveal demand for non-existent features, which can guide product/content development.
  • Clear, straightforward descriptions of offerings on product and homepage pages are crucial, especially for startups and B2B SaaS providers.
  • There is growing attention on how different AI systems re-rank and select content, akin to historical SEO tracking of Google algorithm updates.

Naming the Space and the 'Agent Experience' Era 26:35

  • The field is sometimes termed AEO (AI Engine Optimization) or GEO (Generative Engine Optimization), but Scrunch prefers to frame it as “agent experience.”
  • Unlike classic SEO, the goal expands from being discovered to managing the full AI-mediated user journey, as more customer actions complete within AI tools.

Personalization, Prompt Clusters, and Use Case Observations 29:04

  • Personalization and user profiles (memories, preferences) already influence AI search results and content selection, with increasing effect.
  • Common high-value prompt clusters: coding, shopping, B2B software comparisons, and broader problem-solving rather than classic “search” intent.
  • Many business users leverage AI to directly solve problems (“do the thing for me”), which is higher intent than information gathering alone.

Tactics: Programmatic Content, Technical Dos and Don'ts 39:00

  • LLMs and AI search engines do not execute JavaScript when indexing; ensure key content is server-rendered for inclusion.
  • File formats like LLM.txt (or similar) are not yet major factors in discoverability—traditional, well-structured HTML content remains primary.
  • Over-focusing on embeddings and “semantic” hacks yields little benefit for exposure; clarity and structure are more effective.
  • Programmatic SEO/content generation is widely used; it works best if content brings unique, helpful insights rather than just regurgitated web material.
  • Effective cases include turning support tickets into structured how-to content for AI and users.

Case Studies and Uplifts 47:30

  • Clerk (developer authentication) saw a 6x increase in AI-driven traffic and a 9x increase in conversions by optimizing docs and content for AI platforms.
  • Scrunch acts as a feedback/experimentation tool rather than a content generation service, supporting customers to iterate based on data.
  • Typical uplift for actively engaged clients is double-digit percentage traffic growth within 1-2 months.

Market Share and Platform Dynamics 51:38

  • ChatGPT is the dominant AI platform in terms of traffic, engagement, and brand relationship, with AI overviews (Google, Bing) gaining exposure.
  • Perplexity follows as a strong, niche favorite, especially in tech/early adopter circles, though still smaller in volume.
  • Meta AI and Claude have passionate user bases but lower market share compared to ChatGPT.
  • Market shares are volatile; new entrants can spike in usage but ChatGPT has had the most enduring traction.

Best Practices and Closing Thoughts 55:05

  • The web is rapidly shifting toward AI as a primary interface; being “AI compatible” is increasingly critical for exposure and conversion.
  • Avoid “snake oil” solutions; focus on clear, honest, well-structured content that helps both humans and AI agents.
  • Scrunch is actively hiring, especially for those interested in AI/search/web intersections.