SUMM

The speaker introduces the topic of building enterprise-aware agents and considers how AI brilliance can be applied to the complexities of enterprise operations.
Discusses the common question: should one build workflows or agents?
Workflows are described as systems where LLMs/tools are orchestrated via predefined code paths, either through imperative code or declarative graphs.
Workflows provide predictability and structure; their repeated runs yield consistent results.
Agents are systems where LLMs dynamically direct their own processes, planning and executing steps iteratively to achieve a goal.

Workflows are compared to Toyota: predictable, efficient for repetitive tasks, lower cost and latency, and easier to debug.
Agents are compared to Tesla: open-ended, suitable for research and new problems, leverage LLM improvements, but higher cost/latency and less predictable.
Workflows give greater human control, while agents enable more AI-driven decision-making.
The choice between workflows and agents is difficult and depends heavily on the current capabilities of LLMs.
Some tasks may migrate from requiring workflows to being handled by agents as LLMs improve.

It is possible to blend the strengths of both approaches instead of choosing one over the other.
Agents can be viewed as systems that generate workflows on the fly for each task.
Workflows can serve as evaluation benchmarks for agents by comparing an agent’s execution trace to an established “golden” workflow.
Workflows can help train agents; agents can mimic library workflows for known tasks while using reasoning for novel tasks.
Agents can assist in creating workflows, with users providing high-level descriptions and agents outlining workflow steps that users can edit.
Agents can serve as engines for discovering new workflows as users interact with them; successful agent executions become new workflow templates.

Even with future AGI, knowledge of a company’s processes is vital; an untrained AGI is likened to an intelligent new hire who lacks context and onboarding.
Enterprise-aware AGI would combine intelligence with detailed knowledge of a company’s nuances, best practices, and protocols.
There is a qualitative difference between merely acceptable outputs and great outputs that comply with specific business metrics or protocols.

Two main approaches are discussed for leveraging enterprise workflows and tasks: fine-tuning and dynamic prompting.
Fine-tuning:
- Includes supervised fine-tuning (matching input/output) and RLHF (using ratings or reward signals).
- Works well with large, stable datasets.
- Challenges include keeping pace with evolving LLMs, retraining for data/process changes, and limited flexibility for personalized workflows.
Dynamic prompting through search:
- Build a search engine to retrieve similar past tasks/workflows and use as context/examples for LLMs at runtime.
- Allows flexibility between determinism (for known workflows) and creativity (for novel tasks).
- Enables easier adaptation to changing requirements and personalization.

Fine-tuning is effective for generalized behaviors with stable requirements, but inflexible with change.
Dynamic prompting enables customization and rapid updates, better suited to environments with quickly changing or diverse needs.
Fine-tuning is likened to building custom hardware (highly optimized but hard to change), while prompting is like software (more flexible).

Workflow/task search is similar to document search and should utilize both lexical and semantic (vector) similarity.
In enterprise settings, textual similarity is often insufficient due to many similar-looking workflows.
The concept of "authoritiveness" is key—incorporating social and usage signals (e.g., who created the workflow, success rates, endorsements) is necessary to surface the right workflows.
These signals are hard to encode in LLMs, hence the need for specialized search systems.

Workflows provide determinism and human control; agents offer flexibility and AI-driven discovery.
Synergies include using workflows for agent evaluation and training, and leveraging agents for workflow discovery.
Fine-tuning is suitable for generalized, stable behaviors; dynamic prompting excels at personalization and adaptation.
In the brief Q&A, the speaker notes that data requirements for fine-tuning depend on how different the target tasks are from the LLM’s internal knowledge, and offers to discuss further after the talk.

How to build Enterprise Aware Agents - Chau Tran, Glean