🦸🏻#5: Building Blocks of Agentic Systems

Profiling

Profiling is where an agent is assigned roles that shape its behavior. It defines the agent’s identity, objectives, and boundaries, setting guidelines for interactions with users and systems. By adapting to specific tasks or user preferences, profiling ensures the agent stays aligned with its purpose.

For example, a "Coder Agent" can handle repetitive programming tasks, debug code, or create new scripts, similar to GitHub Copilot. A "Content Creator Agent" might draft articles or design graphics, while a "Project Manager Agent" could prioritize tasks and manage workflows. These roles help agents focus on their domains, improving efficiency and easing workloads for users.

Knowledge

The agent’s knowledge provides domain-specific expertise, helping it understand tasks and make decisions based on factual data. This is often achieved using pre-trained AI models, structured knowledge bases, and mechanisms for continuous learning.

For example, models like GPT or Llama enable natural language understanding, as seen in tools like IBM Watson Health. Knowledge bases offer organized references, such as legal databases for case law. Continuous learning allows agents to adapt and stay relevant, ensuring they handle industry-specific tasks effectively.

Memory

At first glance, you might think memory is simply part of knowledge. While they are closely related, memory is a distinct component with its own research foundation. An agent’s knowledge base typically consists of semantic memory – general facts, concepts, and rules about the world, as well as instructions for handling queries.

Memory, on the other hand, extends beyond semantic knowledge. It includes the ability to store and retrieve interaction-specific data, such as user input from previous queries, past experiences, and their outcomes. This allows agents to adapt and improve over time. Memory systems, encompassing short-term, long-term, episodic, and semantic components, enable agents to retain and reuse information from past interactions.

Effective memory implementation ensures that critical data is saved and accessible, empowering agents to inform current decisions with past insights, maintain continuity in conversations, and enhance user interactions.

Reasoning and planning

To effectively navigate tasks and achieve goals, an agent relies on its reasoning and planning capabilities. This involves breaking down tasks (task decomposition), analyzing them to identify the best course of action, and orchestrating the necessary steps toward success. By applying logical reasoning, the agent uses AI algorithms and heuristics to tackle complex situations, facilitating problem-solving, task decomposition, and strategic planning. Goal management plays a critical role, allowing the agent to set, prioritize, and adapt goals based on importance and feasibility.

Reasoning and decision-making are integral to an agent’s functionality, where logical rules and algorithms help draw conclusions and make decisions based on its knowledge base. Before responding to a query, the agent generates a sequence of actions to ensure a reliable outcome. Planning techniques such as reflection, self-critique, chain-of-thought reasoning, and subgoal decomposition are often employed, enabling agents to operate with precision and adaptability.

Reflection

Although we mentioned reflection as a technique in the previous section, Andrew Ng highlights its importance as a standalone category in agentic workflows. He identifies it as one of four key design patterns poised to drive major progress in AI workflows this year and the next. Here’s why:

Reflection enables agents to process feedback and learn from their experiences. It’s a powerful yet straightforward design pattern that boosts performance by allowing models to critique and refine their outputs. Instead of depending solely on user feedback, the model evaluates its responses critically, identifies improvements, and revises accordingly. For instance, when generating code, the model can check its output for correctness, style, and efficiency, then refine it to produce better results. Iterative reflection often leads to significant enhancements.

Beyond self-reflection, external tools like unit tests or web searches can further validate outputs, helping models identify and resolve errors. Multi-agent frameworks take this concept even further – one agent generates content, while another critiques it, enabling collaborative improvement. Reflection has consistently shown its value across various tasks, including coding, writing, and question answering.

Actions

Finally, the agent's ability to take actions bridges its internal reasoning with the external world, enabling it to achieve its goals through precise execution. This involves function calling as a core mechanism, where the agent interacts with APIs, software, or hardware to perform tasks, seamlessly integrating with external tools or services. By invoking these functions, the agent can communicate with users, systems, or other agents, while following through on its planned steps to complete tasks. Effective action involves calling appropriate methods, accessing services or databases, and monitoring outcomes to guide future decisions, ensuring the agent operates efficiently and effectively.

Connecting these components requires a unified framework that supports all aspects of the agent's functionality. Seamless data exchange between components ensures that, for example, memory feeds into reasoning, and reflection informs future planning. Building components as modular units allows for independent updates and improvements. The operational flow typically starts with profiling to define the agent's role, followed by knowledge to provide foundational information. Memory retains experiences, reasoning and planning devise strategies, actions execute plans, and reflection evaluates outcomes to influence future reasoning and behavior.

Human-AI Communication

Often overlooked in agentic architecture, communication plays a crucial role in connecting agents with their environments. As AI agents become more integrated into our daily lives, the field of Human-AI interaction is soon to be on everybody’s mind.

An intriguing perspective on this topic comes from the paper Mutual Theory of Mind for Human-AI Communication. Researchers from Georgia Tech introduce the Mutual Theory of Mind (MToM) framework, offering a novel approach to enhancing communication between humans and AI. This framework moves beyond traditional human-computer interactions, emphasizing a collaborative model where AI systems adapt and engage in more meaningful and intuitive ways.

Inspired by human Theory of Mind, the MToM framework highlights how humans and AI can build and refine mental models of each other. By focusing on mutual interpretation and feedback, it enables AI systems to adjust their responses to better meet user needs. Through stages of construction, recognition, and revision, these systems develop a more intuitive understanding of their role in interactions.

The researchers tested this approach in practical settings, such as online learning, where AI teaching assistants adapted their behavior based on student input. They also studied user responses to AI errors, especially when the AI misrepresented personal traits. Their findings underline the importance of trust and clear communication in bridging gaps between user expectations and AI capabilities.

This work offers valuable insights into designing AI systems as collaborative partners. By improving understanding between humans and AI, these systems could support applications in areas like education and personal assistance, while fostering more responsible and inclusive designs.

Conclusion

By wiring together these core components – profiling, knowledge, memory, reasoning/planning, reflection, and actions – you create an AI agent capable of sophisticated autonomous behavior. Each component plays a critical role, and their integration ensures the agent can perceive, decide, and act effectively within its environment. In the following episodes of this series, we will explore each of the core elements of the agentic workflow in a separate edition, providing you with the latest practical and theoretical insights.

The key to a successful agentic system lies in how well these components communicate and support one another to achieve the desired objectives. As we explore the technologies that bring these agents to life, the focus should expand beyond what makes an agent work to how they thrive within a connected ecosystem – and how we, as humans, will communicate and collaborate with them.

Bonus: Resources to dive deeper into Agentic workflows

• User Behavior Simulation with Large Language Model based Agents by Wang et al. (Submitted in Jun 2023, last revised Feb 2024)

• The Rise and Potential of Large Language Model Based Agents: A Survey by Xi et al. (September, 2023)

• AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors by Chen et al. (Submitted in Aug 2023, last revised Oct 2023)

• Designing Foundation Model based Agents by Lu et al. (Submitted in Nov 2023, last revised Apr 2024)

• Generative Agents: Interactive Simulacra of Human Behavior by Park et al (Submitted in Apr 2023, last revised Aug 2023)

• What's next for AI agents ft. LangChain's Harrison Chase (March, 2024)

• Andrew Ng on Reflection

• Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents by Liu et al. (May, 2024)

• Mutual Theory of Mind for Human-AI Communication by Wang and Goel (May, 2024)

• A Survey on Large Language Model based Autonomous Agents by Wang et al.

• AI Agents That Matter by Kapoor et al (July, 2024)

• Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agents by Lu et al. (Submitted in Nov 2023, last revised Apr 2024)

• Tracing the path to self-adapting AI agents at Microsoft Research blog (July 2024)