Agentic Architecture

Key Characteristics of AI Agency

An AI agent is a sophisticated software entity capable of perceiving its environment, reasoning about its observations, and taking autonomous actions to achieve specific goals. The efficacy and intelligence of an AI agent are underpinned by a set of fundamental characteristics that, when integrated, enable complex and adaptive behaviors. These core pillars are Autonomy, Reasoning, Goal-Directed Behavior, Planning, and Memory.

Autonomy

Autonomy in an AI agent refers to its capacity to operate independently, making decisions and initiating actions without direct human intervention. This is not a binary state but rather a spectrum that ranges from simple, rule-based systems to highly adaptive, self-governing entities.

The degree of autonomy is determined by the agent’s internal logic, which can be anything from a predefined script to a complex, learned policy that maps observations to actions. A key trade-off in designing autonomous agents is balancing predictability with adaptability. Greater autonomy allows for more flexible and robust behavior in dynamic environments, but it can also make the agent’s actions harder to predict. The level of autonomy is defined in the following maturity model, where each level represents a progressive shift of control to the agent:

• Level 1 – Assisted: tool invocation on demand, no persistence.
• Level 2 – Semi‑Autonomous: scheduled triggers, basic plans, RAG grounding.
• Level 3 – Autonomous with Guardrails: dynamic replanning, Human-In-The-Loop, policy shields.
• Level 4 – Coordinated Multi‑Agent: role specialization, negotiation, shared rewards.
• Level 5 – Self‑Improving: continuous reflection, simulators, safe self‑play, formal verification for safety-critical actions.

Reasoning

Reasoning is the central cognitive process that enables an AI agent to interpret information, draw inferences, and make informed decisions. It is the mechanism by which an agent moves from raw data to actionable intelligence. AI reasoning can be categorized into several types, each mimicking a different aspect of human cognition:

Advanced agents are increasingly capable of not just providing an answer but also articulating the step-by-step reasoning process that led to it, enhancing transparency and trust in their decisions.

GoalDirected Behaviour

A defining characteristic of an AI agent is its goal-directed behavior. Unlike simple automated scripts that execute a predefined sequence of tasks, an agent is driven by one or more objectives and will dynamically choose its actions to best achieve them. These goals can be explicitly provided by a user or implicitly learned through mechanisms like reward functions.

Planning

Planning is the process through which an agent determines how to achieve its goals. This may involve:

• Breaking down complex objectives into smaller actions or subtasks.
• Sequencing actions logically, respecting constraints and dependencies.
• Using heuristics (“if X happens, do Y”) for fast, rule-based decisions.
• Applying advanced reasoning to evaluate multiple possible plans, simulate outcomes, or optimize execution.

Effective planning allows agents to work on long-running tasks, coordinate multiple steps, and anticipate obstacles. A planning-capable agent can rethink its approach if conditions change or if it encounters an unexpected issue.

Memory

Memory supports learning, continuity, and context awareness. It typically consists of three layers:

• Short-term memory (Session): Maintains the context of the current interaction, such as the active conversation, ongoing tasks, or intermediate reasoning. This allows continuity across dialogue turns and multi-step tasks.
• Long‑term memory (Profile): Stores persistent knowledge about users, preferences, instructions, or experiences. Supports personalization and consistent behavior over time.
• Domain Knowledge Memory: Retrieves factual information, documents, or structured data that the agent uses to reason or respond accurately (this method is called Retrieval Augmented Generation – RAG – see Grounding & Retrieval (RAG) Techniques section for further details). Ensures the agent has access to up‑to date, domain specific‑specific knowledge.

Memory enables the agent to become smarter over time, avoid repetitive errors, maintain context across interactions, and act with increasing sophistication.

How Planning differs from Reasoning

Planning and reasoning are closely related, but they serve different purposes within an AI agent’s decision‑making process. Here’s a clear, concise explanation of how they differ:

Purpose

Reasoning is about thinking: drawing logical conclusions, understanding relationships, inferring what is true, and evaluating options.

Planning is about acting: deciding which sequence of steps to take to achieve a goal.

In a nutshell:

Reasoning = understanding

Planning = doing (based on that understanding)

Scope

Reasoning deals with abstract logic, problem understanding, and interpretation.

Example: “If the system is overloaded, performance will degrade. Increasing capacity can prevent failure.”

Planning operates at the level of actions and timelines.

Example: “First provision two new servers, then migrate workloads, then verify performance.”

As a result, reasoning can exist without resulting in action, while planning always leads to actionable sequences.

Input and Output

Reasoning

Input: Knowledge, facts, rules, prior experience

Output: Conclusions, judgments, predictions, or evaluations

Planning

Input: Goals, constraints, resources, and (often) the results of reasoning

Output: A structured action plan or policy

Reasoning supports planning by helping the agent understand the situation and evaluate choices before committing to actions.

Time Orientation

Reasoning may be immediate or context-specific – evaluating a situation as it is now.

Planning is inherently future-oriented – projecting actions into the future to achieve an objective.

Dependency Relationship

Planning depends on reasoning, but reasoning does not depend on planning.

Reasoning is the cognitive engine, whereas planning is the execution blueprint.

An agent usually reasons first (“What’s the best approach?”) and then plans (“Here’s the step-by-step path.”).

Simple Analogy

Imagine a person preparing a trip:

Reasoning:

“Driving will take 5 hours, but the weather is bad, so maybe taking the train is safer.”

Planning:

“Buy train tickets, pack luggage, leave home at 7 AM, get to the station by 7:30.”

Reasoning decides what makes sense.

Planning decides what to do next.

From theory to practice – Basic Agent Architecture

Building on the characteristics outlined in the previous section, we can now introduce the components required to form a basic AI agent architecture. Each characteristic corresponds to a specific capability or service within the system.

The first core capability is Reasoning, which agents achieve by using Large Language Models (LLMs) to interpret user inputs and guide their decision-making. Agents operate with defined objectives (Goal‑Directed Behaviour) and follow specific instructions provided through system or user prompts.

Before taking action or producing outputs, the agent must plan (Planning) – again leveraging LLMs to determine the best course of action. If an objective requires knowledge the agent does not inherently possess, it gathers the needed information from external systems (Memory) using available tools. The agent may also invoke external services – such as MCP servers or REST APIs – to perform actions that help fulfill its assigned task (Autonomy).

Finally, once the necessary steps are completed, the agent generates a response and delivers it back to the requester.

The following schema illustrates the most basic core components and their interactions in an AI agent:

Delving into the details of the above picture, leads to the sequence diagrams illustrated below.

Sequence Diagram 1: tool calling is added to the sequence diagram to depict the action / knowledge retrieval performed by the agent.

The sequence flow (as illustrated in the numbered dispatched messages) is the following:

    1a User Prompt

    2 Set Context System Instructions + User Prompt (Prompt Management)

    Main Agentic Loop (until task completes)

    3 Send Request with Context to LLM

    4 LLM process & decide next action

    If next action is Tool / MCP Call Request

    5 Return Tool / MCP Call Request

    6a Execute Tool / MCP Function (Tools)

    6b Return Tool Result

    7 Update Context with Tool Result

    Else Mark task complete

    8 Return Final Response

    1b Response (to user)

Sequence Diagram 2: apart from tool calling, memory is also added to illustrate how the agent persists context updates and results:

The sequence flow (as illustrated in the numbered dispatched messages) is the following:

    1a User Prompt

    2 Set Context System Instructions + User Prompt (Prompt Management)

    3 Save Prompt (Memory)

    Main Agentic Loop (until task completes)

    4 Send Request with Context to LLM

    5 LLM process & decide next action

    If next action is Tool / MCP Call Request

    6 Return Tool / MCP Call Request

    7 Save Response (Memory)

    8a Execute Tool / MCP Function (Tools)

    8b Return Tool Result

    9 Save Updated Context with Tool Result (Memory)

    10 Update Context with Tool Result

    Else Mark task complete

    11 Save Final Response

    12 Return Final Response

    1b Response (to user)

Architectural Components

A complete architecture of a tool-enabled agentic system includes:

A. The Agent Core

• LLM: Planning, reasoning, critique, tool selection.
• Policy / Guardrails: Ensures compliance, privacy, and safety.
• Agent Runtime: Handles routing, retries, context, and execution logic (in some cases it is called “orchestrator”, however it mustn’t be confused with the Orchestrator agent in multi-agent systems, see Low-Code / No-Code Platforms’ Interoperability.
• Memory: Short-term and long-term context persistence.

B. Tooling Layer

• Tool Registry: Catalog of available tools with metadata, schemas, and versioning.
• Schema Validator: Ensures type safety and argument correctness.
• RBAC & Permission Engine: Controls which tools are available to which roles.
• Human-in-the-Loop (HITL): Approval workflows for sensitive actions.

Execution Layer:

• Sandboxed code environments
• API connectors
• RPA and workflow engines

C. Retrieval and Grounding Layer

• Vector databases
• Document stores and enterprise knowledge bases
• SQL / NoSQL data environments
• Web search/browsing tools

D. Security and Observability

• Full audit logging of tool calls
• Traceability across agent decisions
• Data Loss Prevention (DLP)

E. Enterprise Systems Integration

Includes the full ecosystem where tools operate:

• ERP / CRM
• ITSM systems
• Developer platforms
• Cloud infrastructure
• Messaging and storage systems

The following sections introduces the components in more details.

Memory

In the previous sections, we have already discussed the necessity of incorporating a dedicated memory component within an agentic architecture. Such a component enables the agent to maintain context, retain knowledge, and reason more effectively over time. To implement this capability in a robust and scalable manner, each distinct memory type must be backed by the appropriate technology stack. The following mapping outlines the recommended technologies for each memory category:

Short-Term Memory

Short-term memory is designed for fast, transient storage of contextual data – such as conversation state, temporary variables, or recent interactions.

Recommended technology: Redis

Redis provides in-memory data storage with extremely low latency, making it ideal for rapid read/write operations and ephemeral data retention.

Long-Term Memory

Long-term memory must support structured, durable storage of information that the agent may need to retrieve over extended periods.

Recommended technology: PostgreSQL

PostgreSQL offers reliable persistence, strong consistency guarantees, and rich relational capabilities, making it well suited for long-term state, logs, or historical data.

Knowledge Memory – Semantic Search (Vector Database)

To enable semantic retrieval, embeddings must be efficiently stored and searched using vector similarity operations.

Recommended technologies: PostgreSQL + pgvector**,** Qdrant

PostgreSQL with pgvector extends the relational database with native vector types and similarity indexing, allowing seamless integration with structured data.

Qdrant is a high-performance, purpose-built vector database optimized for large-scale embedding stores and real-time semantic search.

Note: See PostgreSQL + pgvector vs. Qdrant – Vector Search Comparison for a comparison between the two products.

Knowledge Memory – Entities and Relationships (Graph Database)

When representing knowledge as interconnected concepts – such as entities, attributes, and relationships – a graph data model is more appropriate.

Recommended technology: Neo4j, FalkorDB

Neo4j provides expressive graph modeling, efficient traversal algorithms, and a mature query language (Cypher), making it ideal for building knowledge graphs and enabling reasoning over relationships.

FalkorDB is a low latency graph database, optimized for real-time AI applications.

Note: FalkorDB Runs as a module in RedisServer

References:

https://www.falkordb.com/blog/falkordb-vs-neo4j-for-ai-applications/

https://www.falkordb.com/falkordb-vs-neo4j/

Prompt Management

Prompt management encompasses the full set of platform capabilities dedicated to designing, organizing, maintaining, and continuously improving the prompts that guide AI agents. This includes creating new prompts, structuring them into reusable libraries, implementing version control to track changes over time, and running systematic evaluations or experiments to measure their effectiveness. It also involves refining and optimizing prompts to enhance agent reasoning, improve tool orchestration, and generate more accurate and contextually appropriate natural language outputs.

Examples of Prompt Management in Action

Versioning Prompts for an AI Customer Support Agent

Scenario: The support team updates the AI’s “Troubleshooting Connectivity Issues” prompt to reduce unnecessary escalations.

Action:

• v1: Basic troubleshooting steps.
• v2: Adds clarifying questions for modem type.
• v3: Optimized wording to reduce false positives when identifying network outages.

Outcome: Ticket deflection rate improves after A/B testing shows v3 resolves more issues without escalation.

Maintaining a Central Prompt Library Across Teams

Scenario: Multiple teams (Support, Sales, Product) use AI agents and need consistent definitions of product features.

Action:

• Create a shared prompt library with components like “Product Description,” “Safety Guidelines,” and “Brand Voice.”
• Teams assemble agents using these standardized building blocks.

Outcome: All agents speak with the same voice, avoid contradictory answers, and simplify governance.

Automated Evaluation and Testing of Prompts

Scenario: A QA team wants the AI to reliably summarize documents without leaking sensitive data.

Action:

• Run nightly tests where the AI processes sample documents.
• Evaluate output for correctness, redaction compliance, and hallucinations.
• Flag prompts that degrade after model updates.

Outcome: Early detection of regressions ensures safe and consistent summarization quality.

Optimizing Tool-Use Prompts for Higher Accuracy

Scenario: An AI agent frequently uses a Calendar API but sometimes misinterprets time zones.

Action:

Update the prompt to explicitly instruct:
“Always include event start and end times in ISO 8601 with user’s local timezone.”

Add examples demonstrating correct and incorrect formats.

Outcome: Tool execution accuracy improved reducing booking errors.

Prompt Experiments to Improve Reasoning

Scenario: A data analysis agent struggles with multi-step reasoning in complex tasks.

Action:

• Test variants of reasoning strategies:
• “Chain-of-thought” prompt
• “Plan-before-solving” prompt
• “Tool-first reasoning” prompt
• Compare performance across tasks while hiding internal reasoning in the final output.

Outcome: The “Plan-before-solving” version yields the best reliability on multi-stage calculations.

Localizing Prompts for Different Markets

Scenario: An AI assistant needs to serve users in Greece, Germany, and Brazil.

Action:

• Adapt prompts to reflect local language, cultural norms, regulatory requirements, and date formats.
• Maintain versions by locale (e.g., billing_de, billing_el, billing_pt-br).

Outcome: The assistant feels natural and compliant in each region.

Debugging Unexpected AI Behavior

Scenario: An agent starts generating overly verbose answers after a platform update.

Action:

• Inspect prompt history and dependencies.
• Identify an inherited system prompt update that encourages long explanations.
• Override with a concise-response directive in the local prompt.

Outcome: The agent returns to expected behavior without requiring code changes.

Guardrails

Guardrails are one of the most critical components in the design, deployment, and operation of AI agents – especially those intended to operate autonomously, integrate with enterprise systems, or take actions on behalf of users. Guardrails are policies, constraints, and mechanisms that ensure an AI agent behaves safely, ethically, and predictably within its intended scope. Think of them as the combination of traffic rules, seatbelts, and crash barriers that allow powerful AI systems to operate without causing harm.

The Benefits of Guardrails

A. Safety and Harm Prevention

AI agents can take actions – send emails, write code, modify infrastructure, interpret documents, or make decisions. Without guardrails, these actions may produce:

• Harmful outputs
• Financial or operational risks
• Privacy violations
• Security breaches
• Bias or unfair outcomes

Guardrails minimize these risks by enforcing policy-aligned behavior and preventing unsafe or unintended actions. Some of the key safety roles are the following:

• Prevent generation of harmful instructions
• Enforce ethical use
• Avoid prohibited content
• Redirect unsafe queries to safer alternatives

B. Security and Protection of Sensitive Environments

An AI agent integrated with enterprise systems may have access to:

• Internal documents
• Customer data
• APIs and knowledge bases
• Workflows or automated processes

Guardrails ensure that the agent does not misuse privileged access. Some examples of security guardrails are the following:

• Role-based access control (RBAC)
• Data-loss prevention (DLP) rules
• Prompt filtering and input validation
• Output monitoring and redaction

By enforcing the appropriate measures application owners prevent:

• Unauthorized data exposure
• Unintended system changes
• Propagation of malicious inputs (prompt injection)

C. Alignment with Organizational Policies

AI agents must follow organizational rules just like employees, therefore guardrails ensure that:

• The AI respects company compliance standards
• Communications match brand tone and approved terminology
• Outputs follow legal, regulatory, and audit requirements

D. Reliability and Predictability

For AI agents to be trusted, their behavior must be consistent, the employment of guardrails ensures that:

• Stable outputs in repeat scenarios
• Controlled autonomy (bounded decision-making)
• Prevention of hallucinations or fabricated information

The benefits of having reliable and predictable agentic systems, boosts confidence for:

• Automation workflows
• Decision support systems
• User-facing chat experiences

E. Mitigation Against Adversarial Behavior

AI agents can be vulnerable to the following misuse:

• Prompt injection
• Jailbreaking
• Model exploit attempts
• Social engineering through input manipulation

Guardrails such as layered defenses, content filters, input sanitizers, and behavior monitoring help protect against these adversarial threats.

F. Ethical and Responsible AI Practices

Guardrails embed ethical principles into the agentic system:

• Fairness
• Transparency
• Privacy
• Inclusiveness
• Accountability

They ensure the AI system operates within human-defined moral and societal boundaries and does not promote:

• Harmful stereotypes
• Discriminatory actions
• Misinformation

Guardrails Types in AI Systems

A. Model-level guardrails

• Safety fine-tuning
• Reinforcement learning from human feedback (RLHF)
• Policy training

B. Prompt-level guardrails

• Prompt templates
• Instruction tuning
• Input/output filtering

C. System-level guardrails

• Access control
• Audit logs
• DLP, filtering, encryption
• API Gateways

D. Application-level guardrails

• Workflow constraints
• Context boundaries
• Rate limiting
• Fallback behaviors

Guardrails - The 7-Layers

Layer 1 – Access Control & Authentication

Goal: Prevent unauthorized access from users or systems.

Mechanisms:

• RBAC (role-based access control) / ABAC (attribute based)
• Identity integration (Entra ID)
• API key and OAuth enforcement
• Context-aware access policies

This prevents unauthorized users from interacting with high‑power agents.

Layer 2 – Data Boundary Guardrails

Goal: Ensure the agent only accesses data it is allowed to.

Mechanisms:

• Fine-grained data access policies
• Row/column-level access controls
• DLP filters
• Retrieval-augmentation filtering
• Tenant isolation for multitenant systems

This stops sensitive data from leaking through prompts or RAG.

Layer 3 – Input Guardrails

Goal: Validate and sanitize what the user or external systems send to the agent.

Mechanisms:

• Prompt injection detection
• Regex and semantic filters for forbidden content
• Input type and schema validation
• Intent classification before execution

This protects the agent from hostile or malformed input.

Layer 4 – LLM-Level Guardrails

Goal: Prevent unsafe generation inside the model.

Mechanisms:

• System prompts with enforced persona/constraints
• Content filters (toxicity, self-harm, hate, sexual content, illegal acts)
• Role-aligned prompt engineering
• Safety-tuned models (custom fine-tuned models)

This ensures the model internally rejects harmful requests.

Layer 5 – Output Guardrails

Goal: Apply filters to agent response before returning it.

Mechanisms:

• Output redaction (PII, secrets, harmful content)
• Policy alignment checking
• Tone & compliance validation
• Factuality verification (hallucination detection)
• Action validation (e.g., “is this email safe to send?”)

This is the last line of defense before the user sees the output.

Layer 6 – Action Guardrails (For Autonomous Agents)

Goal: Prevent unsafe or irreversible actions.

Mechanisms:

• Action validation layer
• Policy-based “allow/deny/approve” logic
• Transaction limits (e.g., spending caps)
• Safe-mode simulation before real execution
• Human-in-the-loop approval flows
• Guardrail gating for high-risk operations

This protects enterprise systems when the agent has real authority.

Layer 7 – Observability, Monitoring & Audit

Goal: Make AI behavior transparent, explainable, and auditable.

Mechanisms:

• Telemetry
• Prompt/response logs (with PII redaction)
• Risk analytics dashboards
• Drift detection
• Incident response pipelines
• Compliance audit trails

No enterprise AI solution is complete without this layer.

The following diagram illustrates the function of the guardrails sub-system:

The next diagram shows the sequence of actions of the guardrails sub-system:

Resiliency

AI-powered applications depend heavily on LLMs, but what happens when the main LLM provider / service / endpoint suddenly becomes unavailable? A simple network issue or provider outage can bring an entire AI system to a standstill. In this section, practical disaster‑recovery strategies for LLM-based architectures are being laid out, like how to respond when an LLM provider stops responding, how to switch to alternative providers and how to use LLM gateways to enable smooth failover.

No AI provider offers 100% uptime – even industry leaders have occasional outages or rate-limit issues. If the system depends on a single LLM API and that API fails to respond, users may see errors or stalled features. For critical apps, even a brief outage can hurt user trust and business operations. Thus, architecting for failure is essential.

There are typically three main reasons why an LLM API might fail

• Hitting rate limits
• Exceeding usage quotas (similar to rate limits, but enforced over longer periods)
• Network issues or server-side failures (such as HTTP 500 errors)

Note: There are other reasons an API request might fail – such as invalid or incorrect credentials, or improperly formatted parameter values – but these issues typically stem from the client rather than the provider.

Rate & Quota Limits

When a rate limit or quota cap is exceeded, the service providers (typically) respond with an HTTP 429 error. These limits can apply to:

• Requests per minute (RPM)
• Tokens per minute (TPM)
• Concurrent connections
• Daily or monthly usage quotas
• Shared resource limits (workspaces, API keys, teams)

To avoid hitting 429s, techniques fall into four categories:

• Traffic shaping
• Error-handling + retry strategies
• Optimization of prompt and token usage
• Infrastructure‑level scaling strategies

Traffic shaping Techniques

A. Implement Client‑Side Rate Limiting

Use leaky‑bucket or token‑bucket rate limiters to ensure your application never exceeds documented limits (see Confront LLM Rate Limits Efficiently for more detailed information).

Example strategies:

• Fixed request spacing: sleep (60 / rpm_limit)
• Distributed token bucket shared across workers
• Queues with controlled dequeue rates

This is the single most effective way to eliminate 429s.

B. Use Asynchronous Batching

Many LLMs allow batching or “completions.create_batch” calls.

Benefits:

• Fewer requests → fewer opportunities to exceed RPM
• More efficient throughput

Example use cases:

• Embedding many items at once
• Classifying multiple documents per request

C. Avoid Burst Traffic

Most APIs enforce short‑window limits (e.g., per 1-10 seconds), not only per minute.

To avoid micro-bursts:

• Pace requests evenly
• Randomize request intervals (jitter)
• Avoid batch jobs that trigger mass parallel calls at the same second

Error‑Handling and Retry Strategies

A. Use Exponential Backoff

A standard defensive pattern:

• Retry after: 1s, 2s, 4s, 8s, etc.
• Add jitter (random delta) to avoid thundering herds after outages

retry_wait = base * 2^attempt + random_jitter()

Backoff must be combined with client-side limiting, retries alone can worsen congestion.

B. Honor Retry‑After Headers

Many LLM APIs return the HTTP header:

Retry-After: <seconds>

Always respect this value. It precisely indicates when the rate window resets.

C. Queue and Re-Route Failed Requests

If the system is high-volume:

• Push requests into a message queue (i.e. Kafka see Event Bus section)
• Let worker pools drain the queues at safe speeds
• Re‑queue on temporary 429 errors

This ensures no data loss and controlled throughput.

Token Usage Optimization

429 errors can be caused not only by RPM limits but also by TPM (Tokens per Minute) limits. To minimize TPM usage:

A. Shorten Prompts

Reduce:

• System instructions
• Repetitive context
• Long conversation histories

Use compression techniques:

• Summarize long chat history in place of full logs
• Embed + retrieve small snippets rather than large documents

B. Use Context Caching and RAG

Instead of sending the entire document:

• Pre-embed documents once (see Grounding & Retrieval (RAG) Techniques)
• Retrieve only the relevant paragraphs
• Send a minimal prompt to the LLM

This dramatically reduces TPM consumption and keeps you under limits.

C. Use Smaller Models When Appropriate

Large models can consume more tokens and often have stricter limits.

Choose:

• Smaller LLMs for simple classification
• Larger ones only when necessary

TPM usage can be decreased by 50–80% when using model‑size tiering.

Infrastructure-Level Scaling Strategies

A. Use Multiple API Keys or Applications (If Allowed)

Some service tiers allow:

Per-key rate limits

Per-application or per-endpoint throttles

You can allocate:

One key per service

One key per processing node

B. Upgrade Service Tier or Apply for Higher Limits

Cloud providers often offer:

• Throughput upgrades
• Dedicated capacity
• Enterprise rate tiers

For Azure OpenAI, for example:

• You can increase TPM/RPM per deployment
• Add more model deployments to parallelize load

C. Use a Load Balancer Across Deployments

If the provider allows multiple “model deployments” you can:

• Route requests across replicas
• Multiply effective rate limits
• Reduce hotspots

D. Introduce Local or On‑Prem Models for Bulk Processing

Implement hybrid setups:

• Run small tasks on local LLMs (Llama‑3, Mistral 3, Gemma 3 families)
• Send only high‑value tasks to cloud LLMs

With this approach, the gains are:

• Reduces request pressure
• Improves cost efficiency
• Lowers probability of 429 error spikes

Monitoring & Observability Techniques

A. Monitor RPM/TPM in Real Time

Use metrics dashboards for:

• Requests/sec
• Tokens/sec
• Success vs. 429 error rates
• Queue backlog size
• Worker utilization

Raise alerts when approaching 80% of your rate limit.

B. Implement Predictive Load Control

Build “load shedding” logic:

• If requests exceed expected capacity → queue/hold/deny gracefully
• Ensure upstream systems never overload the LLM backend

Summary

The following table summarizes the techniques discussed in previous paragraphs to cope with API rate limit and usage caps for LLM services.

See also Python Sample Middleware for LLM APIs for sample Python code that handles the 429 errors.

Tools / APIs Integration

LLMs have demonstrated unprecedented capabilities in reasoning, planning, and natural language understanding. However, LLMs alone cannot perform deterministic operations, interact with enterprise systems, or execute tasks beyond text generation. To operate as agents, LLMs require controlled extensions known as tools – API endpoints, code execution environments, retrieval systems, or automation frameworks that act as the agent’s hands and eyes in the real world.

Tools extend the AI agent’s capabilities beyond text prediction (probabilistic reasoning), enabling it to grounded, verifiable action, making them suitable for enterprise workloads that demand reliability, traceability, and compliance.

In this context, a Tool is a formally-defined, permission-controlled function that an AI agent can invoke to perform an action outside the model’s native capabilities.

Tools typically include:

• Retrieval: Vector search, knowledge base queries, SQL access.
• Computation: Python/JS sandboxes, mathematical libraries.
• Action APIs: CRM, ERP, ITSM, CI/CD, cloud resource management.
• Automation: RPA flows, workflow engines, orchestration platforms.
• Observation: Web search, telemetry queries, system status checks.

A tool is not simply a backend integration, it is an AI-introspectable abstraction with a schema the agent can interpret and reason about.

The following diagram illustrates the sequence of events when the agent needs to get data or act with the help of a tool (MCP / REST API).

Why Tools Are Essential for Agentic Intelligence

A. Grounding and Factual Accuracy

LLMs cannot natively access real-time or enterprise-managed data. Tool integration provides authoritative sources that reduce hallucination and enable fact-based reasoning.

B. Deterministic Computation

LLMs perform arithmetic and logic inconsistently. Tools enable precise, auditable computation.

C. Enterprise Interoperability

Tools are the mechanism through which AI links with mission-critical systems like CRM (Siebel), ERP (SAP), databases (Oracle, Snowflake, PostrgreSQL), code repositories (GitHub), office suites (Microsoft 365), and cloud platforms.

D. Autonomous Task Execution

Agentic systems must break tasks into steps, validate assumptions, and execute actions. Tools enable multi-step autonomy with controlled actions.

E. Safety, Governance, and Accountability

Tools are permission-gated, logged, monitored, and governed – allowing enterprises to safely adopt advanced AI capabilities without losing control.

Agent to Agent Integration

Agent-to-agent integration is the set of mechanisms, architectures, and protocols that enable autonomous agents to work together, coordinate, communicate, and achieve both individual and shared goals. In modern distributed AI systems this integration is not optional, it is the foundation upon which multi-agent systems functionality is built.

Note: Although agent-to-agent integration is a fundamental capability of multi-agent systems, it is presented in this section, alongside all other major agentic functionalities.

Currently, the community accepted agent-to-agent protocol is Google’s Agent-to-Agent – A2A (see MCP, A2A & ACP for more information regrading integration protocols).

The primary benefit of the A2A protocol is its universal interoperability, enabling AI agents – s regardless of their underlying frameworks, vendors, or platforms – to seamlessly communicate, collaborate, and coordinate tasks in a standardized, secure, and scalable way.

A2A achieves this by using common web standards like HTTP, Server-Sent Events (SSE), and JSON-RPC, which makes integration with existing enterprise systems straightforward. This eliminates the need for custom connectors or proprietary APIs, allowing agents from different ecosystems to work together without compatibility issues.

Key characteristics:

• Cross-Platform Collaboration – Agents can discover each other’s capabilities via Agent Cards and dynamically form multi-agent workflows, even if they don’t share memory or tools.
• Secure by Design – Built-in enterprise-grade authentication and authorization ensures safe data exchange between agents, protecting sensitive business information.
• Support for Complex, Long-Running Tasks – Agents can manage extended workflows with real-time updates, making it ideal for scenarios like supply chain optimization, hiring pipelines, or research tasks.
• Multi-Modal Communication – Supports text, audio, video, and interactive elements, enabling richer and more adaptive collaboration.

Agent Registry

An AI Agent Registry is essentially a central catalog or directory where AI agents are described, versioned, and managed – similar to ML model registries or how service registries are used in microservice architectures.

As organizations increasingly deploy specialized AI agents for various tasks, a system to manage and orchestrate them becomes crucial. An AI Agent Registry provides the necessary framework for this, offering several key benefits (depending on the agent registry implementation):

Event Bus

In modern multi‑agent AI systems, the communication substrate is as important as the agents themselves. An event bus / message queue system is the backbone enabling coordination, resilience, and scalability.

Note: Although agent-to-agent integration is a fundamental capability of multi-agent systems, it is presented in this section, alongside all other major agentic functionalities.

The benefits of using a queueing system are laid out in the following paragraphs.

Decoupling Agents (Spatial, Temporal, and Logical Independence)

Agents should not be tightly coupled to the availability, identity, or internal logic of other agents.

Without a queue:

• Agent A must call Agent B synchronously.
• If B is down, A fails.
• If B changes schema or behavior, A must update too.

With a queue:

• A publishes a message.
• B subscribes and processes when ready.
• They evolve independently.

This decoupling allows agents to scale, be replaced, or updated without cascading failures.

Handling Asynchronous, Long‑Running, or Uncertain Workflows

Multi-agent environments frequently involve:

• Long‑running tasks (simulation, planning)
• Variable latency (LLM calls)
• External APIs with unpredictable response times

A queue allows:

• Fire‑and‑forget tasks
• Retries
• Scheduled or delayed execution
• Prioritization strategies (critical vs background tasks)

Asynchronous workflows become native, not an afterthought.

Concurrency Control & Load Regulation

Agents can easily overwhelm each other.

The queue:

• Meters how fast events are consumed
• Absorbs bursts (shock absorber)
• Controls max concurrency
• Provides backpressure

This ensures:

• No agent is overloaded
• APIs aren’t rate‑limited or blocked
• System throughput stays stable

Event-Driven Coordination & Emergent Behaviors

Multi-agent systems typically rely on:

• State changes
• Triggers
• Signals
• Policies
• Tool-use decisions

A queue enables these behaviors naturally:

New document ingested → Trigger analysis agent → Trigger summarization agent → Trigger reviewer agent → Publish result to orchestrator.

This creates loose but intelligent chaining: agents listen to events relevant to their domain and react autonomously.

Reliability, Persistence, and Guaranteed Delivery

Agents may fail, restart, be updated, or temporarily disconnect.

A queue provides:

• Message persistence
• “exactly once” or “at least once” semantics
• Dead-letter queues
• Durable state for workflow continuity

This is critical in AI systems where tasks might take minutes, hours, or involve external dependencies.

System Observability & Auditing

The event bus becomes a central source of truth for:

• What happened
• When
• Who triggered what
• How long it took

This enables:

• Logging
• Tracing
• Replayability
• Debugging complex agent interactions

In multi‑agent ecosystems, this visibility is crucial for safety and correctness.

Cost Control

AI agents typically depend on LLM calls to interpret instructions, generate responses, reason about tasks, or take actions in an environment. Each of these calls consumes tokens – units of text used to feed prompts into the model and retrieve outputs. Because most commercial LLMs operate on a pay‑per‑token basis, every request translates directly into a monetary cost.

As a result, agentic systems must implement robust mechanisms for LLM token accounting. This includes monitoring how many tokens are used per prompt and per response, calculating cumulative usage across the agent’s operations, and estimating or tracking the associated financial cost. Effective token accounting allows developers to optimize agent behavior, enforce cost limits, detect inefficiencies, and make informed decisions about prompt design, model selection, and execution strategies.

In practice, this may involve logging token consumption for every call, aggregating usage for long‑running workflows, applying heuristics to minimize unnecessary LLM interactions, or dynamically adjusting the agent’s reasoning depth based on budget constraints. Token visibility is essential not just for cost management, but also for ensuring predictable, scalable, and sustainable operation of AI‑driven systems.

One way to prevent LLM token costs from rising quickly, is by caching responses wherever possible. See next section for caching strategies in agentic systems.

Links:

https://pypi.org/project/tiktoken/

https://platform.openai.com/tokenizer

Caching

Agentic AI introduces new considerations due to the nature of multi-turn interactions / conversations that is based on.

Every LLM model call incurs costs, and often it takes time to generate the response to a query. Caching as a technique to speed up execution and increase user experience in all kinds of systems, is not new. Agentic systems aren’t the exception to this, thus several caching strategies tailored for AI agents have been emerged to help applications perform well, fast and be as cost‑efficient as possible.

Semantic Caching

Semantic caching represents one of the biggest changes in how caching is conceptualized and applied in APIs. Traditional caches rely on exact matches. – pairing a specific key to a specific resource. Semantic caching, by contrast, takes context and meaning into account, enabling an AI agent to identify the best match even when requests are phrased differently.

Consider a user asking an AI assistant, “How do I recover my password?” versus “I can’t log in to my account.” While these questions look different, they point to the same intent – something that’s difficult to express with strict key-based logic. Semantic caching uses embeddings (vector representations of text) to find and return responses that are meaningfully similar when no exact match exists.

To make semantic caching effective, it’s crucial to set an appropriate similarity threshold. Too narrow, and you’ll miss valid matches, too broad, and you may return irrelevant results. This problem is similar to a RAG system where grounding truth is contained in documents (see Grounding & Retrieval (RAG) Techniques for more information).

Types of Caching

Different agentic workflows require different caching solutions.

A. Response Caching

This is the simplest and most common method, storing an LLM’s response so it can be reused. It’s ideal for relatively static domains such as FAQ answers. Combining response caching with semantic caching allows for correct responses even when user phrasing varies. Versioning can also be added to track the freshness of stored responses.

C. Embeddings Caching

Embedding caching stores vector embeddings for known inputs to avoid repeated computation. This is especially useful in systems like product recommendation engines. The trade-off is that embeddings can become outdated as the underlying data evolves, so periodic regeneration is necessary to maintain accuracy.

D. Workflow‑level Caching

Agentic systems often involve multiple interconnected components. Workflow‑level caching captures and stores the outputs of each stage in a multi‑step process. For instance, a travel app may need to combine route data, flight schedules, and pricing. Caching each component’s results can make retrieval far faster, easier, and more cost‑effective.

Logging & Monitoring

Agentic AI systems differ fundamentally from traditional software: they act with autonomy, make context‑dependent decisions, call tools, interact with external systems, and evolve their internal state as they operate. Because of this, logging and monitoring become not just operational necessities, but core enablers of safety, reliability, and trust. Below is a structured deep dive into the most important aspects.

Observability Enables Safety and Controllability

Agentic systems exhibit emergent behavior. Without proper tracing, you cannot reliably understand why an agent took a particular action.

Key Goals

• Traceability: Every decision, action, and tool call must be logged.
• Explainability: Logs allow engineers to reconstruct the agent’s reasoning.
• Intervention Ability: Monitoring alerts help you stop runaway or harmful-agent behaviors.

Example

A retrieval‑augmented agent that autonomously updates a knowledge base may:

• Query internal APIs
• Generate transformations
• Execute actions with external systems

If something unexpected happens (i.e. the agent loops or corrupts data) only robust logs allow you to identify the root cause and intervene.

Detecting Hallucinations, Drift, and Misalignment

Agentic LLMs are probabilistic. They can:

• Hallucinate tool arguments
• Misinterpret user intents
• Misuse capabilities
• Drift over long sequences of reasoning steps

Monitoring reduces the risk of silent failures. Essential Monitors:

Drift detection is especially important for agents that run continuously or process diverse, unpredictable prompts.

Performance and Cost Optimization

Agentic applications often chain multiple LLM calls, external tool operations, and context retrieval steps. This introduces:

• Latency variance
• Resource spikes
• Potential cost explosions

Monitoring allows you to optimize for

• Token usage per agent step
• Depth of reasoning chains
• Frequency of tool calls
• Execution time across workflows
• Bottlenecks between model calls and external systems

Without monitoring, an agent could silently generate a massive bill or degrade performance.

Security and Compliance

Autonomous AI systems introduce new security attack vectors:

• Prompt injection
• Data exfiltration
• Credential misuse through tool calls
• Unauthorized request amplification

Security logs are mandatory for compliance reasons (i.e. GDPR, SOC 2, ISO‑27001). Some of the actions that must be logged:

• Data accessed by the agent
• Authentication and authorization steps
• Tool actions affecting external systems
• Sensitive data flowing through prompts or model outputs

Monitoring enables:

• Auditing
• Forensics
• Detection of malicious input patterns

Debugging and Continuous Improvement

Agentic apps rarely behave consistently without iteration. Observability allows developers to:

• Understand failure modes
• Improve prompt engineering
• Refine tool schemas
• Update reasoning strategies
• Tune fallbacks or guardrails

Useful logging types are:

• Input/Output logs for every model call
• Chain-of-thought approximations (short-form reasoning traces)
• Tool call events
• Error and exception logs
• State transitions

Without these, debugging becomes guesswork.

Human‑in‑the‑Loop Oversight (HITL)

Not all autonomous actions should proceed without approval. Logging and monitoring enable:

• Real‑time dashboards to track agent behavior
• Approval gates for sensitive tool actions
• Human review loops
• Rollbacks if needed

This preserves both safety and accountability in systems with high autonomy.

Evaluation and Benchmarking

Monitoring supports continuous evaluation in production:

• Success/failure rates of tasks
• Accuracy of retrieved results
• Quality of generated actions
• Alignment with user instructions

This enables creation of:

• Leaderboards for agent performance
• Regression detection systems
• Feedback-based improvement loops

Agentic Patterns

One Shot Generation

This is the most basic pattern that accepts the user prompt and generates the response (assuming no specific tools are needed), it’s a one shot LLM query:

Prompt à Generate

This pattern is only for demonstration purposes to show the usage of guardrails in input (user prompt) and output (LLM / agent response) messages and should not be implemented in production agents.

Prompt à Validate à Generate à Validate Response à Accept

In case validation fails for input or output content, the agent should respond with a corresponding error.

The flow is illustrated ion the following diagram:

Note: In the following sections, both input and output validations (guardrails) are assumed to exist, even though they are not shown in the diagrams or described in the process flow. This omission is intentional to keep the focus on the core pattern and its benefits.

Self-Reflection & Refine Loop

This pattern resembles the human learning cycle:

Generate à Reflect à Refine

The agent’s output is continuously evaluated until it reaches an acceptable level of quality response. If the output is below the acceptable threshold, the agent revises its work based on feedback. The flow is illustrated ion the following diagram:

The steps of the process are:

1. Receive user input
2. Generate initial response based on user input
3. Evaluate the response and provide feedback
4. Check the score of the provided response
5. Revise if the score is below the accepted threshold
   1. Both the response and the feedback (critique) are used in next iteration
6. If the response is at acceptable level, accept it

Warning: To avoid endless loops in self-reflection, it is essential to define stopping criteria that prevent the system from continuing the process indefinitely. These criteria can include a fixed number of iterations, a threshold for quality or correctness, or a maximum time limit for the reflection process. By implementing these stopping criteria, the model can ensure that the self-reflection loop is not prolonged unnecessarily, allowing for a more efficient and effective improvement process.

Reason & Act

In this pattern the agent has been assigned a tool to use to resolve the user query.

The agent’s output is continuously evaluated until it gets the information that satisfies the user question. The flow is illustrated in the following diagram:

The steps of the process are:

1. Receive user input
2. The agent thinks whether it can generate answer directly, or it needs to use an external tool to get more information
3. If more information is required, it selects a tool to use
4. The agent executes the tool (i.e. a web search tool to get public information)
5. The tool results along with the user query are used in step #2 (loop).
6. If the response is valid, return it

A basic system prompt for accomplishing this task:

You will be given a user question.

You can use the following tools to retrieve information to better answer the query:

• Web Search: Searches the web for information

• Weather Tool: Retrieves weather information for specific location

• …

Query: [QUERY]

Context: [CONTEXT]

Note: Always request structured output from LLMs, in order to semantic information (if needed) in responses and allow the consumer (agent) to easily parse the generated response.

A sample process flow (“dialog”) in the agent is the following:

— User query —

Who is the CEO of the company that is the most valuable brand in Europe and what is the gross revenue of that company?

— Thinking (LLM Prompt) —

System: You will be given a user question. Use the following tools to better answer the query:

• Web Search: Searches the web for information

User: Who is the CEO of the company that is the most valuable brand in Europe and what is the gross revenue of that company?

— Thinking (LLM Response) —

Use Tool: Web Search

— Tool Call —

Results:
“Telekom increases brand value significantly by 16.4 percent to 85.3 billion US dollars. Company is once again the most valuable telco brand in the world and the most valuable corporate brand in Europe”
[https://www.telekom.com/en/media/media-information/archive/telekom-achieves-highest-brand-value-in-its-30-year-history-1086040]

…

— Thinking (with Tool Call Results) —

…

— LLM Response —

Telekom is the most valuable telco brand in Europe. There was no information regarding revenue numbers.

Note: Since the example above uses a web search tool, it is essential that the search results are re-ranked before being returned to the caller (see Re-ranking for more information).

Planning

The Reason & Act pattern was able to get the correct response for one part of the question, however it missed the second. What is missing is a clear breakdown of the tasks (a plan) the agent needs to perform to successfully resolve the user query. The Planning pattern helps creating a list of tasks that will allow the agent to reach the requested goal. The flow is illustrated in the following diagram:

The steps of the process are:

1. Receive user input
2. Decompose the query to smaller tasks so as to satisfy the request
3. Sequentially or in parallel (if there are no dependencies between tasks) the agent executes the tool
4. When all required information is assembled, the agent synthesizes the final response

A basic system prompt for accomplishing this task:

Given a user question, decompose it into simpler sub‑questions whose answers would collectively yield the correct solution.

A sample process flow (“dialog”) in the agent is the following:

— User query —

Who is the CEO of the company that is the most valuable brand in Europe and what is the gross revenue of that company?

— Thinking (LLM Prompt) —

System: Given a user question, decompose it into simpler sub‑questions whose answers would collectively yield the correct solution.

User: Who is the CEO of the company that is the most valuable brand in Europe and what is the gross revenue of that company?

— LLM Response —

1. What is currently the most valuable brand in Europe?

(Identify which company holds the top brand value ranking in Europe.)

1. Which company owns or operates that brand?

(Some brand rankings refer to a brand name; you may need to determine the parent company.)

1. Who is the current CEO of that company?

(Look up the current chief executive officer of the identified company)

1. What is the most recent gross revenue reported by that company?

(From the latest annual report or financial disclosures)

— Agent Execute (LLM) —

[ for each task call the tool & assemble information, example for question #1 ]
System: You will be given a user question. Use the following tools to better answer the query:
- Web Search: Searches the web for information
User: What is currently the most valuable brand in Europe?

— Q1 Thinking (LLM Response) —

Use Tool: Web Search

— Q1 Tool Call —

Results:
“The T brand is the No 1 Most Valuable Brand in Europe in 2025”
[https://www.portugalbusinessesnews.com/post/top-10-most-valuable-brands-in-europe-in-2025]

…

— Q1 Thinking (with Tool Call Results) —

…

— Q1 LLM Response —

The T brand is the No 1 Most Valuable Brand in Europe in 2025.

— Synthesize (LLM) —

[ generate the response ]

— Agent Final Response —

“The most valuable brand in Europe is the T brand of Deutsche Telekom. Its CEO is Timotheus Höttges, and the company’s gross revenue is approximately USD 121 billion.”

Note: The example focused on decomposing the initial request to sub-requests (it is a similar approach to Multi-Query RAG presented later). The planning pattern can generate sub-tasks that need diverse set of skill to complete and is commonly used by agent orchestrators in multi-agent systems (see Centralized Orchestration).

Note: Since the example above uses a web search tool, it is essential that the search results are re-ranked before being returned to the caller (see Re-ranking for more information).

Ensemble

The ensemble pattern is commonly seen as a multi-agent pattern, however it can be implemented as a single agent as well, which uses more than one LLMs to serve the user request.

Process flow:

• Parallel Exploration: A user’s query is sent to multiple specialist agents (or LLMs in the simplest approach) at the same time. These agents are given different personas to encourage diverse thinking.
• Processing / Generation: Each agent / LLM works on the problem in isolation, generating its own response.
• Aggregation: The responses from all independent agents / LLMs are collected.
• Synthesize: A final aggregator (or judge) LLM will generate the final response, based on the feedback from all previous layers. It weighs the different viewpoints and synthesizes a comprehensive response.

The flow is illustrated in the following diagram:

Architecture Blueprint

Based on the architectural components discussed in previous sections, the proposed architecture for implementing AI agents is illustrated in the diagram in following pages.

Refer to Pre-deployment checklist for Agentic AI for a comprehensive list of items to check before releasing an agent (or an AI solution in general) to production.

Guardrails for Input/Output Safety and Compliance

Design Principles:

• Defense-in-depth: Apply guardrails at multiple layers: pre-processing, model-level, and post-processing.
• Policy Centralization: Guardrail policies should be centrally defined, not embedded directly in each low-code / no-code agent, to avoid fragmentation.
• Context-Aware Filtering: Guardrails must consider prompt context, user role, and system state.

Integration Standards:

Mandatory pre-prompt validation:

• PII checks
• Toxicity filters
• Hallucination-sensitive content blockers
• Allowed-action validation based on agent capabilities

Mandatory post-response validation:

• Safety classifier scoring
• Governance policy enforcement
• Response redaction/sanitization

Support for real-time policy updates from a centralized governance engine.

Agent Interoperability & A2A Protocol Adherence

Design Principles:

• Protocol-Centric Architecture: All agents. – regardless of their underlying implementation (low-code / no-code or custom) – must communicate via a standardized Agent-to-Agent (Google’s A2A) protocol, ensuring interoperability across the enterprise.
• Separation of Intent vs. Execution: Agents should clearly differentiate intention messages, action requests, and state updates, as defined by the A2A spec.
• Stateless Core Interactions: Where possible, inter-agent communication should be stateless, pushing temporal context management to orchestrators or session managers.

Integration Standards:

• Must support A2A message schemas (JSON/Protobuf defined by the enterprise).
• Must implement standard lifecycle hooks:

on_intent, on_action_request, on_observation, on_error.

• Must support capability declaration endpoints, enabling dynamic agent discovery.
• Ensure compatibility with enterprise identity & access management.

Human-in-the-Loop (HITL) for High-Risk Agentic Actions

Design Principles:

• Human Override as a First-Class Citizen: Any action that could cause financial, operational, legal, or security damage must require human approval.
• Explainability: For human review, agents must provide rationale, supporting evidence, and confidence scores.

Standardized HITL workflow:

• Submit action for review
• Await human approval/rejection
• Resume execution

All HITL events must be included in OpenTelemetry traces.

Unified Observability with OpenTelemetry

Design Principles:

• End-to-End Traceability: Every agent interaction–prompt input, transformation steps, model invocation, external API actions, and output delivery–should create traceable spans.
• Structured Logging: All logs must be structured, queryable, and persistable via a central telemetry backend.

Integration Standards:

Mandatory support for OpenTelemetry:

• Tracing: A2A message spans, agent invocation spans, model operation spans.
• Metrics: Latency, cost, token usage, error rates, retries, human escalations.
• Logs: All agent events emitted in OpenTelemetry format.

Agents must propagate trace IDs across A2A boundaries.

Operational Resilience & Versioned Delivery

Design Principles:

• Safe Deployments: New agent or workflow releases must go through canary testing, policy checks, and simulation.
• Predictable Versioning: Every agent has a semantic version with compatibility constraints.

Integration Standards:

• Fallback strategies for failed requests or guardrail violations.
• Standard disaster-recovery patterns and retry policies.