redpanda-agents

Redpanda Agent is an enterprise AI Agent Python SDK (requiring Python ≥ 3.13, current version 0.1.0) officially maintained by Redpanda, Inc. Its core design embeds LLM agents as processor plugins within Redpanda Connect streaming pipelines, using gRPC (protobuf v1alpha1) for cross-process communication to reuse the full Redpanda Connect input/output/processor ecosystem.

Core Capabilities#

• Agent Orchestration: Agent class wrapping LLM calls with custom system prompts, Pydantic v2 structured output validation, tool calls, multi-agent in-process orchestration, and lifecycle hooks (on_start/on_end/on_tool_start/on_tool_end)
• Multi-Model Invocation: Unified calls to OpenAI, Gemini, Bedrock etc. via LiteLLM, using "provider/model" format
• MCP Protocol Integration: Full support for four transport modes — SSE, Stdio, HTTP-Stream, WebSocket
• Redpanda Connect Pipeline Integration: Declarative configuration of input/output/tools/tracer; tools defined via mcp/resources/processors/*.yaml reusing Redpanda Connect processor capabilities
• Observability: OpenTelemetry SDK integration with tracing to Jaeger/OTel Collector, covering both pipeline and Python process layers
• Declarative Configuration: YAML (redpanda_agents.yaml) and Starlark (Python dialect) options
• Enterprise Features: Redpanda Broker/ACL/Auth integration for multi-agent authz/authn management

Architecture Overview#

Redpanda Connect Runtime
  (input → processor plugin → output pipeline)
        │ gRPC (protobuf v1alpha1)
        ▼
redpanda.runtime (serve())
  gRPC Server exposing Agent as plugin
        │
        ▼
redpanda.agents (SDK Core)
  Agent · Tool · AgentHooks · MCPEndpoint
  ├── LiteLLM (unified multi-model)
  ├── MCP Client (4 transport protocols)
  └── Pydantic v2 (structured validation)

Key Mechanism: redpanda.runtime.serve() starts a gRPC server exposing the Python Agent process as a callable Redpanda Connect processor plugin; tool definitions are marked via meta.mcp.enabled: true, referenced by label from the Agent, with actual execution delegated to Redpanda Connect processors. Configuration uses a dual-layer structure — redpanda_agents.yaml for pipeline topology, agents/*.py for agent logic.

Quick Start#

Option 1: rpk scaffolding (recommended)

rpk connect agent init my_first_agent
rpk connect agent run my_first_agent

Option 2: Direct install

pip install redpanda-agents

Minimal working code:

from redpanda.agents import Agent
import redpanda.runtime

my_agent = Agent(
    name="my_first_agent",
    model="openai/gpt-4o",
    instructions="These are your instructions - good luck!",
)
asyncio.run(redpanda.runtime.serve(my_agent))

Use Cases#

• Event-driven AI processing pipelines (streaming messages → AI processing → output)
• Inter-agent async communication (via Kafka-compatible protocol)
• MCP toolchain integration (e.g., Redpanda MCP Server)
• Structured data extraction (Pydantic model auto-validation and serialization)
• Enterprise agent deployments requiring audit log persistence

Unconfirmed Information#

• PyPI publication status unconfirmed (GitHub Releases shows no releases)
• Exact rpk connect agent command availability version unconfirmed
• ai.redpanda.com website content not verified
• Audit log shipping to Redpanda/Kafka marked as partial, implementation extent unclear