A modular voice AI platform based on LiveKit and OpenAI Realtime API, integrating RAG knowledge retrieval, JWT authentication, and MongoDB persistence for real-time outbound calling and domain-specific conversations.
Overview#
SmartCall-Agent is a modular voice AI platform built with Python (FastAPI). It integrates the LiveKit Agents framework with the OpenAI Realtime API to initiate real phone calls via SIP Trunk and conduct low-latency voice conversations.
Core Features#
RAG (Retrieval-Augmented Generation)#
- Uses Pinecone vector database for document embeddings storage
- Generates embeddings via OpenAI
text-embedding-3-small - Supports automatic extraction and chunking of PDF, TXT, DOCX, and web URLs (1000 chars/chunk, 200 chars overlap)
- Namespace isolation (
agent_{agent_id}) for multi-agent knowledge separation
Outbound Calling#
- Initiate outbound calls via LiveKit SIP Trunk
- Real-time audio stream processing
- Voicemail detection and human transfer support
Voice AI#
- Integrates OpenAI Realtime API for real-time voice conversations
- TTS/ASR support via OpenAI voice models
- Configurable voice, tone, temperature parameters
User & Permission Management#
- JWT authentication (HS256 algorithm)
- bcrypt password hashing
- User registration/login/logout APIs
Data Persistence#
- MongoDB for Agent, User, CallLog, Document metadata storage
- Pinecone for vector embedding storage
Observability#
- Structured logging
- Health check endpoint
/health - Analytics dashboard endpoint
/api/analytics/dashboard
Architecture Components#
| Component | File | Responsibility |
|---|---|---|
| FastAPI Backend | main.py | RESTful API, user auth, Agent CRUD, document vectorization, call initiation |
| LiveKit Agent Worker | worker.py | Independent process running KnowledgeAgent, handling real-time voice streams and tool calls |
| MongoDB | - | Structured data storage (User, Agent, CallLog) |
| Pinecone | - | Vector data storage (Document Chunks) |
Key API Endpoints#
| Path | Method | Function |
|---|---|---|
/health | GET | Health check |
/api/register | POST | User registration |
/api/login | POST | User login (returns JWT) |
/api/agent | POST | Create Agent (with RAG docs) |
/api/agent/{agent_id} | GET/PATCH/DELETE | Get/Update/Delete Agent |
/api/agents | GET | List all Agents |
/api/agent/{agent_id}/query | GET | Query Agent knowledge base |
/api/call | POST | Initiate outbound call |
/api/calls | GET | Get call records |
/api/analytics/dashboard | GET | Dashboard metrics |
Deployment Requirements#
- Python 3.10+
- OpenAI API Key
- Pinecone API Key
- LiveKit Cloud account with SIP Trunk
- MongoDB instance
Use Cases#
- Automated customer support callbacks
- Telemarketing campaign execution
- Appointment reminders and notifications
- Voice Q&A systems based on private knowledge bases