A benchmark platform for evaluating the performance of AI large language model agents on executing real-world professional tasks in a simulated software company environment, featuring diverse task roles and a comprehensive scoring system.
One-Minute Overview#
TheAgentCompany is an innovative AI agent benchmarking platform designed to evaluate the performance of large language model (LLM) agents on executing real-world professional tasks in a simulated software company environment. It simulates the workflow of digital workers by having AI agents browse the web, write code, run programs, and communicate with colleagues. This project is valuable for researching the impact of AI on the labor market, assessing AI agents' workplace capabilities, and promoting AI adoption in business workflows.
Core Value: Provides the most realistic benchmark for evaluating AI agents in professional work environments, filling a critical gap in assessing AI agent performance on consequential real-world tasks.
Quick Start#
Installation Difficulty: Medium - Requires Docker and Docker Compose knowledge, 30GB+ disk space, and network access permissions
# Linux/Mac users
sudo chmod 666 /var/run/docker.sock
curl -fsSL https://github.com/TheAgentCompany/the-agent-company-backup-data/releases/download/setup-script-20241208/setup.sh | sh
Core Capabilities#
1. Diverse Task Roles - Simulating Real Work Environments#
- Multiple roles including Software Engineer, Product Manager, Data Scientist, Human Resources, Financial Staff, and Administrator
2. Diverse Data Types - Comprehensive Capability Testing#
- Covers coding tasks, conversational tasks, mathematical reasoning, image processing, and text comprehension
3. Multiple Agent Interaction - Team Collaboration Assessment#
- Supports interaction and collaboration between multiple AI agents
4. Comprehensive Scoring System - Precise Performance Evaluation#
- Result-based primary evaluation and secondary checkpoint systems
5. Multiple Evaluation Methods - Flexible Testing Options#
- Deterministic evaluators and LLM-based evaluators
Technology Stack & Integration#
Development Languages: Python, Shell Major Dependencies: Docker, Docker Compose, GitLab, Plane, ownCloud, RocketChat, LiteLLM, OpenHands (optional)