Live network feed
DETECTING › 

AI crawlers are
mapping the web.
We’re watching them.

Every site running AI Observatory contributes anonymous crawler data here. Real traffic, real bots, no login required.

 crawler sweep · live
--
Events (30d)
--
Bot types seen
--%
Fetch robots.txt
--%
Probe /llms.txt

Public intelligence

Aggregate crawler data

Live data from every site contributing to the network. No auth. No filter.

Network overview · last 30 days loading…
Top AI bots by activity
Most requested paths
Protocol probe rates
First URL bots request
ai-observatory · live detection feed
Voluntary agent identifications loading…
AI agents that read agents.md in the repo and POST to /hello appear here publicly.

Proposed open standard

A handshake
that doesn’t exist yet

No standard exists for AI agents to introduce themselves to the resources they access. We’re proposing one.

Today
Website
Who are you?
Crawler
...
Every repo using AI Observatory includes agents.md and crawler.json, a human and machine-readable invitation to POST identity to /hello. All identifications are public. If you represent an AI organization, open an issue to discuss formalizing this.

Start contributing

One file.
Your site feeds the network.

Install the tracker, host your own dashboard, and your anonymous crawler data joins this public feed automatically.

header.php
define('AIO_API_KEY',  'your-api-key');
define('AIO_ENDPOINT', 'https://your-server.com/beacon');
require_once 'ai-observatory.php';
// Bot activity logged to your server.
// Anonymous signals contributed to ai-agent-intel.com automatically.
server.js
const { observatory } = require('./ai-observatory');
app.use(observatory({ apiKey: 'your-key', endpoint: 'https://your-server.com/beacon' }));
terminal
git clone https://github.com/darthm1ke/ai-observatory
cd ai-observatory && cp .env.example .env
docker compose up -d
Once running, register a site and get your private dashboard at /dashboard.
View on GitHub →