How It Works
Prowlo turns the communities you watch into a clean, searchable, agent-ready Dataset. Here's what happens between a Reddit post and a record your agent can query.
The pipeline
Every record in your Dataset has been through five steps:
- Watch — you define which sources Prowlo reads
- Crawl — a worker fetches new posts from only those sources
- Index — matching posts are normalized and stored in your Dataset
- Embed — every record gets a vector embedding for semantic search
- Serve — you query the Dataset from the dashboard, an agent, or the API
Watch
You create Watchers that define what Prowlo ingests — one channel each: a subreddit, a Reddit user, a keyword search across all of Reddit, Hacker News, a Mastodon account, an RSS feed, or an X account — plus optional intake filters that decide which posts are worth indexing. There is no global crawl: with no Watchers, Prowlo makes zero requests. You always control exactly what it reads. See Watchers.
Crawl
A crawl worker fetches only your watched channels, through a residential-proxy stack, on a tier-bounded cadence — roughly hourly for active communities, slower for quiet ones. Keyword-search channels are re-searched about once a day, ingesting the newest matches. Each channel keeps a cursor of the newest post it has seen, so every crawl pulls only what's new ("stop at seen") rather than re-reading the whole feed.
Index
Posts that pass a Watcher's intake filters are normalized — title, body, author, score, comment count, timestamps, links — cleaned, tagged by your tagging rules, and stored as records in your Dataset. The Dataset is the asset: a typed, schema-stable corpus that is yours and grows over time.
Field freshness
A record's score and comment count reflect the moment it was crawled; they aren't continuously updated. Opening a record in the dashboard refreshes its live stats.
Embed
Every record is embedded into a vector with Google Vertex AI, stored in PostgreSQL via pgvector. That powers semantic search — finding records by meaning, not just exact words.
Prowlo v1 is embeddings-only: each record is its crawl fields plus one embedding, with no heavy per-record LLM scoring. The philosophy is that your agent classifies on the way out — you decide what "relevant" means for your use case. Records may also carry a lightweight intent signal (BUYING, SELLING, or DISCUSSING).
Serve
Once a record is in your Dataset, you can reach it four ways:
- Dashboard — browse your Dataset and run keyword or semantic search
- MCP — let an AI agent query and reason over it (see MCP Integration)
- REST API — call the v1 endpoints from your code (see API Reference)
- Alerts & webhooks — get pushed new matching records as they appear
Prowlo is read-only throughout — it reads public data and serves it back to you. It never posts, comments, or votes.