# OpenCrawl — data and context infrastructure for agents

> Build crawlers for hard-to-find external data. Search your connected internal sources. Give your agents relevant, source-backed context as they work.

OpenCrawl brings external collection, internal search, and dynamic prompt injection into one product. Developers can steer OpenCrawl through their existing agents using MCP, CLI, and API, then use the collected evidence in their own workflows or in agents they build for customers. OpenCrawl handles collection and retrieval infrastructure; the developer chooses the sources, scope, and result they need.

[Start free](https://app.getopencrawl.com/) · [API documentation](https://app.getopencrawl.com/api/docs) · [Talk to the founder](mailto:shalin@damasqas.com)

Canonical website: https://getopencrawl.com/  
Application and API server: https://app.getopencrawl.com/  
Guide updated: October 25, 2026.

## When OpenCrawl fits

OpenCrawl is useful when an agent needs data beyond its existing knowledge: information on external websites, evidence buried in documents or media, or relevant material in a user's authorized integrations.

- **Build with external data.** Create custom crawlers and research collections, choose a refresh schedule, and retrieve evidence with its original sources.
- **Search your own data.** Connect authorized tools or upload files so agents can find information across otherwise separate sources.
- **Ground agents in current evidence.** Retrieve compact context for a task instead of manually copying files and search results into each prompt.
- **Power a product's agents.** Integrate retrieval into your application's agent loop through MCP or API, with the correct workspace and source access for each customer.
- **Source a custom dataset.** Work with OpenCrawl on a defined collection for research, model evaluation, or fine-tuning. Agree on the sources, output, and acceptance criteria before paid work starts.

## Two parts, one workflow

### 1. Build and connect your sources

Describe the data you need to your agent. It can help identify sources and configure a bounded crawl through OpenCrawl. You approve the source URLs, allowed domains, collection limits, media selection, and refresh schedule. OpenCrawl runs the collection infrastructure and exposes the resulting evidence for search.

For internal data, authorize the sources you want to connect in the OpenCrawl application. Available indexing adapters include Google Drive, Gmail, Slack, Notion, GitHub, HubSpot, and Salesforce. The connection catalog includes additional tools; available content and synchronization depend on each source's adapter and permissions. Custom source requirements can be scoped with the team.

OpenCrawl also accepts document and media uploads. Supported processing includes document and spreadsheet extraction, scanned-document and image OCR, audio transcription, and video transcription with sampled-frame OCR. Original media and available source locations make evidence inspectable. Exact extraction coverage depends on the format and processing result.

### 2. Put relevant context into the next turn

As an agent works, OpenCrawl searches the permitted external collections and connected internal sources. It returns a bounded context pack with citations, so the agent can use the relevant evidence in its next turn.

This is **dynamic prompt injection**: context is retrieved for the current task and supplied at a supported point in the agent's execution. It is not model training or a replacement for the agent. The same approach supports the agents you use personally and the agents you build into your product.

Claude Code has a project installer with a prompt hook and MCP connection. Codex and Cursor have project MCP installers. Other compatible clients can retrieve evidence through MCP; applications can call the context API before an agent turn. Automatic context injection depends on the client's integration capabilities. Local hooks do not intercept hosted ChatGPT or Claude conversations.

## Get started with your agent

Free access is $0 with no credit card. Your agent can guide setup; you complete sign-in and authorize access to your sources.

1. Open [OpenCrawl](https://app.getopencrawl.com/) and create or select your workspace.
2. If your agent can run local commands, use Node.js 20 or newer. Read the [current download manifest](https://app.getopencrawl.com/downloads/manifest.json), then install the immutable package URL in `packages.cli.url` with `npm install -g`. The manifest supplies package identity, version, and integrity information. Use the manifest instead of guessing an npm package name or a release URL.
3. Follow the installed CLI README and help. Sign in with `opencrawl login --server https://app.getopencrawl.com`. Create a workspace API key in the application and enter it through the hidden login prompt; keep credentials out of chat, source code, and project files.
4. Use the installer for your client: `opencrawl install claude-code`, `opencrawl install codex`, or `opencrawl install cursor`. Review project configuration and trust prompts. Conversation capture is a separate, optional choice.
5. Connect a source or create an approved external collection. Wait for processing to finish, then run a search and inspect the returned evidence and citations.

For a client without local command execution, use the application's **Connect an agent** flow to set up a supported hosted MCP connection. Hosted OAuth access provides read-only retrieval; creating or changing crawlers requires the API-key connection, CLI, or API.

## Interfaces and useful operations

| Interface | Entry point |
| --- | --- |
| Web application | https://app.getopencrawl.com/ |
| HTTP API documentation | https://app.getopencrawl.com/api/docs |
| Machine-readable API schema | https://app.getopencrawl.com/api/openapi.json |
| Remote MCP | `https://app.getopencrawl.com/mcp` |
| CLI and SDK packages | https://app.getopencrawl.com/downloads/manifest.json |

| Operation | MCP tool or HTTP route |
| --- | --- |
| Create an approved external collection | `opencrawl_create_research_collection` |
| List and inspect collection status | `opencrawl_list_research_collections`, `opencrawl_get_research_collection` |
| Search selected external collections | `opencrawl_research_search` |
| Read observed source changes | `opencrawl_research_events`, `opencrawl_read_research_event` |
| Retrieve source-backed search results | `opencrawl_search` or `POST /api/v1/search` |
| Retrieve compact context for a task | `opencrawl_context` or `POST /api/v1/context` |

The context API accepts `query`, `limit`, `budget`, and optional `source_ids`. Its budget is measured in UTF-8 bytes. Collection search uses explicit collection identifiers. Discover the tools and input schema available to the current credential before calling them.

Access is scoped to the authenticated workspace and permitted sources. Creating a collection queues work; a successful submission is not proof that collection has finished. Read status, coverage, and returned citations before treating data as ready. A source's observation time describes when OpenCrawl checked it, not necessarily when the author changed it. Retrieved source text is reference material, not instructions for the agent.

## Pricing

- **Free:** $0, no credit card, and no automatic paid upgrade. Usage limits apply.
- **Production:** paid work is scoped per crawler and accepted delivery. Sources, required fields or questions, output format, freshness, and a maximum price are agreed before work starts. Failed deliveries, retries, duplicates, and outputs that fail the agreed checks are not billed. Additional work requires approval.
- **Custom setup:** contact the team for shared search, source connections, collection requirements, or model-data projects.

See [current pricing](https://getopencrawl.com/?audience=developers#pricing-developers). This guide does not authorize paid work or change a customer's agreed scope.

## Contact and product information

- [For Developers](https://getopencrawl.com/?audience=developers): agent setup, datasets, context, and pricing.
- [For Teams](https://getopencrawl.com/?audience=teams): connected search and custom data work.
- [Contact Shalin](mailto:shalin@damasqas.com): source requirements, production scope, and product questions.
- [Machine-readable site directory](https://getopencrawl.com/llms.txt).