MCP Integration
MCP Overview
Model Context Protocol server for AI agent integration
Essence includes a built-in Model Context Protocol (MCP) server that exposes scraping tools directly to AI agents like Claude, Cursor, and Windsurf.
Endpoint
GET /mcpThe MCP server uses Streamable HTTP transport and is available at the same address as the REST API.
Available Tools
The MCP server exposes six tools:
scrape
Scrape a single page to Markdown or other formats.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | — | Required. URL to scrape |
formats | string[] | ["markdown"] | Output formats |
engine | string | "auto" | Engine selection |
timeoutMs | integer | 30000 | Timeout in milliseconds |
crawl
Crawl a website following links.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | — | Required. Starting URL |
maxDepth | integer | 2 | Maximum link depth |
limit | integer | 100 | Maximum pages |
includePaths | string[] | — | Glob patterns to include |
excludePaths | string[] | — | Glob patterns to exclude |
allowBackwardLinks | boolean | false | Follow upward links |
allowExternalLinks | boolean | false | Follow external links |
map
Discover URLs via sitemaps and link extraction.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | — | Required. Site URL |
search | string | — | Filter query |
ignoreSitemap | boolean | false | Skip sitemap.xml |
includeSubdomains | boolean | true | Include subdomains |
limit | integer | 5000 | Max URLs |
search
Search the web via DuckDuckGo, optionally scraping results.
| Parameter | Type | Default | Description |
|---|---|---|---|
query | string | — | Required. Search query |
limit | integer | 10 | Number of results |
scrapeResults | boolean | false | Scrape each result |
extract
Extract structured data from web pages using CSS selectors or LLM-based extraction.
| Parameter | Type | Default | Description |
|---|---|---|---|
urls | string[] | — | Required. URLs to extract from (1-10) |
schema | object | — | JSON Schema for output structure |
prompt | string | — | Natural language extraction prompt |
selectors | object | — | CSS selector mappings |
mode | string | "auto" | "auto", "css", or "llm" |
llmBaseUrl | string | — | OpenAI-compatible API URL |
llmModel | string | — | LLM model name |
llmApiKey | string | — | LLM API key |
llmstxt
Generate llms.txt and llms-full.txt from a documentation website.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | — | Required. Website URL |
maxUrls | integer | 20 | Max pages to process |
llmBaseUrl | string | — | LLM API URL for descriptions |
llmModel | string | — | LLM model name |
llmApiKey | string | — | LLM API key |
showFullText | boolean | true | Include full page content |
Next Steps
- MCP Setup Guide — configure Claude Desktop, Claude Code, Cursor, and more