Extract

Extract structured data from web pages using CSS selectors, LLM-based extraction, or both. Returns JSON objects conforming to your schema.

Endpoint

POST /api/v1/extract

Parameters

Extraction Modes

CSS Mode ("css")

Rule-based extraction using CSS selectors. No external dependencies, no API keys.

Each key in selectors becomes a field in the output. The CSS selector is run against the page HTML, and the matched element's text content is extracted. If a schema is provided, values are coerced to the specified types.

LLM Mode ("llm")

AI-powered extraction using any OpenAI-compatible API. Requires llmBaseUrl and optionally llmApiKey. The page content (as Markdown) is sent to the LLM along with your schema and prompt.

Supports both Chat Completions (/v1/chat/completions) and Responses (/v1/responses) API formats — auto-detected from the URL.

Auto Mode ("auto", default)

Tries CSS extraction first (if selectors provided). If the result is less than 50% complete or fails schema validation, falls back to LLM extraction (if credentials provided). If neither selectors nor LLM credentials are given, returns an error.

Schema Validation and Batch Budgets

Schemas support drafts 4, 6, 7, 2019-09 and 2020-12. The $schema declaration selects the draft; omitting it uses 2020-12. Unsupported dialects, invalid schemas and unresolved references fail before scraping or model calls. Schema references resolve only from the supplied document: no HTTP requests or file reads are made to retrieve schemas. Standard format constraints are checked, and unknown custom formats are rejected.

Every successful CSS or LLM result must pass local schema validation. Type mismatches, missing required fields, invalid enum values and other schema violations become per-URL errors containing JSON pointer paths (for example, /price). In auto mode a schema-invalid CSS candidate can trigger the LLM fallback.

The server runs up to five URLs concurrently by default. Set ESSENCE_EXTRACT_CONCURRENCY to an integer from 1 to 10 to fit browser capacity and model provider limits; an invalid setting returns a configuration error. Each URL's timeout starts when its concurrency slot becomes available and includes URL validation, scrape preflight, fetching, formatting and extraction. Time spent waiting for a batch slot is excluded, so a batch may take several per-URL budgets. A timed-out URL is reported as a failed item; remaining URLs continue.

Examples

CSS Extraction

curl -X POST http://localhost:8080/api/v1/extract \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "mode": "css",
    "selectors": {
      "title": "h1",
      "price": "p.price_color",
      "availability": "p.availability",
      "description": "#product_description ~ p"
    },
    "schema": {
      "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "availability": {"type": "string"},
        "description": {"type": "string"}
      }
    }
  }'

LLM Extraction

curl -X POST http://localhost:8080/api/v1/extract \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/about"],
    "mode": "llm",
    "prompt": "Extract company information from this page",
    "schema": {
      "properties": {
        "companyName": {"type": "string"},
        "founded": {"type": "string"},
        "employees": {"type": "number"},
        "description": {"type": "string"}
      }
    },
    "llmBaseUrl": "https://api.openai.com",
    "llmModel": "gpt-4o-mini",
    "llmApiKey": "sk-..."
  }'

Python

import requests

response = requests.post("http://localhost:8080/api/v1/extract", json={
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "mode": "css",
    "selectors": {
        "title": "h1",
        "price": "p.price_color"
    },
    "schema": {
        "properties": {
            "title": {"type": "string"},
            "price": {"type": "number"}
        }
    }
})

data = response.json()
for item in data["data"]:
    print(f"{item['title']}: {item['price']}")

JavaScript

const response = await fetch("http://localhost:8080/api/v1/extract", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    urls: ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    mode: "css",
    selectors: { title: "h1", price: "p.price_color" },
    schema: {
      properties: {
        title: { type: "string" },
        price: { type: "number" },
      },
    },
  }),
});

const { data } = await response.json();
console.log(data[0].title, data[0].price);

Response

{
  "success": true,
  "allSuccessful": true,
  "data": [
    {
      "title": "A Light in the Attic",
      "price": 51.77,
      "availability": "In stock",
      "description": "It's hard to imagine a world without A Light in the Attic..."
    }
  ],
  "results": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "success": true,
      "data": {
        "title": "A Light in the Attic",
        "price": 51.77,
        "availability": "In stock",
        "description": "It's hard to imagine a world without A Light in the Attic..."
      }
    }
  ]
}

Both data and results preserve input order. The existing data field keeps bare successful values; failures use {"url": "...", "error": "..."} in that position. The additive results field identifies every source URL and has success plus either data or error for each item. Prefer results when processing a batch.

Top-level success: true means the batch request was accepted and processed, even if some or all URLs failed. allSuccessful is true only when every URL succeeded. Individual failures retain the successful results and add a batch warning. Invalid global configuration or schema returns a request-level error.

Schema Type Coercion

When a schema is provided with CSS extraction, values are coerced: