# Refetch LLM Integration Reference

This is the authoritative, self-contained reference for integrating with the Refetch API (refetch.cloud). You do not need any other documentation. If you are an LLM writing code to fetch web pages, search the web, or search news, everything you need is here.

## Base URL

```
https://refetch.cloud
```

## Authentication

All fetch and search endpoints require a Bearer token in the `Authorization` header.

```
Authorization: Bearer ftch_xxxxxxxx
```

API keys are prefixed with `ftch_`. Sign up at [www.refetch.cloud](https://www.refetch.cloud) to get a key. Every new account receives 500 free requests with no credit card required. Additional credits cost $2.50 per 1,000 credits.

## Endpoints

### Fetch endpoints (1 credit each)

| Method | URL pattern | Description |
|--------|-------------|-------------|
| GET | `/md/{url}` | Fetch a webpage and return clean Markdown (recommended for reading and summarization) |
| GET | `/html/{url}` | Fetch a webpage and return raw HTML |
| GET | `/ld+json/{url}` | Extract JSON-LD structured data from a webpage |

The `{url}` is placed directly in the path, including the scheme. For example:

```
GET https://refetch.cloud/md/https://example.com/page?q=hello
GET https://refetch.cloud/html/https://en.wikipedia.org/wiki/Cat
GET https://refetch.cloud/ld+json/https://www.allrecipes.com/recipe/12345
```

### Search endpoints (1 credit each)

| Method | URL pattern | Description |
|--------|-------------|-------------|
| GET | `/s/web?q={query}` | Search the web, returns structured JSON results |
| GET | `/s/news?q={query}` | Search news articles, returns structured JSON results |

### Utility endpoints (free)

| Method | URL pattern | Auth required | Description |
|--------|-------------|---------------|-------------|
| GET | `/usage` | Yes | View your credit balance and usage history |
| GET | `/limits` | No | View current rate limits and pricing |

### MCP endpoint

| Method | URL pattern | Description |
|--------|-------------|-------------|
| POST | `/mcp` | Model Context Protocol server (streamable HTTP) |

MCP client configuration example:

```json
{
  "mcpServers": {
    "refetch": {
      "type": "streamableHttp",
      "url": "https://refetch.cloud/mcp",
      "headers": {
        "Authorization": "Bearer ftch_your_key_here"
      }
    }
  }
}
```

## Request Headers

| Header | Required | Description |
|--------|----------|-------------|
| `Authorization` | Yes (except `/limits`) | `Bearer ftch_xxxxxxxx` |
| `X-Block-Patterns` | No | Comma-separated strings for soft block detection. If any pattern appears in the response body, the request is retried through an alternate route. Example: `X-Block-Patterns: premium required,subscribe to read` |

## Response Headers

| Header | Description |
|--------|-------------|
| `X-Credits-Remaining` | Your credit balance after the request was processed |
| `X-Resolved-URL` | Present when a news redirect URL was automatically resolved to the real article URL |

## Credit System

- Each fetch or search request costs **1 credit**.
- Free endpoints (no credit charged): documentation pages (`/`, `/md`, `/html`, `/ld+json`, `/s` without `?q=`), `/usage`, `/limits`.
- Credits are deducted **before** the upstream request is made. If the upstream request fails, the credit is still consumed. This is by design.
- Penalty hits during a domain rate limit window also cost credits (see the domain penalty section below).

## Error Handling

All errors return JSON in this format:

```json
{
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable description"
  }
}
```

### Complete Error Reference

#### Authentication errors. Do not retry.

| Code | HTTP Status | Billed | Action |
|------|-------------|--------|--------|
| `UNAUTHORIZED` | 401 | No | Add the `Authorization: Bearer ftch_...` header to your request. |
| `INVALID_API_KEY` | 401 | No | The API key is not recognized. Check the key or generate a new one at www.refetch.cloud. |
| `INSUFFICIENT_CREDITS` | 402 | No | Credit balance is zero. Purchase more credits at www.refetch.cloud. |

#### Client errors. Do not retry. Fix your request.

| Code | HTTP Status | Billed | Action |
|------|-------------|--------|--------|
| `INVALID_URL` | 400 | Yes | The URL is malformed, missing its scheme (http/https), or points to a private IP address. Fix the URL. |
| `MISSING_PARAM` | 400 | No | A required parameter is missing (e.g. `?q=` on search endpoints). |
| `CONTENT_TYPE_NOT_ALLOWED` | 400 | Yes | The target page returned a non-text content type (e.g. image, video, PDF). Only text-based content is supported. |
| `RESPONSE_TOO_LARGE` | 400 | Yes | The response body exceeds 10 MB. |
| `RECURSIVE_PROXY` | 400 | Yes | You attempted to fetch refetch.cloud itself. |
| `SEARCH_NOT_SUPPORTED` | 400 | No | You attempted to fetch a search engine results page directly. Use `/s/web` or `/s/news` instead. |
| `NOT_FOUND` | 404 | No | The endpoint does not exist. |
| `NO_LD_JSON` | 404 | Yes | The page contains no JSON-LD structured data. |

#### Rate limit errors.

| Code | HTTP Status | Billed | Action |
|------|-------------|--------|--------|
| `KEY_RATE_LIMITED` | 429 | No | You exceeded the per-key rate limit. Wait briefly, then retry. |
| `DOMAIN_RATE_LIMITED` | 429 | Varies | See the domain penalty section below. Stop all requests to this domain for at least 60 seconds. |
| `RATE_LIMITED` | 429 | Yes | Upstream rate limit. Retry with exponential backoff. |

#### Upstream and server errors. Retry with exponential backoff.

| Code | HTTP Status | Billed | Action |
|------|-------------|--------|--------|
| `UPSTREAM_BLOCKED` | 502 | Yes | The target site blocked the request. Retry after a delay, or try a different URL. |
| `UPSTREAM_ERROR` | 502 | Yes | The upstream request failed. Retry with backoff. |
| `NEWS_RESOLVE_FAILED` | 502 | Yes | A news redirect URL could not be resolved. Retry with backoff. |
| `INTERNAL_ERROR` | 500 | Yes | An internal error occurred. Retry with backoff. |
| `SERVICE_UNAVAILABLE` | 503 | No | The service is temporarily unavailable. Retry with backoff. |
| `CAPACITY_EXCEEDED` | 503 | No | The service is at capacity. Retry shortly. |

## CRITICAL: Domain Rate Limit Penalty

This is the most important rate limiting behavior to understand. Getting this wrong will waste credits.

1. When you hit the per-domain rate limit for the first time, you receive `DOMAIN_RATE_LIMITED` (429). This first hit is **free**.
2. This starts a **60-second penalty window** for that specific domain.
3. Every subsequent request to the **same domain** during the penalty window costs **1 credit AND returns 429**. You pay but get nothing.
4. You **MUST** stop all requests to the penalized domain for at least 60 seconds.
5. You **can** continue fetching other domains normally during this time.

If you are building a crawler or batch processor, you must track per-domain penalty expiry timestamps and skip penalized domains.

## Rate Limits

| Scope | Burst | Sustained |
|-------|-------|-----------|
| Per API key | 100 requests per second | 1,000 requests per minute |
| Per domain | 100 requests per second | 1,000 requests per minute |

Both use fixed-window counters. Per-domain limits also trigger the penalty mechanic described above.

## Recommended Retry Strategy

Use this decision tree based on the HTTP status code:

- **401**: Do not retry. Fix authentication.
- **402**: Do not retry. Purchase credits.
- **400**: Do not retry. Fix the request.
- **404**: Do not retry. The endpoint or data does not exist.
- **429 with `KEY_RATE_LIMITED`**: Wait 1 second, then retry.
- **429 with `DOMAIN_RATE_LIMITED`**: Stop all requests to that domain for 60+ seconds. Continue with other domains.
- **429 with `RATE_LIMITED`**: Retry with exponential backoff (1s, 2s, 4s, 8s). Max 3 retries.
- **500**: Retry with exponential backoff. Max 3 retries.
- **502**: Retry with exponential backoff. Max 2 retries.
- **503**: Retry with exponential backoff (start at 2s). Max 3 retries.

## Search Parameters

### Web search: `/s/web`

| Parameter | Required | Description |
|-----------|----------|-------------|
| `q` | Yes | Search query |
| `country` | No | ISO 3166-1 alpha-2 country code (default: `US`) |
| `lang` | No | ISO 639-1 language code (auto-selected from country if omitted) |
| `when` | No | Time filter: `d` (past day), `w` (past week), `m` (past month), `y` (past year) |

### News search: `/s/news`

| Parameter | Required | Description |
|-----------|----------|-------------|
| `q` | Yes | Search query |
| `country` | No | ISO 3166-1 alpha-2 country code (default: `US`) |
| `lang` | No | ISO 639-1 language code (auto-selected from country if omitted) |
| `when` | No | Lookback window: `1d`, `7d`, `30d`, `180d` |
| `limit` | No | Max results, 1 to 100 (default: `100`) |

### Supported countries

`US`, `GB`, `CA`, `AU`, `AE`, `SE`, `NO`, `DK`, `FI`, `DE`, `AT`, `CH`, `FR`, `ES`, `IT`, `NL`, `BE`, `PT`, `PL`, `CZ`, `IL`, `SA`

## Response Shapes

### Web search response

```json
{
  "query": "machine learning frameworks",
  "params": {
    "country": "US",
    "lang": "en"
  },
  "count": 3,
  "results": [
    {
      "title": "Top Machine Learning Frameworks in 2026",
      "url": "https://example.com/ml-frameworks",
      "snippet": "A comprehensive comparison of the most popular ML frameworks..."
    },
    {
      "title": "PyTorch vs TensorFlow",
      "url": "https://example.com/pytorch-vs-tf",
      "snippet": "An in-depth look at the two dominant deep learning libraries..."
    },
    {
      "title": "Getting Started with JAX",
      "url": "https://example.com/jax-tutorial",
      "snippet": "JAX combines NumPy with automatic differentiation and GPU support..."
    }
  ]
}
```

### News search response

```json
{
  "query": "renewable energy",
  "params": {
    "country": "US",
    "lang": "en",
    "when": "7d"
  },
  "count": 2,
  "articles": [
    {
      "title": "Solar capacity breaks new record",
      "url": "https://example.com/solar-record",
      "source": {
        "name": "Energy News",
        "url": "https://example.com"
      },
      "publishedAt": "2026-03-12T14:30:00Z"
    },
    {
      "title": "Wind energy investment surges in Europe",
      "url": "https://example.com/wind-investment",
      "source": {
        "name": "Reuters",
        "url": "https://reuters.com"
      },
      "publishedAt": "2026-03-11T09:15:00Z"
    }
  ]
}
```

### Usage endpoint response

```json
{
  "key": "ftch_abc1...",
  "credits_remaining": 342,
  "usage": {
    "2026-03-13": {
      "/md": 45,
      "/s/web": 12,
      "/s/news": 3
    },
    "2026-03-12": {
      "/md": 88,
      "/html": 5,
      "/s/web": 20
    }
  }
}
```

## Complete Python Example

```python
import time
import requests
from urllib.parse import urlparse

BASE_URL = "https://refetch.cloud"
API_KEY = "ftch_your_key_here"  # Replace with your API key

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
}

# Track domain penalties: domain -> expiry timestamp
_domain_penalties: dict[str, float] = {}


def _is_domain_penalized(domain: str) -> bool:
    """Check if a domain is currently in a penalty window."""
    expiry = _domain_penalties.get(domain)
    if expiry is None:
        return False
    if time.time() >= expiry:
        del _domain_penalties[domain]
        return False
    return True


def _mark_domain_penalized(domain: str):
    """Mark a domain as penalized for 65 seconds (60s penalty + 5s buffer)."""
    _domain_penalties[domain] = time.time() + 65


def _extract_domain(url: str) -> str:
    return urlparse(url).hostname or ""


def fetch_markdown(url: str, block_patterns: list[str] | None = None, max_retries: int = 3) -> str:
    """Fetch a URL and return its content as clean Markdown.

    Handles retries for transient errors and respects domain penalties.
    Returns the Markdown text on success.
    Raises an exception on permanent failure.
    """
    domain = _extract_domain(url)
    if _is_domain_penalized(domain):
        raise Exception(
            f"Domain {domain} is in a penalty window. "
            f"Wait until {_domain_penalties[domain] - time.time():.0f}s have passed."
        )

    headers = dict(HEADERS)
    if block_patterns:
        headers["X-Block-Patterns"] = ",".join(block_patterns)

    for attempt in range(max_retries):
        resp = requests.get(f"{BASE_URL}/md/{url}", headers=headers)

        if resp.status_code == 200:
            return resp.text

        # Parse error
        try:
            err = resp.json()["error"]
            code = err["code"]
            message = err["message"]
        except (KeyError, ValueError):
            code = "UNKNOWN"
            message = resp.text

        # Domain penalty: stop immediately, do not retry this domain
        if code == "DOMAIN_RATE_LIMITED":
            _mark_domain_penalized(domain)
            raise Exception(f"Domain rate limited: {domain}. Backing off for 60+ seconds.")

        # Per-key rate limit: short pause then retry
        if code == "KEY_RATE_LIMITED":
            time.sleep(1)
            continue

        # Upstream rate limit: exponential backoff
        if code == "RATE_LIMITED":
            time.sleep(2 ** attempt)
            continue

        # Retryable server errors (500, 502, 503)
        if resp.status_code in (500, 502, 503):
            time.sleep(2 ** attempt)
            continue

        # Non-retryable errors (400, 401, 402, 404)
        raise Exception(f"Refetch error {code}: {message}")

    raise Exception(f"Failed after {max_retries} retries")


def search_web(query: str, country: str = "US", lang: str = "", when: str = "") -> dict:
    """Search the web and return structured results.

    Returns the parsed JSON response with query, params, count, and results.
    """
    params = {"q": query, "country": country}
    if lang:
        params["lang"] = lang
    if when:
        params["when"] = when

    resp = requests.get(f"{BASE_URL}/s/web", headers=HEADERS, params=params)

    if resp.status_code == 200:
        return resp.json()

    try:
        err = resp.json()["error"]
        code = err["code"]
        message = err["message"]
    except (KeyError, ValueError):
        code = "UNKNOWN"
        message = resp.text

    if code == "KEY_RATE_LIMITED":
        time.sleep(1)
        return search_web(query, country, lang, when)

    raise Exception(f"Refetch search error {code}: {message}")


def search_news(
    query: str,
    country: str = "US",
    lang: str = "",
    when: str = "",
    limit: int = 100,
) -> dict:
    """Search news articles and return structured results.

    Returns the parsed JSON response with query, params, count, and articles.
    """
    params: dict = {"q": query, "country": country, "limit": str(limit)}
    if lang:
        params["lang"] = lang
    if when:
        params["when"] = when

    resp = requests.get(f"{BASE_URL}/s/news", headers=HEADERS, params=params)

    if resp.status_code == 200:
        return resp.json()

    try:
        err = resp.json()["error"]
        code = err["code"]
        message = err["message"]
    except (KeyError, ValueError):
        code = "UNKNOWN"
        message = resp.text

    if code == "KEY_RATE_LIMITED":
        time.sleep(1)
        return search_news(query, country, lang, when, limit)

    raise Exception(f"Refetch news error {code}: {message}")


# --- Example usage ---

if __name__ == "__main__":
    # Fetch a page as Markdown
    md = fetch_markdown("https://example.com")
    print(md[:500])

    # Search the web
    results = search_web("python async frameworks", country="US")
    for r in results["results"][:3]:
        print(f"  {r['title']}: {r['url']}")

    # Search news
    news = search_news("AI regulation", country="US", when="7d", limit=5)
    for a in news["articles"]:
        print(f"  [{a['source']['name']}] {a['title']}")
```