Skip to content
On this page

MCP server

WebSkrap ships a Model Context Protocol server. MCP clients such as Claude Desktop, Claude Code, and Codex can call it to drive a real browser directly. It runs over stdio and exposes fetch tools for one-shot scraping plus browser_* tools for persistent interactive sessions.

Both fetch and stealth_fetch run the same stealth path the CLI uses (headless Chrome on WebSkrap's own CDP client, which never enables the Runtime domain), so JS-heavy and anti-bot pages that block naive scrapers still load. They return clean visible page text by default, with no HTML tags, scripts, or CSS noise, so the model spends tokens on content instead of markup (typically 5-10x fewer tokens than raw HTML). Use stealth_fetch for finer fingerprint/WebRTC/UA control. Pass text_only=false when you need the HTML.

Both fetch tools default to domcontentloaded and resource_policy="auto". Auto skips images, fonts and media for text; HTML loads all resources. Explicit all, lite, or documents policies take priority. Use documents only when removing CSS preserves the required result. Use ready_selector for deferred content. Keep cookie rejection on unless the page is known not to need it. Earlier load states can read before deferred content arrives. Routing disables the HTTP cache and does not intercept service-worker-owned requests.

Install

pip install webskrap
webskrap install

Run the server:

webskrap-mcp

You can also run it as a module:

python -m webskrap.mcp_server

Tools

Tool Purpose
fetch Fetch a URL on the stealth path (waits for networkidle).
stealth_fetch Same stealth path with finer fingerprint/WebRTC/UA controls.
fetch_session_close Close an idle warm fetch session; persistent profile files survive.
search Find URLs for a query on Bing (default) or DuckDuckGo, with the stealth_fetch stealth controls.
doctor Check that Chromium can launch.
browser_open Start (or reuse) a persistent headless browser session.
browser_goto Navigate the session's current page.
browser_text Visible body text, windowed in the browser without an accessibility tree.
browser_find Find accessible elements by text or regex with minimal ancestor context.
browser_snapshot Aria snapshot of the page with eN element refs.
browser_interact Click, fill, type, select, hover, check, or uncheck an element.
browser_press Press a keyboard key on the page.
browser_wait_for Wait for text to appear or disappear, a ref or selector state, or a load state.
browser_view Inline viewport PNG and CSS coordinate metadata, without a file write.
browser_mouse Coordinate click, double-click, move, scroll, or drag.
browser_insert_text Literal Unicode text into the focused control.
browser_screenshot Screenshot the current page to a PNG file.
browser_eval Evaluate JavaScript and return the result.
browser_close Close a session (delete_data removes its profile).
browser_list List sessions and whether each is running.
browser_tabs List, open, select, or close tabs. Other tools act on the current tab.

Both fetch tools return status, final_url, title, ok, headers, and the page content in text (capped by max_chars, with text_length and text_truncated reporting the full size). By default text is clean visible text; set text_only to false to get raw HTML.

Both also auto-decline cookie banners before reading the page, and report which strategy fired in cookie_notice_declined. Pass decline_cookies: false to leave the notice in place.

Tool arguments

fetch accepts:

Argument Default Notes
url required URL to load.
profile desktop-chrome Profile label; the browser's native identity is kept unless profile_mode asks for more.
channel chrome Browser channel; use chromium on Linux ARM64.
wait_until domcontentloaded commit, domcontentloaded, load, or networkidle.
resource_policy auto auto, all, lite, or documents.
timeout_ms 60000 Timeout applied separately to navigation and readiness.
ready_selector null CSS selector or snapshot ref that must become visible after consent dismissal.
max_chars 8000 Maximum returned text characters.
text_only true Return clean visible text; set false for raw HTML.
decline_cookies true Click a cookie consent notice's reject button after load.
session null Opt in to a warm browser with shared cookies/storage.

Example arguments:

{
  "url": "https://example.com",
  "profile": "desktop-chrome",
  "resource_policy": "lite",
  "wait_until": "load",
  "max_chars": 5000
}

stealth_fetch accepts the same URL/profile/timeout/output-size/session controls, resource_policy (default auto) and wait_until (default domcontentloaded), plus stealth options:

{
  "url": "https://example.com",
  "channel": "chrome",
  "headless": false,
  "profile_mode": "native",
  "reduce_fingerprint_surface": false,
  "mask_headless_user_agent": false,
  "virtual_display": false,
  "gpu": "auto",
  "webrtc_ip_handling_policy": null
}

Both fetch tools and search check for known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai), and return a blocked tool error, including pages served with HTTP 200. The hint differs for a block (change the exit IP or try virtual_display=True) and a challenge (try virtual_display=True, headed mode, or a persistent user_data_dir). See the stealth guide for the exact markers. browser_goto and browser_open with a url run the same check but report it in blocked instead of failing, so the page stays open. Ordinary HTTP error pages still return ok: false. With ready_selector set, a missing element returns timeout; if a known challenge replaced the page while waiting, the error is blocked. Detection does not identify every challenge or guarantee that later interactions will succeed.

When user_data_dir is set, it must be relative to ~/.webskrap/profiles. Set WEBSKRAP_MCP_PROFILE_DIR in the MCP server environment to move that root. Absolute paths, .. traversal, and symlinks resolving outside the root are rejected. This confinement applies only to MCP tool input; Python callers can still choose any SessionConfig.user_data_dir.

Warm fetch sessions

For repeated extraction, pass the same session name on each fetch:

{
  "url": "https://example.com",
  "session": "crawl",
  "resource_policy": "lite",
  "decline_cookies": false
}

The first call launches the browser; later calls open new pages in its existing context. Cookies and storage are shared within that name. Keep profile and config options identical; changes are rejected. timeout_ms, decline_cookies, output options and wait_until can vary. Concurrent calls can use the same session. Up to eight fetches run at once; extra calls wait. Up to eight named sessions stay open; close unused ones before creating more.

Call fetch_session_close(session="crawl") when done. It returns {"closed": true}, or false for an absent session. Busy sessions are refused. The server closes all fetch sessions on shutdown. These sessions are separate from browser_* sessions and do not survive a server restart. Persistent user_data_dir files survive closing. Without session, each call still uses a fresh browser and temporary profile; an explicit user_data_dir retains its on-disk state.

search takes query, engine (bing by default, or ddg), max_results, and the channel, profile, fingerprint and user_data_dir arguments stealth_fetch takes. It returns hits (title, url, snippet) with the engine's click-tracking unwrapped, plus hits_total and hits_truncated. Google is not offered. A blocked error means the engine served a bot challenge. Switch engine or exit IP instead of retrying.

Interactive browser tools

The browser_* tools drive the same persistent sessions as webskrap browser in the CLI: browser_open launches a detached headless Chromium that keeps running between tool calls (and between MCP server restarts), and every other tool reconnects to it over CDP, with the same automation and virtual-screen launch flags as stealth_fetch. Sessions are named (session, default default) and store their profile under ~/.webskrap/browser/<name>/ (root overridable with WEBSKRAP_BROWSER_DIR), so cookies and logins persist.

browser_open accepts webrtc_ip_handling_policy (for example disable_non_proxied_udp) to keep the host's addresses out of WebRTC candidates. A proxy is deliberately not a tool argument, like the sandbox switch: page text can steer a model, and a model-chosen proxy could reroute all of a session's traffic. Open a proxied session from the CLI with webskrap browser open --proxy URL; the browser tools then reuse it, and browser_open reports its proxy_server.

A typical flow:

  1. browser_open with an optional url.
  2. browser_snapshot, where each element carries a ref like [ref=e15]. Iframe content is listed under its iframe line, with refs like f12345e15.
  3. browser_interact with action: "click" and target: "e15" (or any CSS selector). fill and type take one entry in values; select takes one or more. click, dblclick and type use human mouse and keystroke timing, so they take noticeably longer than fill; see Humanizer.
  4. browser_eval or browser_screenshot to read results.
  5. browser_close when done (delete_data: true to drop the profile).

Snapshots default to depth=6; pass depth: null for the full tree. Use browser_text for reading without element refs. It accepts max_chars and offset, and returns next_text_offset for paging through the current text. Offsets remain valid only while the text stays unchanged. Pass pattern (a Python regex of up to 200 characters) to get only the matching non-blank lines, each with context lines around it (0 to 3) and at most max_matches (1 to 50). The regex runs in Python, never in the page, and a pattern that takes longer than 2 seconds fails with usage.

browser_goto and browser_open accept decline_cookies: true to click a consent notice's reject control (waiting up to 10 seconds for it) and report the strategy in cookie_notice_declined. browser_screenshot accepts target, a snapshot ref (iframe refs such as f12345e15 included) or CSS selector, to capture only that element's bounding box; the result reports it as clip.

Snapshots are truncated to max_chars (default 8000) and report snapshot_truncated; refs describe the current DOM, so take a fresh snapshot after the page changes. Failed actions return a one-line error.

Vision and coordinate input

browser_view(session="visual") returns an inline image/png image block, a text block with metadata, and the same metadata in structuredContent: url, title, width, height, scroll_x, scroll_y, mime_type, and coordinate_system: "viewport-css-pixels". It writes no file. Use it with an image-capable model and client; browser_screenshot remains the file/export tool.

Open with browser_open(session="visual", url=...), inspect browser_view(session="visual"), then call:

{"action": "click", "x": 120, "y": 80, "session": "visual"}

with browser_mouse. Choose coordinates from the observed target. For a focused input, browser_insert_text(text="hello 世界", session="visual") inserts literal text; browser_press(key="Enter", session="visual") sends a key. Insertion emits an input event, not individual keydown/keyup events. Use browser_interact with action="type" when a control requires individual keystrokes.

Mouse actions are click, dblclick, move, scroll, and drag. All require x and y. button is left (default), middle, or right, for click, double-click, or drag only. Drag also requires end_x and end_y. Scroll accepts delta_x and delta_y in CSS pixels, targeting the element under the pointer; positive delta_y scrolls down. Invalid, non-finite, unused, or out-of-viewport arguments return a usage error before mouse input is sent.

Use the original PNG dimensions: each image pixel is one CSS pixel, independent of device scale. (0,0) is the viewport's top-left. Map any resized preview back to width/height, and do not add scroll offsets. Full-page screenshots do not share this mouse coordinate mapping.

Prefer refs/selectors for identifiable controls. Coordinate clicks hit whatever is visible at the point. After an action, wait for the expected change, then take a fresh view before choosing more coordinates. Wheel input can finish after the call returns. Run dependent actions sequentially in each session. The image shows the page, including canvas and frames, not browser chrome, OS dialogs, or a live desktop stream. Headless rendering needs no Xvfb.

Screenshot output is confined

browser_screenshot writes only under ./webskrap-output, relative to the directory the server runs in. Set WEBSKRAP_OUTPUT_DIR to move that root, and point it somewhere you are willing to have written to, not a source tree or $HOME.

path is a relative destination inside that root. Nested paths work (runs/today/page.png, directories created as needed); absolute paths, .. traversal, and symlinks leaving the root are rejected before the browser is touched. The model driving these tools reads untrusted pages, so its file destinations are treated as untrusted input.

Chromium sandbox

MCP sessions keep Chromium's OS sandbox. Hosts that cannot sandbox must set WEBSKRAP_CHROMIUM_SANDBOX=0 in the server's environment; it is deliberately not a tool argument, so a page cannot talk the model into weakening renderer isolation. Session profiles under ~/.webskrap/browser/<name>/ are created 0700 on POSIX.

Limitations, the same as the CLI: bundled Chromium only, and no tools for network mocking, tracing, or video. MCP sessions are always headless. The CLI additionally offers back, forward, reload, and headed mode.

Register with a client

Claude Code

claude mcp add webskrap -- webskrap-mcp

Claude Desktop

{
  "mcpServers": {
    "webskrap": {
      "command": "webskrap-mcp"
    }
  }
}

Codex

Add a server entry to ~/.codex/config.toml:

[mcp_servers.webskrap]
command = "webskrap-mcp"
args = []

Stealth

stealth_fetch accepts the same controls as the Stealth guide, including channel, headless, user_data_dir, profile_mode, reduce_fingerprint_surface, mask_headless_user_agent, virtual_display, gpu, and webrtc_ip_handling_policy. On Linux with Xvfb, prefer virtual_display over mask_headless_user_agent: it runs Chrome headed on a private display instead of rewriting the user agent, which empties the high-entropy client hints.

For headless best-effort stealth from MCP, use real Chrome and opt in only to the native browser controls you need:

{
  "url": "https://example.com",
  "channel": "chrome",
  "headless": true,
  "user_data_dir": "headless-profile",
  "virtual_display": true,
  "profile_mode": "locale",
  "webrtc_ip_handling_policy": "disable_non_proxied_udp"
}

Targeted snapshots

Use browser_find(text="Save changes") to find known controls. Supply exactly one of text (case-insensitive substring) or regex (Python regex, with inline flags such as (?i)). Results include matching lines, ancestor paths and refs for browser_interact. max_matches accepts 1..50, default 20. Match counts precede character truncation. Caller regex runs in a worker with a two-second limit per frame.

Use browser_snapshot(target="e15", depth=2) for one subtree. Unique main-frame CSS selectors also work. Set boxes=true only when viewport geometry is needed. The existing character limit and continuation fields remain unchanged for both tools. Repeat the same query with next_snapshot_offset while the DOM is stable.

Snapshots omit password values, hidden and inert content, and background content behind a native modal dialog. Visible page text and other input values can still contain sensitive data. Open shadow roots and available iframe content use the existing native CDP driver. Closed roots and nested out-of-process frames keep its existing limits. Interactions do not append snapshots. No diff cache is kept.