MCP server
WebSkrap ships a Model Context Protocol
server. MCP clients such as Claude Desktop, Claude Code, and Codex can call it to
drive a real browser directly. It runs over stdio and exposes fetch tools for
one-shot scraping plus browser_* tools for persistent interactive sessions.
Both fetch and stealth_fetch run the same stealth path the CLI uses
(headless Chrome on WebSkrap's own CDP client, which never enables the
Runtime domain), so JS-heavy and
anti-bot pages that block naive scrapers still load. They return clean visible
page text by default, with no HTML tags, scripts, or CSS noise, so the model
spends tokens on content instead of markup (typically 5-10x fewer tokens than
raw HTML). Use stealth_fetch for finer fingerprint/WebRTC/UA control. Pass
text_only=false when you need the HTML.
Both fetch tools default to domcontentloaded and resource_policy="auto".
Auto skips images, fonts and media for text; HTML loads all resources.
Explicit all, lite, or documents policies take priority. Use documents
only when removing CSS preserves the required result. Use ready_selector for
deferred content. Keep cookie rejection on unless the page is known not to need it.
Earlier load states can read before deferred content arrives. Routing disables
the HTTP cache and does not intercept service-worker-owned requests.
Install
pip install webskrap
webskrap installRun the server:
webskrap-mcpYou can also run it as a module:
python -m webskrap.mcp_serverTools
| Tool | Purpose |
|---|---|
fetch |
Fetch a URL on the stealth path (waits for networkidle). |
stealth_fetch |
Same stealth path with finer fingerprint/WebRTC/UA controls. |
fetch_session_close |
Close an idle warm fetch session; persistent profile files survive. |
search |
Find URLs for a query on Bing (default) or DuckDuckGo, with the stealth_fetch stealth controls. |
doctor |
Check that Chromium can launch. |
browser_open |
Start (or reuse) a persistent headless browser session. |
browser_goto |
Navigate the session's current page. |
browser_text |
Visible body text, windowed in the browser without an accessibility tree. |
browser_find |
Find accessible elements by text or regex with minimal ancestor context. |
browser_snapshot |
Aria snapshot of the page with eN element refs. |
browser_interact |
Click, fill, type, select, hover, check, or uncheck an element. |
browser_press |
Press a keyboard key on the page. |
browser_wait_for |
Wait for text to appear or disappear, a ref or selector state, or a load state. |
browser_view |
Inline viewport PNG and CSS coordinate metadata, without a file write. |
browser_mouse |
Coordinate click, double-click, move, scroll, or drag. |
browser_insert_text |
Literal Unicode text into the focused control. |
browser_screenshot |
Screenshot the current page to a PNG file. |
browser_eval |
Evaluate JavaScript and return the result. |
browser_close |
Close a session (delete_data removes its profile). |
browser_list |
List sessions and whether each is running. |
browser_tabs |
List, open, select, or close tabs. Other tools act on the current tab. |
Both fetch tools return status, final_url, title, ok, headers, and the
page content in text (capped by max_chars, with text_length and
text_truncated reporting the full size). By default text is clean visible
text; set text_only to false to get raw HTML.
Both also auto-decline cookie banners before reading the page, and report which
strategy fired in cookie_notice_declined. Pass decline_cookies: false to
leave the notice in place.
Tool arguments
fetch accepts:
| Argument | Default | Notes |
|---|---|---|
url |
required | URL to load. |
profile |
desktop-chrome |
Profile label; the browser's native identity is kept unless profile_mode asks for more. |
channel |
chrome |
Browser channel; use chromium on Linux ARM64. |
wait_until |
domcontentloaded |
commit, domcontentloaded, load, or networkidle. |
resource_policy |
auto |
auto, all, lite, or documents. |
timeout_ms |
60000 |
Timeout applied separately to navigation and readiness. |
ready_selector |
null |
CSS selector or snapshot ref that must become visible after consent dismissal. |
max_chars |
8000 |
Maximum returned text characters. |
text_only |
true |
Return clean visible text; set false for raw HTML. |
decline_cookies |
true |
Click a cookie consent notice's reject button after load. |
session |
null |
Opt in to a warm browser with shared cookies/storage. |
Example arguments:
{
"url": "https://example.com",
"profile": "desktop-chrome",
"resource_policy": "lite",
"wait_until": "load",
"max_chars": 5000
}stealth_fetch accepts the same URL/profile/timeout/output-size/session controls,
resource_policy (default auto) and wait_until (default domcontentloaded),
plus stealth options:
{
"url": "https://example.com",
"channel": "chrome",
"headless": false,
"profile_mode": "native",
"reduce_fingerprint_surface": false,
"mask_headless_user_agent": false,
"virtual_display": false,
"gpu": "auto",
"webrtc_ip_handling_policy": null
}Both fetch tools and search check for known challenge pages (DataDome,
Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai), and
return a blocked tool error, including pages served with HTTP 200. The hint
differs for a block (change the exit IP or try virtual_display=True) and a
challenge (try virtual_display=True, headed mode, or a persistent
user_data_dir). See the stealth guide for the
exact markers. browser_goto and browser_open with a url run the same
check but report it in blocked instead of failing, so the page stays open. Ordinary
HTTP error pages still return ok: false. With ready_selector set, a missing
element returns timeout; if a known challenge replaced the page while waiting,
the error is blocked. Detection does not identify every challenge or guarantee
that later interactions will succeed.
When user_data_dir is set, it must be relative to
~/.webskrap/profiles. Set WEBSKRAP_MCP_PROFILE_DIR in the MCP server
environment to move that root. Absolute paths, .. traversal, and symlinks
resolving outside the root are rejected. This confinement applies only to MCP
tool input; Python callers can still choose any SessionConfig.user_data_dir.
Warm fetch sessions
For repeated extraction, pass the same session name on each fetch:
{
"url": "https://example.com",
"session": "crawl",
"resource_policy": "lite",
"decline_cookies": false
}The first call launches the browser; later calls open new pages in its existing
context. Cookies and storage are shared within that name. Keep profile and
config options identical; changes are rejected. timeout_ms, decline_cookies,
output options and wait_until can vary. Concurrent calls can use the same session.
Up to eight fetches run at once; extra calls wait. Up to eight named sessions
stay open; close unused ones before creating more.
Call fetch_session_close(session="crawl") when done. It returns
{"closed": true}, or false for an absent session. Busy sessions are refused.
The server closes all fetch sessions on shutdown. These sessions are separate
from browser_* sessions and do not survive a server restart. Persistent
user_data_dir files survive closing. Without session, each call still uses
a fresh browser and temporary profile; an explicit user_data_dir retains its
on-disk state.
search takes query, engine (bing by default, or ddg), max_results, and the
channel, profile, fingerprint and user_data_dir arguments stealth_fetch
takes. It returns hits (title, url, snippet) with the engine's
click-tracking unwrapped, plus hits_total and hits_truncated. Google is
not offered. A blocked error means the engine served a bot challenge.
Switch engine or exit IP instead of retrying.
Interactive browser tools
The browser_* tools drive the same persistent sessions as
webskrap browser in the
CLI: browser_open launches a detached headless Chromium that keeps running
between tool calls (and between MCP server restarts), and every other tool
reconnects to it over CDP, with the same automation and
virtual-screen launch flags as stealth_fetch. Sessions are named (session, default default)
and store their profile under ~/.webskrap/browser/<name>/
(root overridable with WEBSKRAP_BROWSER_DIR), so cookies and logins persist.
browser_open accepts webrtc_ip_handling_policy (for example
disable_non_proxied_udp) to keep the host's addresses out of WebRTC
candidates. A proxy is deliberately not a tool argument, like the sandbox
switch: page text can steer a model, and a model-chosen proxy could reroute all
of a session's traffic. Open a proxied session from the CLI with
webskrap browser open --proxy URL; the browser tools then reuse it, and
browser_open reports its proxy_server.
A typical flow:
browser_openwith an optionalurl.browser_snapshot, where each element carries a ref like[ref=e15]. Iframe content is listed under itsiframeline, with refs likef12345e15.browser_interactwithaction: "click"andtarget: "e15"(or any CSS selector).fillandtypetake one entry invalues;selecttakes one or more.click,dblclickandtypeuse human mouse and keystroke timing, so they take noticeably longer thanfill; see Humanizer.browser_evalorbrowser_screenshotto read results.browser_closewhen done (delete_data: trueto drop the profile).
Snapshots default to depth=6; pass depth: null for the full tree.
Use browser_text for reading without element refs. It accepts max_chars and
offset, and returns next_text_offset for paging through the current text.
Offsets remain valid only while the text stays unchanged. Pass pattern (a
Python regex of up to 200 characters) to get only the matching non-blank lines,
each with context lines around it (0 to 3) and at most max_matches (1 to
50). The regex runs in Python, never in the page, and a pattern that takes
longer than 2 seconds fails with usage.
browser_goto and browser_open accept decline_cookies: true to click a
consent notice's reject control (waiting up to 10 seconds for it) and report the strategy in
cookie_notice_declined. browser_screenshot accepts target, a snapshot ref
(iframe refs such as f12345e15 included) or CSS selector, to capture only
that element's bounding box; the result reports it as clip.
Snapshots are truncated to max_chars (default 8000) and report
snapshot_truncated; refs describe the current DOM, so take a fresh snapshot
after the page changes. Failed actions return a one-line error.
Vision and coordinate input
browser_view(session="visual") returns an inline image/png image block,
a text block with metadata, and the same metadata in structuredContent:
url, title, width, height, scroll_x, scroll_y, mime_type, and
coordinate_system: "viewport-css-pixels". It writes no file. Use it with an
image-capable model and client; browser_screenshot remains the file/export tool.
Open with browser_open(session="visual", url=...), inspect
browser_view(session="visual"), then call:
{"action": "click", "x": 120, "y": 80, "session": "visual"}with browser_mouse. Choose coordinates from the observed target. For a focused
input, browser_insert_text(text="hello 世界", session="visual") inserts literal
text; browser_press(key="Enter", session="visual") sends a key. Insertion emits
an input event, not individual keydown/keyup events. Use browser_interact with
action="type" when a control requires individual keystrokes.
Mouse actions are click, dblclick, move, scroll, and drag. All require
x and y. button is left (default), middle, or right, for click,
double-click, or drag only. Drag also requires end_x and end_y. Scroll accepts
delta_x and delta_y in CSS pixels, targeting the element under the pointer;
positive delta_y scrolls down. Invalid, non-finite, unused, or out-of-viewport
arguments return a usage error before mouse input is sent.
Use the original PNG dimensions: each image pixel is one CSS pixel, independent
of device scale. (0,0) is the viewport's top-left. Map any resized preview back
to width/height, and do not add scroll offsets. Full-page screenshots do not
share this mouse coordinate mapping.
Prefer refs/selectors for identifiable controls. Coordinate clicks hit whatever is visible at the point. After an action, wait for the expected change, then take a fresh view before choosing more coordinates. Wheel input can finish after the call returns. Run dependent actions sequentially in each session. The image shows the page, including canvas and frames, not browser chrome, OS dialogs, or a live desktop stream. Headless rendering needs no Xvfb.
Screenshot output is confined
browser_screenshot writes only under ./webskrap-output, relative to the
directory the server runs in. Set WEBSKRAP_OUTPUT_DIR to move that root, and
point it somewhere you are willing to have written to, not a source tree or
$HOME.
path is a relative destination inside that root. Nested paths work
(runs/today/page.png, directories created as needed); absolute paths, ..
traversal, and symlinks leaving the root are rejected before the browser is
touched. The model driving these tools reads untrusted pages, so its file
destinations are treated as untrusted input.
Chromium sandbox
MCP sessions keep Chromium's OS sandbox. Hosts that cannot sandbox must set
WEBSKRAP_CHROMIUM_SANDBOX=0 in the server's environment; it is deliberately
not a tool argument, so a page cannot talk the model into weakening renderer
isolation. Session profiles under ~/.webskrap/browser/<name>/ are created
0700 on POSIX.
Limitations, the same as the CLI: bundled Chromium only, and no tools for
network mocking, tracing, or video. MCP sessions are
always headless. The CLI additionally offers back, forward, reload, and
headed mode.
Register with a client
Claude Code
claude mcp add webskrap -- webskrap-mcpClaude Desktop
{
"mcpServers": {
"webskrap": {
"command": "webskrap-mcp"
}
}
}Codex
Add a server entry to ~/.codex/config.toml:
[mcp_servers.webskrap]
command = "webskrap-mcp"
args = []Stealth
stealth_fetch accepts the same controls as the
Stealth guide, including
channel, headless, user_data_dir, profile_mode,
reduce_fingerprint_surface, mask_headless_user_agent, virtual_display,
gpu, and webrtc_ip_handling_policy. On Linux with Xvfb, prefer virtual_display
over mask_headless_user_agent: it runs Chrome headed on a private display
instead of rewriting the user agent, which empties the high-entropy client hints.
For headless best-effort stealth from MCP, use real Chrome and opt in only to the native browser controls you need:
{
"url": "https://example.com",
"channel": "chrome",
"headless": true,
"user_data_dir": "headless-profile",
"virtual_display": true,
"profile_mode": "locale",
"webrtc_ip_handling_policy": "disable_non_proxied_udp"
}Targeted snapshots
Use browser_find(text="Save changes") to find known controls. Supply exactly
one of text (case-insensitive substring) or regex (Python regex, with inline
flags such as (?i)). Results include matching lines, ancestor paths and refs
for browser_interact. max_matches accepts 1..50, default 20. Match counts
precede character truncation. Caller regex runs in a worker with a two-second
limit per frame.
Use browser_snapshot(target="e15", depth=2) for one subtree. Unique main-frame
CSS selectors also work. Set boxes=true only when viewport geometry is needed.
The existing character limit and continuation fields remain unchanged for both
tools. Repeat the same query with next_snapshot_offset while the DOM is stable.
Snapshots omit password values, hidden and inert content, and background content behind a native modal dialog. Visible page text and other input values can still contain sensitive data. Open shadow roots and available iframe content use the existing native CDP driver. Closed roots and nested out-of-process frames keep its existing limits. Interactions do not append snapshots. No diff cache is kept.