Skip to content
On this page

CLI

WebSkrap installs the webskrap command.

Install browsers

webskrap install
webskrap install --format json

webskrap install downloads Chrome for Testing into the user cache folder. --force downloads it again. Use JSON output in scripts.

Check environment

webskrap doctor
webskrap doctor --format json

The CLI fetch command runs headless Chrome, so doctor checks that headless Chrome can launch. It also reports the host timezone and warns when it is UTC, which pages read as a server.

List profiles

webskrap profiles
webskrap profiles --format json

Fetch a page

webskrap fetch https://example.com

The default human output prints status, final URL, title, and artifact paths.

For LLMs and shell automation, use JSON:

webskrap fetch https://example.com --format json --max-chars 12000

JSON output includes url, final_url, status, ok, title, headers, text, text_length, text_truncated, and elapsed_ms.

Wait for deferred content with a visible CSS selector or snapshot ref:

webskrap fetch https://example.com --ready-selector '#content' --format json

--timeout-ms applies separately to navigation and this readiness wait, which starts after consent dismissal. A missing selector exits 3 (timeout). Known challenge and block pages (DataDome, Cloudflare, Akamai, AWS WAF) exit 11 (blocked) without retrying, even when served with HTTP 200. Ordinary HTTP errors still return ok: false.

Search the web

webskrap search "example domain"
webskrap search "example domain" --engine bing --max-results 5 --format json

search loads Bing's results page (--engine bing, the default) or DuckDuckGo's HTML page (--engine ddg) in the same stealth browser and prints each hit's title, URL and snippet. The engine's click-tracking redirect is unwrapped, so the URL is the destination. Google is not offered. JSON output carries query, engine, hits, hits_total and hits_truncated.

A blocked failure (exit 11) means the engine served a bot challenge instead of results. Switch engine, reuse --user-data-dir, or change the exit IP. Nothing is retried and no CAPTCHA is solved.

Text and stdout

Print raw fetched content to stdout:

webskrap fetch https://example.com --stdout

Return readable body text instead of HTML:

webskrap fetch https://example.com --stdout --text-only
webskrap fetch https://example.com --format json --text-only

Suppress the human summary when writing artifacts:

webskrap fetch https://example.com --quiet --output example.html

Screenshot and output

webskrap fetch https://example.com --screenshot example.png
webskrap fetch https://example.com --output example.html
webskrap fetch https://example.com -o example.html --screenshot example.png

Parent directories are created automatically for --output and --screenshot.

Wait and timeout

webskrap fetch https://example.com --wait-until load --timeout-ms 60000

Supported wait states are commit, domcontentloaded, load, and networkidle. Prefer domcontentloaded for fast HTML collection, load for regular page assets, and networkidle only when a page actually needs a quiet network.

Resource policy

webskrap fetch https://example.com --resource-policy lite
webskrap fetch https://example.com --resource-policy documents -o page.html

The fetch default is auto. With --text-only, it skips images, fonts and media unless a screenshot is requested. HTML and screenshots load all resources. Explicit policies take priority. lite blocks images, fonts and media. documents also blocks stylesheets; use it only when removing CSS preserves the required result.

webskrap fetch clicks the reject control of a cookie consent notice before reading the page, so banners do not bury the text and no optional cookies are accepted. The human output prints Cookie notice: declined (...) when it fired, and the JSON output carries cookie_notice_declined.

webskrap fetch https://example.com --no-decline-cookies
webskrap fetch https://example.com --decline-cookies-timeout-ms 4000

The timeout is how long to wait for a late-injected notice, and the per-fetch cost on pages that have none. Use 0 for a single immediate check.

Browser channels

--channel defaults to chrome, which does not exist on every platform (Linux ARM64 has no Chrome build). When the channel cannot launch, fetch and search print a notice on stderr and try installed Edge, then Chrome for Testing (the full browser in new headless mode), so piped stdout stays clean. Sandbox failures stop without trying another channel. If nothing launches you get a one-line error and Run: webskrap install, not a traceback.

webskrap doctor follows the same Chrome, Edge, Chromium order. JSON output names the working channel and browser_identity, the host cpu_architecture, and font_count, the number of distinct Fontconfig families. font_count is null where fc-list is unavailable. Human output shows these values too.

Headless stealth controls

webskrap fetch always runs headless. Use real Chrome and a stable profile directory when browser continuity matters:

webskrap fetch https://example.com \
  --channel chrome \
  --user-data-dir .webskrap/headless-profile \
  --virtual-display

--virtual-display runs Chrome headed on a private Xvfb display (Linux, Xvfb installed), so no HeadlessChrome token or empty client hints reach the site. --mask-headless-user-agent rewrites the token under headless mode instead, but Chromium then empties the high-entropy client hints. --gpu mesa renders WebGL with Mesa's lavapipe instead of SwiftShader (Linux, mesa-vulkan-drivers); see the Stealth guide's WebGL renderer section.

For fingerprint-statistics or WebRTC leak-test pages, apply profile locale/timezone/media metadata and block non-proxied WebRTC UDP candidates without viewport, user-agent, or JavaScript patches:

webskrap fetch https://amiunique.org/fr/fingerprint \
  --channel chrome \
  --virtual-display \
  --profile-mode locale \
  --reduce-fingerprint-surface \
  --webrtc-ip-handling-policy disable_non_proxied_udp

Pass additional browser flags with repeated --launch-arg=... options. Use the equals form when the browser flag itself starts with --.

webskrap fetch https://example.com \
  --launch-arg=--no-first-run \
  --launch-arg=--no-default-browser-check

Interactive browser sessions

webskrap browser drives a persistent browser with commands modeled on the official Playwright CLI, implemented natively on WebSkrap's own CDP client. open launches a detached browser that keeps running between commands; every other command reconnects to it over CDP, acts on the current page, and exits. It uses installed Chrome, then Edge, then Chrome for Testing. Installed Chrome and Edge include the platform's sandbox support, such as the AppArmor profile Ubuntu 23.10+ needs.

The session uses the same stealth path as webskrap fetch: commands never enable the CDP Runtime domain, so no Runtime.enable leak reaches the page or its workers, and the browser launches with --disable-blink-features=AutomationControlled (navigator.webdriver stays false) plus the 1920x1080 virtual screen used by headless fetches. Headless sessions still report HeadlessChrome in the user agent; pass --headed when a site checks it. browser eval runs in the page's own JavaScript world, so page globals are visible.

webskrap browser open https://example.com
webskrap browser snapshot
webskrap browser click e15
webskrap browser fill "input[name=q]" "chromium"
webskrap browser press Enter
webskrap browser screenshot page.png --full-page
webskrap browser screenshot button.png --target e15   # one element
webskrap browser close

snapshot prints an aria snapshot of the page where each element carries a ref like [ref=e15]. Interaction commands (click, dblclick, hover, fill, type, select, check, uncheck) accept either a ref or a CSS selector. Refs describe the current page, so take a fresh snapshot after the page changes. Iframe content is listed under its iframe line. Refs in a frame look like f12345e15. CSS selectors match in the main frame only.

click and dblclick scroll with the mouse wheel, move the cursor along a curved path and hold the button for a human 60-140 ms; type clicks into the field and types with human keystroke timing. Both refuse a target covered by another element. They are slower than mechanical input on purpose (see Humanizer); fill still sets a value at once.

Navigation uses goto, back, forward, and reload; press sends a key to the page, and eval runs JavaScript and prints the JSON result. All commands support --format json for scripting and agents, and fail with a one-line error instead of a traceback. The exit status tells the kind of failure; JSON output carries the same name in code:

Exit Code
1 internal
2 usage
3 timeout
4 navigation
5 browser_launch
6 sandbox
7 no_session
8 session_unreachable
9 stale_ref
10 path_rejected
11 blocked

open URL and goto take --decline-cookies to click a consent notice's reject control. They also run the fetch tools' challenge and block check, but report a match instead of failing: the human output prints a blocked: line, and JSON output carries it in blocked. The page stays open.

Sessions are named with -s/--session (default default) and listed with webskrap browser list. Each session stores its browser profile under ~/.webskrap/browser/<name>/ (override the root with WEBSKRAP_BROWSER_DIR), so cookies and logins persist across open/close. Those directories are created 0700 on POSIX, because they hold cookies and logged-in state; they are not encrypted, so use webskrap browser close --delete-data to remove a session's profile, or close --all to stop every session. A root you set up yourself via WEBSKRAP_BROWSER_DIR keeps whatever permissions you gave it.

Open and close operations for the same session are serialized across processes. Concurrent commands cannot launch two browsers against one profile or leave a stale state file behind.

Vision and coordinate input

webskrap browser open https://example.com -s visual
webskrap browser view viewport.png -s visual --format json
# Inspect the PNG with your image-viewing tool before choosing coordinates.
webskrap browser mouse click 120 80 -s visual
webskrap browser insert-text "hello 世界" -s visual
webskrap browser press Enter -s visual
webskrap browser mouse scroll 400 300 --delta-y 500 -s visual
webskrap browser view viewport.png -s visual --format json
# Inspect the new view before choosing the drag points.
webskrap browser mouse drag 100 150 --end-x 250 --end-y 180 -s visual
webskrap browser view viewport.png -s visual --format json
webskrap browser close -s visual

view writes a viewport PNG and returns path (absolute), url, title, width, height, scroll_x, scroll_y, mime_type, and coordinate_system (viewport-css-pixels). Omitting the path generates a unique PNG in the current directory. JSON contains metadata, not base64 image data: the agent must open the file with an image-viewing tool.

One image pixel equals one CSS pixel, even on high-DPI pages. (0, 0) is the viewport's top-left. Map resized previews back to the original dimensions; do not add document scroll offsets. screenshot --full-page remains available for export, but its coordinates cannot be used directly with mouse.

mouse accepts click, dblclick, move, scroll, or drag, with required x y. --button is left (default), middle, or right for click/double-click/ drag. Drag requires both --end-x and --end-y. Scroll uses --delta-x and --delta-y over the element under x y; positive vertical deltas scroll down, including in nested scroll areas. Non-finite, out-of-viewport, and unused arguments fail with usage. Coordinate input uses direct browser events; selector-based click and type retain their human timing and covered-target checks.

insert-text inserts literal Unicode into the focused control with an input event. It does not interpret Enter as a key or emit per-character keydown/ keyup events. Focus with a click or Tab; use press for shortcuts and keys, or selector-based type for controls that require individual keystrokes.

Take a fresh view after navigation, scrolling, or layout changes. Wheel input may return before scrolling finishes; wait for the expected page condition. Prefer refs/selectors for known controls, and run dependent session actions sequentially. This shows the page viewport, not browser chrome or OS dialogs.

Chromium sandbox

open keeps Chromium's OS sandbox, which is what contains a renderer compromised by a hostile page. Some environments cannot start it, such as containers without unprivileged user namespaces or images that disable them. There the browser exits during startup with a message saying so.

webskrap browser open --no-sandbox            # this session only
WEBSKRAP_CHROMIUM_SANDBOX=0 webskrap browser open   # every session on this host

--no-sandbox wins over the environment variable, and a session started without the sandbox prints a warning. webskrap browser list shows a Sandbox column for running sessions. Prefer fixing the host: sandboxing is the difference between a compromised renderer and a compromised machine. WebSkrap never drops the sandbox on its own, and never retries a failed launch without it.

To route a session through a proxy, pass it when the browser starts:

webskrap browser open --proxy socks5://proxy.example:1080
webskrap browser open --proxy http://proxy.example:8080 \
  --webrtc-ip-handling-policy default_public_interface_only

With --proxy, WebRTC defaults to disable_non_proxied_udp, so pages see neither the host's LAN address nor its direct public address in ICE candidates; --webrtc-ip-handling-policy overrides it, with or without a proxy. Both settings are fixed at launch.

For an http or https proxy that asks for a password, put the credentials in the environment, not in the URL:

export WEBSKRAP_PROXY_USERNAME=user
read -rs WEBSKRAP_PROXY_PASSWORD && export WEBSKRAP_PROXY_PASSWORD
webskrap browser open --proxy http://proxy.example:8080

WebSkrap starts a helper process that keeps the credentials in memory. It answers the proxy for every page, frame and worker, and stops when the browser closes. The credentials are not written to disk, not put on a command line, and not passed on to the browser's environment. SOCKS proxies cannot use credentials, because Chromium does not answer their challenge. If the helper stops, later commands fail with session_unreachable: close the session and open it again.

Reopening a running session with a different proxy or policy fails with a usage error instead of reusing a browser that bypasses the proxy; reopening it without --proxy reuses it as it is. webskrap browser list shows each running session's proxy, with (auth) when the helper answers it.

Commands act on the session's current tab. Manage tabs with these commands:

webskrap browser tab-list              # * marks the current tab
webskrap browser tab-new https://example.com
webskrap browser tab-select 0          # index from tab-list, in opening order
webskrap browser tab-close             # the current tab, or pass an index

A popup or a new tab becomes current. When it closes, the tab selected before it is current again. When the current tab closes, the newest tab left is current.

Deliberate limitations versus the official Playwright CLI: Chromium-based browsers only, and no commands for network mocking, tracing, or video. back and forward always reload the page, because the back/forward cache is disabled to keep history navigation deterministic.

The MCP server exposes the same sessions to agents as browser_* tools.