Stealth
WebSkrap drives Chromium through its own Chrome DevTools Protocol (CDP) client.
It never enables the Runtime or Console domains, so the Runtime.enable
leak that detectors look for never happens, and its page scripts run in
isolated worlds. WebSkrap does not inject JavaScript fingerprint patches; it
relies on real browser behavior, persistent profiles, and coherent profile
settings instead.
Establish a baseline
Check webskrap doctor --format json and the MCP doctor result before comparing
configurations. Compare package versions and executable paths: a connected MCP
process can run a different installation from the checkout being edited. Restart
that process using the intended installation before testing a code change.
Keep the browser version, persistent profile, and network stable during a
comparison. Start with all resources enabled and native surfaces:
resource_policy="all", mask_headless_user_agent=False, and
reduce_fingerprint_surface=False. On Linux with Xvfb, compare that headless
baseline with virtual_display=True. Keep languages and timezone consistent
with the actual browsing environment. These controls already exist; they do not
guarantee access to a particular site.
If Chromium reports a sandbox error, check user namespaces and any AppArmor policy for its actual executable path before testing stealth. A policy for a system Chrome binary may not cover downloaded Chromium. Restore sandbox support instead of changing fingerprint options. WebSkrap does not disable the sandbox automatically.
Known challenge and block pages raise blocked, including HTTP 200 pages:
- Challenges: DataDome, Cloudflare "Just a moment", Akamai's sec-cpt page, and
AWS WAF's challenge or CAPTCHA action (
x-amzn-waf-action). - Blocks: Cloudflare's own error pages for codes 1005-1010, 1012, 1015 and
1020, a site-styled Cloudflare 403 page that shows the response's Ray ID,
and an Akamai edge page (
server: AkamaiGHost, HTTP 400 or 403) with an Akamai reference number.
The Akamai sec-cpt page and the AWS WAF challenge action reload the page from
their own script. WebSkrap waits up to 15 seconds for that reload and then
reads the real page; it reports blocked only if the interstitial stays. A
plain 403, 404 or 202 without these provider markers is not a block: it
returns ok: false or ok: true as usual. The localized title of a block
page is never a signal.
The hint depends on the kind. For a block, change the exit IP or try
virtual_display=True. For a challenge, try virtual_display=True, headed
mode, or a persistent user_data_dir. On Linux with Xvfb installed, the hint
names virtual_display=True first. Errors carry no page text, IP addresses,
Ray IDs or reference IDs. Detection cannot identify every challenge or
guarantee that later interactions will succeed. Stop automatic retries on
blocked.
Some sites challenge a datacenter exit IP whatever the browser does. For
example, a deal forum behind Cloudflare still challenged virtual_display=True
from a cloud VM. There, only a residential or mobile exit IP helps. WebSkrap
does not solve CAPTCHAs.
Pass ready_selector to require a visible content element before extraction; a
successful HTTP status alone does not establish that the intended page loaded.
Browser channel
Real Chrome is the strongest channel. webskrap install downloads Chrome for
Testing for hosts without Chrome:
pip install webskrap
webskrap installfrom webskrap import SessionConfig
config = SessionConfig(
channel="chrome",
headless=False,
)If user_data_dir is omitted, WebSkrap creates a temporary profile per
session and removes it when the session closes. Pages use the browser's real
window size; no viewport is emulated by default.
Profile mode
By default (profile_mode="native"), sessions use the host browser's native
locale, timezone, headers, and media preferences. This is the strict anti-bot mode because fewer context
overrides means fewer behavioral mismatches.
Fingerprint-statistics pages such as AmiUnique also score how common each browser-visible attribute is. For those cases, you can opt in to applying the selected profile's context metadata while still avoiding viewport, user-agent, and JavaScript fingerprint patches. This can align language, timezone, and media metadata, but it does not make high-entropy fingerprints non-unique by itself:
from webskrap import SessionConfig
config = SessionConfig(
channel="chrome",
headless=True,
virtual_display=True,
profile_mode="locale",
reduce_fingerprint_surface=True,
webrtc_ip_handling_policy="disable_non_proxied_udp",
)profile_mode="locale" applies locale, timezone_id, navigator_languages, color_scheme,
reduced_motion, and any caller-provided extra_http_headers. On Linux, the
locale and timezone come from the browser's environment, so nothing is
overridden over CDP. Screen and window metrics still come from the browser or
from headless_screen. profile_mode="full" also emulates the viewport,
screen, device scale, touch and user agent, as WebSkrap 2.x did; each override
is a surface that a page can cross-check.
A profile user agent comes with client hints that agree with it:
navigator.userAgentData and the Sec-CH-UA headers carry its platform,
platform version, model, mobile flag and Chrome version. The brand list
follows Chromium's rules for that version. It names only the browser that
runs: Google Chrome for the chrome channel, Microsoft Edge for msedge,
and no brand for Chromium. The full version is the browser's own when the
user agent has the same Chrome major version. Otherwise it is
<major>.0.0.0, so keep the two the same. A user agent without a Chrome/
token gets no client hints.
The overrides also reach cross-site iframes and workers. Each one starts paused, gets the profile, and then runs. Two limits remain:
- Chromium applies the screen and device-scale override to top-level pages only. A cross-site iframe reports the browser's real screen size and pixel ratio.
- A shared worker reports the browser's own
navigator.userAgent. Usemask_headless_user_agentto change the user agent for the whole browser.
On Linux the timezone and languages are set the way a real machine
sets them, not overridden over CDP. The browser starts with TZ set to the
profile's timezone_id, LC_ALL/LANG/LANGUAGE derived from its locale and
languages, and Chromium's --accept-lang flag carrying navigator_languages.
So navigator.languages holds the whole list (["fr-FR", "fr", "en-US", "en"],
not just ["fr-FR"]), Chromium writes the Accept-Language q-values itself, and
Intl uses the profile locale. An unknown timezone_id is rejected before
launch, because Chromium would report Etc/Unknown. A --accept-lang passed
in launch_args wins. macOS and Windows take the locale from system settings
instead, so there the profile is still applied through CDP overrides.
Pick a timezone_id and languages that match your exit IP. A server left on
UTC is the most common reason a clean browser still reads as a datacenter;
webskrap doctor warns when the host timezone is UTC.
Set webrtc_ip_handling_policy="disable_non_proxied_udp" when leak-test pages
should not see local or direct public WebRTC ICE candidates. WebSkrap applies
Chromium's native process flags (--webrtc-ip-handling-policy plus
--force-webrtc-ip-handling-policy) instead of patching RTCPeerConnection.
The supported policy values are default,
default_public_and_private_interfaces, default_public_interface_only, and
disable_non_proxied_udp.
Behind a proxy this is the default. A session with proxy set and no explicit
policy gets disable_non_proxied_udp, because otherwise ICE still gathers the
host's LAN address and, through STUN, its direct public address, which is the
address the proxy is meant to hide. Pass webrtc_ip_handling_policy="default"
to keep Chromium's behavior behind a proxy.
Persistent sessions take the same controls:
webskrap browser open --proxy socks5://proxy.example:1080 routes all of the
session's traffic through the proxy and blocks non-proxied UDP, and
--webrtc-ip-handling-policy overrides the policy. Both are fixed when the
browser starts, so reopening a running session with a different proxy or policy
is refused rather than silently reusing a browser that bypasses the proxy.
Persistent sessions cannot authenticate to a proxy (credentials in the URL are
refused); use an unauthenticated or IP-allowlisted proxy there, or a one-shot
session, whose ProxyConfig handles credentials.
WebRTC IP handling only controls ICE candidates. It will not hide the page's normal remote address, and it will not normalize unrelated fingerprint surfaces such as fonts, canvas, battery, device memory, or TLS/session metadata.
Set reduce_fingerprint_surface=True to ask Chromium to disable WebGL and
canvas readback with native flags (--disable-webgl and
--disable-reading-from-canvas). This reduces rendering entropy on
fingerprint-statistics pages, but pages that require WebGL or canvas export may
not work correctly, and a browser with neither is itself rare: fingerprinting
audits report it as blocking. To hide the SwiftShader renderer without losing
WebGL, use gpu="mesa" (see WebGL renderer).
Headless mode
Headless mode is more detectable than headed mode. For best-effort headless stealth, prefer real Chrome and a stable profile:
from pathlib import Path
from webskrap import SessionConfig
config = SessionConfig(
channel="chrome",
headless=True,
user_data_dir=Path(".webskrap/headless-profile"),
)WebSkrap intentionally keeps headless browser surfaces native instead of spoofing them with JavaScript. Broad fingerprint patches often become tampering signals.
Canvas, audio, and WebGL fingerprints are not randomized between requests.
They stay stable for the same browser, profile, and rendering environment;
changing the browser version, GPU backend, fonts, or host can change them.
reduce_fingerprint_surface=True disables canvas readback and WebGL instead
of making their values rotate.
Headless Chrome has no physical display, so screen and window metrics
(screen.width, window.outerWidth, ...) otherwise leak as headless tells,
defaulting to an 800x600 screen with zero outer dimensions. For chromium headless
runs WebSkrap configures a virtual screen at launch via browser flags
(--screen-info, --window-size, --window-position). The window is 80 pixels
smaller in each dimension, and screen.availHeight leaves 80 pixels at the
bottom for a taskbar. The default
screen is 1920x1080; set
headless_screen to a Viewport to change it, or to None to disable:
from webskrap import SessionConfig, Viewport
config = SessionConfig(
channel="chrome",
headless=True,
headless_screen=Viewport(width=1366, height=768),
)The other headless tell is headless mode itself: Chrome stamps
HeadlessChrome into navigator.userAgent and the worker UA (including
SharedWorker, which runs in its own process). Headless Chromium also has no
input devices, so it reports no hover and no fine pointer. WebSkrap declares a
mouse with --blink-settings in headless mode. It does not hide scrollbars or
mute audio.
Set fake_media_devices=True in SessionConfig, or pass
--fake-media-devices to fetch, search, or browser open, to use Chromium's synthetic
camera and microphone. Permission prompts remain active. Sites can detect
the synthetic devices, so the default is off.
Virtual display
On Linux, set virtual_display=True to run the browser headed on a private
Xvfb server instead of in headless mode. Nothing is overridden, so the page sees
a normal headed Chrome: no HeadlessChrome token, the full set of user-agent
client hints, real scrollbars, and an X screen sized by headless_screen
(1920x1080 when that is None). The window stays invisible.
config = SessionConfig(
channel="chrome",
headless=True,
headless_screen=Viewport(width=1366, height=768),
virtual_display=True,
)WebSkrap starts one Xvfb per session and stops it when the session closes (a
process killed with SIGKILL leaves its Xvfb running). The
server listens on no TCP port and only accepts clients holding a random
MIT-MAGIC-COOKIE stored in an owner-only file, so other local users cannot
connect to it. It needs the Xvfb binary (Debian/Ubuntu: apt install xvfb);
without it, or off Linux, the session fails to start with a browser_launch
error. virtual_display only applies when headless=True.
Masking the user agent
mask_headless_user_agent=True keeps headless mode and rewrites
HeadlessChrome to Chrome instead. WebSkrap probes the real UA once, then
applies the cleaned value with Chromium's --user-agent launch flag, which
covers the page, every worker, and request headers. The cost is client hints:
with a command-line UA override, Chromium reports only the low-entropy hints, so
navigator.userAgentData.getHighEntropyValues() and the
Sec-CH-UA-Full-Version-List, -Arch, -Bitness and -Platform-Version
headers come back empty, which a real Chrome never does. Prefer
virtual_display where Xvfb is available. The mask is ignored when
virtual_display is set, and it is off by default so headless stays honestly
headless unless you opt in.
Client hint brands
The Sec-CH-UA header and navigator.userAgentData.brands name the browser
vendor. Chrome sends Google Chrome, Edge sends Microsoft Edge, and bare
Chromium sends only Chromium and a GREASE brand such as Not_A Brand. The
user agent says Chrome in all three. Some detectors score a missing vendor
brand as a weak bot signal. Real Chromium users send the same header, so the
signal is weak.
WebSkrap does not fake the brand. A fake brand needs a per-page CDP override.
That override does not reach every worker, so pages can compare the two values.
To send a vendor brand, install Google Chrome or Microsoft Edge and keep
channel="chrome". Google Chrome has no Linux ARM64 build, so ARM64 Linux hosts
fall back to Chromium. webskrap doctor warns when it resolves to Chromium.
WebGL renderer
Without a usable GPU, headless Chrome renders WebGL with SwiftShader, and
pages can read that through WEBGL_debug_renderer_info:
ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device ...), SwiftShader driver).
Chrome 139 removed the automatic SwiftShader fallback from regular Chrome, so a
real browser no longer reports it; detectors treat it as automation.
On Linux, gpu="mesa" renders through Mesa's software Vulkan driver
(lavapipe) instead, via Chromium's own flags (--use-gl=angle --use-angle=vulkan --ignore-gpu-blocklist):
ANGLE (Mesa, Vulkan ... (llvmpipe ...), llvmpipe). That string is uncommon,
because stock Chrome blocklists software drivers, but it is not specific to
automation. It needs the lavapipe driver (Debian/Ubuntu:
apt install mesa-vulkan-drivers); without it the session fails to start with a
browser_launch error, and webskrap doctor reports whether it is installed.
Keep the default gpu="auto" on a machine with a real GPU.
config = SessionConfig(
channel="chrome",
headless=True,
gpu="mesa",
)Under virtual_display, headed Chrome with no usable GPU has no WebGL at all,
as a real GPU-less desktop does, and gpu="auto" leaves it that way; pass
gpu="mesa" when pages need WebGL. WebGPU is left alone: its adapter stays
null, which is what Chrome reports on machines without a supported GPU.
Launch flags you pass yourself (--use-gl, --use-angle) take precedence.
Practical guidance
- Reuse persistent sessions when realistic continuity matters.
- Keep locale, timezone, and languages coherent.
- Prefer an installed browser channel such as
chromewhen testing headed behavior. - Treat headless stealth as best-effort; headed Chrome, or
virtual_displayon Linux, remains the strict mode. - Avoid randomizing every request; incoherent changes can look less realistic.
Recipes
Headed Chrome for the strictest path:
from pathlib import Path
from webskrap import SessionConfig
config = SessionConfig(
channel="chrome",
headless=False,
user_data_dir=Path(".webskrap/headed-profile"),
)Invisible headed browser on a private Xvfb display (Linux):
from pathlib import Path
from webskrap import SessionConfig, Viewport
config = SessionConfig(
channel="chrome",
headless=True,
user_data_dir=Path(".webskrap/headless-profile"),
headless_screen=Viewport(width=1366, height=768),
virtual_display=True,
)Fingerprint-statistics and WebRTC leak-test mode:
config = SessionConfig(
channel="chrome",
headless=True,
profile_mode="locale",
reduce_fingerprint_surface=True,
webrtc_ip_handling_policy="disable_non_proxied_udp",
)Run the live test suite only when you intentionally want to hit third-party demo sites:
WEBSKRAP_LIVE=1 pytest -q -m live