Client
WebSkrapClient owns the browser lifecycle. It starts a browser on first use
and closes every browser it started when it exits.
Use it as an async context manager:
import asyncio
from webskrap import WebSkrapClient
async def main() -> None:
async with WebSkrapClient() as client:
result = await client.fetch("https://example.com")
print(result.status)
asyncio.run(main())You can also manage the lifecycle manually when embedding WebSkrap in a larger async application:
client = WebSkrapClient()
await client.start()
try:
result = await client.fetch("https://example.com")
finally:
await client.close()One-shot fetch
client.fetch() creates a temporary browser session, opens the page, collects the response data, then closes that temporary session.
result = await client.fetch(
"https://example.com",
wait_until="domcontentloaded",
screenshot="example.png",
)Use client.session() when you want to keep browser state or keep a headed browser open.
Wait states
wait_until sets the load state that navigation waits for:
| Value | Use when |
|---|---|
commit |
You only need the navigation to begin. |
domcontentloaded |
You need parsed HTML and want a fast default. |
load |
You need regular page assets to finish loading. |
networkidle |
You need a quiet network, usually for JS-heavy pages. |
For pages with long-polling or analytics requests, networkidle can be slower
or less reliable than domcontentloaded plus an explicit page wait in a session.
For deferred content, fetches can wait for a visible CSS selector or snapshot ref:
result = await client.fetch(
"https://example.com",
ready_selector="#content",
timeout_ms=30_000,
)The readiness wait starts after consent dismissal and gets its own timeout_ms
budget, or the configured navigation timeout. A missing selector raises a
timeout; a blank selector is a usage error. session.fetch() accepts it too.
Known challenge pages (DataDome, Cloudflare, Akamai, AWS WAF) and provider block pages (Cloudflare, Akamai) raise WebSkrapError with code
blocked, including pages served with HTTP 200. error.hint
says what to try for that kind. They are checked before
consent and extraction, and after a readiness timeout. Ordinary HTTP error pages
still return ok=False. Detection uses known provider markers and headers and
does not identify every challenge or solve verification widgets.
Screenshots and output
The Python API returns page HTML in result.text by default. Pass
text_only=True when you want readable body text instead:
from pathlib import Path
result = await client.fetch("https://example.com", screenshot="example.png")
Path("example.html").write_text(result.text, encoding="utf-8")
print(result.screenshot_path)
text_result = await client.fetch("https://example.com", text_only=True)
print(text_result.text)Pass screenshot=True to let WebSkrap choose a UUID-based file name in the
form webskrap-<32 hexadecimal characters>.png in the current working
directory.
Per-call config
Pass a SessionConfig for one request without changing the client's defaults:
from webskrap import ResourcePolicy, SessionConfig
config = SessionConfig(
headless=True,
resource_policy=ResourcePolicy.LITE,
navigation_timeout_ms=60_000,
)
result = await client.fetch("https://example.com", config=config)Cookie banners
Cookie rejection is opt-in in the Python API. When enabled, fetch() clicks the
reject control before reading the page, so banners do not bury the text and no
optional cookies are accepted. The CLI and MCP server enable this behavior by
default.
config = SessionConfig(
decline_cookies=True,
decline_cookies_timeout_ms=2_000, # wait for a late notice; 0 = check once
)
result = await client.fetch("https://example.com", config=config)
print(result.cookie_notice_declined) # "cmp", "text", or NoneFor pages you drive yourself, call it on demand:
page = await session.context.new_page()
await page.goto("https://example.com", wait_until="domcontentloaded")
await session.decline_cookies(page)decline_cookies_timeout_ms is how long to wait for a notice to appear, and so
the per-fetch cost on pages that have none. Total time is bounded by that plus a
3s internal budget for clicking, including retries while a consent platform's
iframe renders. Fetches that already waited for networkidle shrink the wait to
500ms, because the CMP script has had its chance to inject by then.
Detection is inspired by Brave's cookiecrumbler, which finds consent notices; WebSkrap dismisses them. Consent walls with no reject control on the first layer (pay-or-consent) are left alone.
FetchResult
FetchResult includes:
| Field | Meaning |
|---|---|
url |
URL requested by the caller. |
final_url |
Page URL after redirects and navigation. |
status |
Main response HTTP status, or None when no response is available. |
ok |
True when status is 2xx or 3xx. |
headers |
Main response headers. |
text |
Final page HTML from page.content(). |
title |
Final document title. |
cookies |
Browser-context cookies visible after the fetch. |
timings |
Timing data, currently including elapsed_ms. |
screenshot_path |
Screenshot path when screenshot capture was requested. |
cookie_notice_declined |
Strategy that dismissed a cookie consent notice (cmp, text), or None. |
Check status before assuming headers or HTTP semantics exist:
if result.status is None:
print("Navigation did not return a main response")
elif result.ok:
print("Loaded", result.final_url)
else:
print("HTTP error", result.status)Select elements
result.css(selector) returns matching elements in document order.
result.css_first(selector) returns the first match or None. Each Element
has tag, attrs, parent, children, and text, and supports the same two
methods. Use parse_html(html) to query HTML from another source.
from webskrap import WebSkrapClient
async with WebSkrapClient() as client:
result = await client.fetch("https://example.com")
for row in result.css("table.prices tr"):
print([cell.text.strip() for cell in row.css("td")])
link = result.css_first("a[href^='https://']")
print(link.attrs["href"] if link else None)Supported selectors are type and *, #id, .class, attribute tests
([a], =, ~=, |=, ^=, $=, *=), the descendant and >
combinators, and , lists. Other syntax, such as pseudo-classes and sibling
combinators, raises ValueError.
The parser runs offline on result.text. A text_only fetch returns plain
text, not HTML, so fetch without text_only before you query. The parser
handles void elements and common implied end tags. It does not run the full
HTML5 tree-construction algorithm, so badly broken markup can nest
differently than in the browser. It also does not insert implied elements
such as tbody. To query the browser's live tree, use a page from a session.