Client

WebSkrapClient owns the Playwright lifecycle.

Use it as an async context manager:

import asyncio
 
from webskrap import WebSkrapClient
 
 
async def main() -> None:
    async with WebSkrapClient() as client:
        result = await client.fetch("https://example.com")
    print(result.status)
 
 
asyncio.run(main())

You can also manage the lifecycle manually when embedding WebSkrap in a larger async application:

client = WebSkrapClient()
await client.start()
try:
    result = await client.fetch("https://example.com")
finally:
    await client.close()

One-shot fetch

client.fetch() creates a temporary browser session, opens the page, collects the response data, then closes that temporary session.

result = await client.fetch(
    "https://example.com",
    wait_until="domcontentloaded",
    screenshot="example.png",
)

Use client.session() when you want to keep browser state or keep a headed browser open.

Wait states

wait_until is passed to Playwright navigation and supports:

For pages with long-polling or analytics requests, networkidle can be slower or less reliable than domcontentloaded plus an explicit page wait in a session.

Screenshots and output

The Python API returns page HTML in result.text by default. Pass text_only=True when you want readable body text instead:

from pathlib import Path
 
result = await client.fetch("https://example.com", screenshot="example.png")
Path("example.html").write_text(result.text, encoding="utf-8")
print(result.screenshot_path)
 
text_result = await client.fetch("https://example.com", text_only=True)
print(text_result.text)

Pass screenshot=True to let WebSkrap choose a UUID-based file name in the form webskrap-<32 hexadecimal characters>.png in the current working directory.

Per-call config

Pass a SessionConfig for one request without changing the client's defaults:

from webskrap import ResourcePolicy, SessionConfig
 
config = SessionConfig(
    headless=True,
    resource_policy=ResourcePolicy.LITE,
    navigation_timeout_ms=60_000,
)
result = await client.fetch("https://example.com", config=config)

Cookie banners

Cookie rejection is opt-in in the Python API. When enabled, fetch() clicks the reject control before reading the page, so banners do not bury the text and no optional cookies are accepted. The CLI and MCP server enable this behavior by default.

config = SessionConfig(
    decline_cookies=True,
    decline_cookies_timeout_ms=2_000,  # wait for a late notice; 0 = check once
)
result = await client.fetch("https://example.com", config=config)
print(result.cookie_notice_declined)  # "cmp", "text", or None

For pages you drive yourself, call it on demand:

page = await session.context.new_page()
await page.goto("https://example.com", wait_until="domcontentloaded")
await session.decline_cookies(page)

decline_cookies_timeout_ms is how long to wait for a notice to appear, and so the per-fetch cost on pages that have none. Total time is bounded by that plus a 3s internal budget for clicking, including retries while a consent platform's iframe renders. Fetches that already waited for networkidle shrink the wait to 500ms, because the CMP script has had its chance to inject by then.

Detection is inspired by Brave's cookiecrumbler, which finds consent notices; WebSkrap dismisses them. Consent walls with no reject control on the first layer (pay-or-consent) are left alone.

FetchResult

FetchResult includes:

Check status before assuming headers or HTTP semantics exist:

if result.status is None:
    print("Navigation did not return a main response")
elif result.ok:
    print("Loaded", result.final_url)
else:
    print("HTTP error", result.status)