Where do a browser skill's settings live?

In the main config file, not in a file of the skill's own. OpenClaw's skills config reference states that most skills configuration lives under skills in ~/.openclaw/openclaw.json, and a skill's own options sit on its entry at skills.entries. .config, which the same page calls an optional bag for custom per-skill configuration fields.

How do I check that the skill installed correctly?

Run openclaw skills check, which the skills CLI documents as reporting missing prerequisites, and openclaw skills info for one skill's resolved metadata. There is no openclaw skills test subcommand; the documented set is search, install, update, verify, list, info, check, library, curator and workshop.

Where are the logs when a browser run fails?

In the Gateway log. The openclaw logs command tails Gateway file logs over RPC and takes --follow to watch a run live. OpenClaw documents no per-skill log command, so run the failing task again while openclaw logs --follow is attached.

Back to Guides

IntermediateAdvanced TopicsOpenClaw

Browser Automation with Playwright and OpenClaw

Learn how to use Playwright MCP skills with OpenClaw to automate browser tasks: web scraping, form filling, screenshot capture, and price monitoring.

Sam OkaforUpdated September 19, 202625 minutes

#Why Browser Automation Is a Game-Changer for AI Agents

Most AI agents are limited to APIs and text-based tools. They can search the web, read files, and call endpoints, but they cannot interact with websites the way a human does. Browser automation changes that completely.

With Playwright integrated into OpenClaw, your agent can:

Navigate any website: not just API-friendly ones, but any site with a URL
Fill out forms: job applications, contact forms, registration pages, checkout flows
Extract data: scrape product prices, job listings, news articles, competitor information
Take screenshots: capture visual evidence, monitor UI changes, create reports
Interact with dynamic content: handle JavaScript-heavy SPAs, infinite scroll, modal dialogs
Automate repetitive web tasks: anything you do in a browser, your agent can do too

Browser automation is one of the most powerful capabilities you can give your OpenClaw agent. It transforms your agent from a text-only assistant into something that can truly act on your behalf across the entire web.

The combination of an LLM's reasoning ability with a real browser creates workflows that were previously impossible without custom code. Your agent can look at a webpage, understand its structure, decide what to click, and extract exactly the information you need.

#Playwright MCP Skill Setup

#What Is the Playwright MCP Skill?

The Playwright MCP skill gives OpenClaw access to a headless browser through the Model Context Protocol. It wraps Microsoft's Playwright library, providing tools for navigation, interaction, data extraction, and screenshot capture.

#Installation

Install the Playwright browser automation skill from ClawHub:

$openclaw skills install @<publisher>/playwright-browser

This installs the skill and its dependencies. The first run will download browser binaries (approximately 400 MB for Chromium).

#Browser Selection

Playwright supports three browser engines. The skills config reference puts a skill's settings in the main config file: "Most skills configuration lives under skills in ~/.openclaw/openclaw.json." On an entry under skills.entries.<key>, config is an "Optional bag for custom per-skill configuration fields", so the field names inside that bag belong to the skill rather than to OpenClaw.

playwright-browser is an illustrative skill key, as are those field names and the publisher scope in the install ref above; take the real ones from the skill's ClawHub listing before copying any block below.

json5

// ~/.openclaw/openclaw.json
{
  skills: {
    entries: {
      "playwright-browser": {
        enabled: true,
        config: {
          browser: {
            engine: "chromium", // chromium, firefox or webkit
            headless: true, // run without a visible window
            timeout: 30000, // default timeout in milliseconds
            viewport: { width: 1920, height: 1080 },
            userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
          },
        },
      },
    },
  },
}

Choosing a browser engine:

Engine	Best For	Notes
Chromium	General use, widest compatibility	Default choice, most sites work perfectly
Firefox	Sites that block Chromium headless	Different fingerprint, useful for stealth
WebKit	Safari-specific testing	Smallest footprint, macOS-native rendering

Start with Chromium. Only switch to Firefox or WebKit if you encounter sites that specifically block headless Chromium. Most users never need to change the default.

#Verify Installation

There is no openclaw skills test. The documented readiness command is openclaw skills check, which "reports missing prerequisites independently of agent exclusion", and openclaw skills info <name> resolves one skill's metadata:

$openclaw skills check

$openclaw skills info playwright-browser

A ready skill appears without a Missing requirements entry. Browser binaries are fetched by the skill on its first real run, so ask your agent for one screenshot afterwards to confirm it can launch a browser end to end.

#Basic Browser Operations

Once installed, you can ask your agent to perform browser tasks in natural language. Behind the scenes, the Playwright skill translates these requests into browser commands.

#Navigating Pages

The simplest operation is visiting a URL:

You: "Go to https://news.ycombinator.com and tell me the top 5 stories"

Your agent will:

Launch a headless browser
Navigate to the URL
Wait for the page to load
Extract the story titles
Return them in a formatted list

For programmatic access, the skill exposes these navigation tools:

yaml

# Navigation tools available
tools:
  - browser_navigate:
      url: "https://example.com"
      waitUntil: "networkidle"    # domcontentloaded, load, or networkidle
  - browser_go_back
  - browser_go_forward
  - browser_reload

#Clicking Elements

Your agent can click buttons, links, and any interactive element:

You: "Go to GitHub, click on the Explore tab, and tell me the trending repositories"

The skill uses multiple selector strategies to find elements:

yaml

# Selector strategies (in order of preference)
selectors:
  - text: "Sign In"              # By visible text
  - css: "button.submit-btn"     # By CSS selector
  - xpath: "//button[@id='go']"  # By XPath
  - role: "button[name='Submit']" # By ARIA role
  - testid: "login-button"       # By data-testid attribute

Text-based selectors are the most resilient because they match what humans see on screen. CSS selectors break when developers change class names. Prefer text selectors when possible.

#Filling Forms

Form interaction is where browser automation really shines:

You: "Go to the contact form on our website and fill in:
  Name: John Smith
  Email: john@example.com
  Message: I'd like to schedule a demo"

The skill handles various input types:

yaml

# Form interaction tools
tools:
  - browser_fill:
      selector: "#email"
      value: "john@example.com"
  - browser_select:              # Dropdowns
      selector: "#country"
      value: "United States"
  - browser_check:               # Checkboxes
      selector: "#agree-terms"
  - browser_upload:              # File uploads
      selector: "#resume"
      file: "/path/to/resume.pdf"

#Taking Screenshots

Screenshots are useful for verification, monitoring, and reporting:

You: "Take a screenshot of https://example.com and save it to my desktop"

Screenshot configuration options:

yaml

# Screenshot tools
tools:
  - browser_screenshot:
      fullPage: true              # Capture entire scrollable page
      path: "~/Desktop/shot.png"
      type: "png"                 # png or jpeg
      quality: 90                 # JPEG quality (1-100)
  - browser_screenshot_element:
      selector: ".main-content"   # Screenshot specific element
      path: "~/Desktop/content.png"

#Web Scraping: Extracting Structured Data

Web scraping is one of the most common uses for browser automation. Your agent can extract structured data from virtually any website.

#Basic Data Extraction

Ask your agent to scrape data in natural language:

You: "Go to https://books.toscrape.com and extract all book titles and prices
     from the first page into a JSON file"

The agent will navigate to the page, identify the data structure, extract each book's title and price, and save the results.

#Handling Pagination

Most real-world scraping involves multiple pages:

You: "Scrape all job listings from the first 5 pages of
     https://example-jobs.com/search?q=python+developer
     Save title, company, location, and salary to jobs.csv"

Agent navigates to the first page of results
Extracts all listings on the current page into structured data
Identifies the 'Next Page' button or pagination link
Clicks to advance to the next page and waits for load
Repeats extraction for each page up to the specified limit
Compiles all results and saves to the requested CSV format

#Dealing with Dynamic Content

Many modern websites load content dynamically with JavaScript. The Playwright skill handles this automatically:

yaml

# Dynamic content strategies
strategies:
  wait_for_selector:
    selector: ".results-loaded"
    timeout: 10000
  wait_for_network_idle:
    timeout: 5000
  infinite_scroll:
    scrolls: 10
    delay: 2000                  # Wait 2s between scrolls
  lazy_load:
    scroll_increment: 500        # Scroll 500px at a time
    wait_between: 1000           # Wait 1s for images to load

Always set reasonable timeouts when dealing with dynamic content. A page that never reaches "network idle" will cause your agent to hang until the timeout expires. Use waitForSelector targeting a specific element instead of networkidle when possible.

#Extracting Tables

HTML tables are a common data source. The skill can extract them directly:

You: "Go to https://en.wikipedia.org/wiki/List_of_countries_by_GDP
     and extract the main GDP table into a spreadsheet"

The agent will:

Navigate to the page
Locate the table element
Parse header rows and data rows
Handle merged cells and footnotes
Export as CSV or JSON

#Anti-Bot Protection and Stealth

Many websites employ anti-bot measures. Here is how to handle them responsibly.

#Common Anti-Bot Measures

Measure	Description	Mitigation
User-Agent checks	Block known headless browser UAs	Set realistic user-agent string
JavaScript challenges	Fingerprint browser capabilities	Playwright passes most checks natively
Rate limiting	Throttle rapid requests	Add delays between requests
CAPTCHAs	Human verification challenges	Slow down; some can be solved with services
IP blocking	Block IPs with too many requests	Use residential proxies or reduce frequency
Honeypot fields	Hidden form fields that trap bots	The agent's visual understanding avoids these

#Stealth Mode Configuration

Configure the Playwright skill for stealth operation in the same config bag:

json5

// ~/.openclaw/openclaw.json
{
  skills: {
    entries: {
      "playwright-browser": {
        config: {
          stealth: {
            enabled: true,
            options: {
              mask_webdriver: true, // hide the navigator.webdriver flag
              mock_plugins: true, // simulate browser plugins
              mock_languages: true, // set realistic language headers
              mock_permissions: true, // simulate the permission API
              random_viewport: false, // slight viewport randomization
            },
          },
        },
      },
    },
  },
}

#Rate Limiting and Delays

Always add delays between requests to avoid overwhelming servers:

json5

// ~/.openclaw/openclaw.json
{
  skills: {
    entries: {
      "playwright-browser": {
        config: {
          rate_limit: {
            min_delay: 2000, // minimum 2 seconds between page loads
            max_delay: 5000, // random delay up to 5 seconds
            concurrent_pages: 1, // one page at a time
            requests_per_minute: 15, // hard cap on requests
            backoff: {
              enabled: true,
              initial: 5000, // 5 second backoff on 429 responses
              multiplier: 2, // double the backoff each retry
              max: 60000, // 60 second ceiling
            },
          },
        },
      },
    },
  },
}

Rate limiting is not just about avoiding detection. It is about being a responsible web citizen. Hammering a server with rapid requests can degrade service for real users and may violate terms of service. Always add reasonable delays.

#User-Agent Rotation

Rotate user agents to reduce fingerprinting:

yaml

# User-agent rotation
user_agents:
  rotate: true
  pool:
    - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
    - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
    - "Mozilla/5.0 (X11; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0"
    - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Safari/605.1.15"

#Form Filling Automation

Automating form submissions is one of the most practical browser automation tasks. Your agent can handle everything from simple contact forms to complex multi-step applications.

#Single-Page Forms

You: "Fill out the feedback form at https://example.com/feedback with:
     - Rating: 5 stars
     - Comment: 'Great product, very intuitive interface'
     - Would recommend: Yes
     Then submit it"

#Multi-Step Wizards

Multi-step forms require the agent to navigate between steps:

You: "Go to the job application at https://careers.example.com/apply/12345
     and fill it out with my information from ~/Documents/resume-info.json"

Agent reads resume-info.json for applicant data
Navigates to the application URL
Step 1: Fills personal information (name, email, phone)
Clicks 'Next' and waits for step 2 to load
Step 2: Fills education history
Clicks 'Next' and waits for step 3 to load
Step 3: Fills work experience
Clicks 'Next' and waits for step 4 to load
Step 4: Uploads resume PDF and cover letter
Reviews all information on summary page
Takes a screenshot of the completed form before submitting
Submits the application and captures the confirmation

Always ask your agent to take a screenshot of a completed form before submitting. This gives you a visual record and a chance to verify the information is correct. You can configure an approval step where the agent pauses and sends you the screenshot before clicking Submit.

#Approval Workflow for Form Submissions

For high-stakes submissions, configure an approval step:

yaml

# Form submission approval workflow
form_automation:
  require_approval: true
  approval_channel: "telegram"    # Send screenshot to Telegram for approval
  timeout: 3600                   # Wait up to 1 hour for approval
  on_timeout: "cancel"            # Cancel if not approved in time

The agent will fill the form, screenshot it, send it to your Telegram, and wait for your "approve" or "reject" response before submitting.

#Screenshot Automation

Screenshots serve many purposes beyond simple capture: monitoring, testing, evidence collection, and reporting.

#Scheduled Website Screenshots

Monitor how a website looks over time:

You: "Set up a cron job to screenshot https://example.com every morning at 9 AM
     and save screenshots to ~/Screenshots/daily/ with the date in the filename"

$openclaw cron add --schedule "0 9 * * *" --task "Take a full-page screenshot of https://example.com and save to ~/Screenshots/daily/example-$(date +%Y-%m-%d).png"

#Visual Regression Testing

Compare screenshots to detect changes:

You: "Compare today's screenshot of our landing page with yesterday's.
     Highlight any visual differences and report what changed."

The agent will:

Take a fresh screenshot of the current page
Load the previous day's screenshot
Perform a pixel-level comparison
Generate a diff image highlighting changes
Describe the changes in plain language

#Capturing Evidence

For documentation, compliance, or dispute resolution:

You: "Go to https://competitor.com/pricing, take a timestamped screenshot,
     and save it to ~/Evidence/ with today's date. Also save the page source."

#Practical Example: Price Monitoring

Track prices across e-commerce sites and get alerts when prices drop.

#Setup

You: "I want to track these products:
     1. Sony WH-1000XM5 headphones on Amazon
     2. MacBook Air M4 on Best Buy
     3. Standing desk on IKEA
     Check prices daily at 8 AM and alert me on Telegram if any price drops by 5% or more."

#Configuration

Create a price monitoring configuration:

yaml

# ~/price-monitor.yaml
products:
  - name: "Sony WH-1000XM5"
    url: "https://www.amazon.com/dp/B0BX3K9VLM"
    selector: ".a-price-whole"
    currency: "USD"
    alert_threshold: 5            # Alert on 5% drop
  - name: "MacBook Air M4"
    url: "https://www.bestbuy.com/site/macbook-air-m4/..."
    selector: ".priceView-customer-price span"
    currency: "USD"
    alert_threshold: 3            # Alert on 3% drop
  - name: "Standing Desk"
    url: "https://www.ikea.com/us/en/p/..."
    selector: ".pip-price__integer"
    currency: "USD"
    alert_threshold: 10           # Alert on 10% drop
monitoring:
  history_file: "~/price-history.json"
  alert_channel: "telegram"

#Cron Setup

$openclaw cron add --schedule "0 8 * * *" --task "Check all products in ~/price-monitor.yaml, compare with historical prices, and alert me on Telegram if any price dropped by more than the threshold. Update price history."

#Sample Alert Output

Price Drop Alert!
Sony WH-1000XM5 Headphones
  Previous: $348.00 (Feb 22)
  Current:  $298.00 (Feb 24)
  Drop:     $50.00 (14.4%)
  Link:     https://www.amazon.com/dp/B0BX3K9VLM
Historical prices (last 30 days):
  High: $378.00 (Feb 1)
  Low:  $298.00 (today!)
  Avg:  $352.40

#Practical Example: Job Board Scraping

Aggregate job listings from multiple sites into a single spreadsheet.

#Setup

You: "Scrape Python developer jobs from Indeed, LinkedIn, and Glassdoor.
     Filter for remote positions in the US with salary > $120k.
     Save results to a Google Sheet and update daily."

#Workflow Configuration

yaml

# ~/job-search.yaml
search:
  keywords: "Python Developer"
  location: "United States"
  remote: true
  salary_min: 120000
  experience: "mid-senior"
sources:
  - name: "Indeed"
    url: "https://www.indeed.com/jobs?q=python+developer&l=remote&salary=120000"
    max_pages: 5
  - name: "LinkedIn"
    url: "https://www.linkedin.com/jobs/search/?keywords=python+developer&remote=true"
    max_pages: 3
  - name: "Glassdoor"
    url: "https://www.glassdoor.com/Job/python-developer-jobs-SRCH..."
    max_pages: 3
output:
  format: "csv"
  path: "~/job-listings.csv"
  fields:
    - title
    - company
    - location
    - salary_range
    - posted_date
    - url
    - source
  deduplicate_by: ["title", "company"]

$openclaw cron add --schedule "0 7 * * 1-5" --task "Run job board scraper using ~/job-search.yaml config, deduplicate against existing listings, and send me a Telegram summary of new listings found"

#Practical Example: Competitor Tracking

Monitor competitor websites for changes in pricing, features, and content.

#Setup

You: "Monitor these competitor pages for changes:
     1. https://competitor-a.com/pricing
     2. https://competitor-b.com/features
     3. https://competitor-c.com/blog
     Check daily, screenshot each page, and notify me of any significant changes."

#How It Works

Agent navigates to each competitor URL
Takes a full-page screenshot for visual records
Extracts the page text content
Compares text content with the previous day's snapshot
Generates a diff highlighting additions, removals, and changes
Classifies changes by significance (pricing change vs minor text edit)
Sends a summary to your configured notification channel
Archives screenshots and text snapshots in dated folders

#Sample Change Report

Competitor Tracking Report - Feb 24, 2026
Competitor A - Pricing Page
  SIGNIFICANT CHANGE DETECTED
  - Pro plan price increased from $49/mo to $59/mo
  - New "Enterprise" tier added at $199/mo
  - Free tier now limited to 100 requests (was 500)
  Screenshot: ~/competitor-tracking/competitor-a/2026-02-24.png
  Diff: ~/competitor-tracking/competitor-a/2026-02-24-diff.html
Competitor B - Features Page
  MINOR CHANGE
  - Added "AI-powered analytics" to feature list
  - Updated testimonial section
  Screenshot: ~/competitor-tracking/competitor-b/2026-02-24.png
Competitor C - Blog
  NEW CONTENT
  - New post: "How We Scaled to 1M Users"
  - Published: Feb 23, 2026
  Screenshot: ~/competitor-tracking/competitor-c/2026-02-24.png

Track mentions, engagement, and trends across social platforms.

#Setup

You: "Monitor social media for mentions of 'OpenClaw' and our competitors.
     Track Twitter, Reddit, and Hacker News.
     Send me a weekly report every Monday morning."

#Monitoring Configuration

yaml

# ~/social-monitor.yaml
keywords:
  brand: ["OpenClaw", "openclaw", "@openclaw"]
  competitors: ["CompetitorX", "CompetitorY"]
  industry: ["AI agent", "personal AI assistant", "autonomous agent"]
sources:
  twitter:
    enabled: true
    search_url: "https://twitter.com/search?q=OpenClaw"
    track_metrics: ["mentions", "sentiment", "engagement"]
  reddit:
    enabled: true
    subreddits: ["selfhosted", "artificial", "MachineLearning", "LocalLLaMA"]
    search_terms: ["OpenClaw", "AI agent"]
  hackernews:
    enabled: true
    search_url: "https://hn.algolia.com/?q=OpenClaw"

$openclaw cron add --schedule "0 8 * * 1" --task "Run social media monitoring report using ~/social-monitor.yaml, compile weekly summary with sentiment analysis, and send to Telegram"

#Data Extraction Workflows

#Saving to CSV

You: "Scrape the product catalog from https://example-store.com
     and save all products with name, price, category, and rating to products.csv"

The agent outputs clean CSV:

csv

name,price,category,rating,url
"Widget Pro",29.99,"Electronics",4.5,"https://example-store.com/widget-pro"
"Gadget X",49.99,"Electronics",4.2,"https://example-store.com/gadget-x"
"Super Tool",15.99,"Tools",4.8,"https://example-store.com/super-tool"

#Saving to JSON

For structured data that preserves hierarchy:

You: "Extract the full product catalog with all details including
     descriptions, images, and specifications. Save as JSON."

json

{
  "scraped_at": "2026-02-24T08:00:00Z",
  "source": "https://example-store.com",
  "total_products": 156,
  "products": [
    {
      "name": "Widget Pro",
      "price": 29.99,
      "category": "Electronics",
      "rating": 4.5,
      "description": "Professional-grade widget with...",
      "images": ["https://example-store.com/img/widget-pro-1.jpg"],
      "specifications": {
        "weight": "0.5 kg",
        "dimensions": "10x5x3 cm",
        "color": "Matte Black"
      }
    }
  ]
}

#Piping Data to Other Tools

The real power comes from chaining browser data extraction with other skills:

You: "Scrape the latest tech job salaries from levels.fyi,
     analyze the trends, create a chart, and post it to our Slack channel"

This chains: Playwright (scrape) -> File Manager (save data) -> Analysis (process) -> Slack (notify).

#Scheduling with Cron

#Common Scheduling Patterns

bash

# Every morning at 8 AM
openclaw cron add --schedule "0 8 * * *" --task "..."
# Every weekday at 9 AM and 5 PM
openclaw cron add --schedule "0 9,17 * * 1-5" --task "..."
# Every 6 hours
openclaw cron add --schedule "0 */6 * * *" --task "..."
# Every Monday at 7 AM
openclaw cron add --schedule "0 7 * * 1" --task "..."
# First day of each month
openclaw cron add --schedule "0 9 1 * *" --task "..."

#Managing Scheduled Jobs

$openclaw cron list

ID  Schedule        Task                                    Status    Last Run
1   0 8 * * *       Price monitor                           Active    Feb 24, 08:00
2   0 7 * * 1-5     Job board scraper                       Active    Feb 24, 07:00
3   0 9 * * *       Competitor tracking                     Active    Feb 24, 09:00
4   0 8 * * 1       Social media report                     Active    Feb 17, 08:00

$openclaw cron runs 1 --limit 5

View the recent run history for any scheduled task to verify it is running correctly. There is no cron logs subcommand and no --last flag; run history is runs with --limit, and the Gateway's own stream is openclaw logs --follow.

#Error Handling and Resilience

Web scraping is inherently fragile. Sites change their structure, go down, or add new protections. Build resilience into your workflows.

#Retry Logic

yaml

# Error handling configuration
error_handling:
  retries: 3
  retry_delay: 5000              # 5 seconds between retries
  backoff_multiplier: 2          # Exponential backoff
  retry_on:
    - "timeout"
    - "navigation_failed"
    - "element_not_found"
  fail_on:
    - "blocked"
    - "captcha_required"

#Handling Site Changes

When a site changes its structure, selectors break. Build in fallback strategies:

yaml

# Fallback selectors for price extraction
selectors:
  primary: ".product-price .current"
  fallbacks:
    - ".price-current"
    - "[data-price]"
    - ".price"
    - "span:has-text('$')"
  on_all_fail: "screenshot_and_notify"

When all selectors fail, have your agent take a screenshot and send it to you with a message like "The pricing page structure has changed. Here is what it looks like now. Should I update the selectors?" This turns a failure into a quick manual fix.

#Health Checks

Add health checks to your scraping workflows:

yaml

# Health check configuration
health_check:
  verify_data:
    min_results: 1               # Expect at least 1 result
    max_results: 10000           # Sanity check upper bound
    required_fields: ["name", "price"]
  on_anomaly:
    action: "notify"
    channel: "telegram"
    message: "Scraping anomaly detected: {details}"

#Ethics and Legal Considerations

Browser automation is powerful, but it comes with responsibilities. Always operate ethically and within legal boundaries.

#Respect robots.txt

Before scraping any site, check its robots.txt file:

You: "Check if https://example.com allows scraping in their robots.txt"

robots.txt is a voluntary standard, not a legal requirement. However, ignoring it can lead to your IP being blocked, legal action from the site owner, or terms of service violations. Always respect it.

#Rate Limiting Is Non-Negotiable

Never send more requests than a human would. Good rules of thumb:

Minimum 2-second delay between page loads
Maximum 15-20 pages per minute for general browsing
Reduce frequency for small sites with limited server capacity
Stop immediately if you receive 429 (Too Many Requests) responses

#Terms of Service

Many websites explicitly prohibit automated access in their terms of service. Before setting up automated scraping:

Read the target site's Terms of Service
Check their robots.txt
Look for an official API (always prefer APIs over scraping)
Consider reaching out to ask permission for large-scale data collection

#Data Privacy

Be careful with scraped data:

Do not scrape personal information without consent or legal basis
GDPR and CCPA apply to data you collect, even through scraping
Do not store sensitive data (emails, phone numbers, addresses) without a legitimate purpose
Delete data when it is no longer needed

#Best Practices Checklist

Check robots.txt before scraping any site
Read the site's Terms of Service for automation restrictions
Prefer official APIs over scraping when available
Always add 2+ second delays between requests
Set a reasonable maximum requests-per-minute cap
Never scrape personal or sensitive data without legal basis
Store only the data you actually need
Monitor your scraping jobs for errors and anomalies
Be prepared to stop if the site owner requests it
Document your scraping activities for compliance purposes

#Complete Browser Automation Config Example

Here is a full configuration bringing together everything covered in this guide, as one entry in ~/.openclaw/openclaw.json:

json5

// ~/.openclaw/openclaw.json
{
  skills: {
    entries: {
      "playwright-browser": {
        enabled: true,
        config: {
          browser: {
            engine: "chromium",
            headless: true,
            timeout: 30000,
            viewport: { width: 1920, height: 1080 },
          },
          stealth: {
            enabled: true,
            options: {
              mask_webdriver: true,
              mock_plugins: true,
              mock_languages: true,
            },
          },
          user_agents: {
            rotate: true,
            pool: [
              "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
              "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
              "Mozilla/5.0 (X11; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0",
            ],
          },
          rate_limit: {
            min_delay: 2000,
            max_delay: 5000,
            concurrent_pages: 1,
            requests_per_minute: 15,
            backoff: { enabled: true, initial: 5000, multiplier: 2, max: 60000 },
          },
          error_handling: {
            retries: 3,
            retry_delay: 5000,
            backoff_multiplier: 2,
            screenshots_on_error: true,
            error_screenshot_dir: "~/browser-errors/",
          },
          output: {
            default_format: "json",
            default_dir: "~/browser-data/",
            timestamp_files: true,
            compress_old: true,
            retention_days: 90,
          },
        },
      },
    },
  },
}

#Troubleshooting

#Common Issues

Issue	Cause	Solution
"Browser not found"	Chromium not downloaded	Reinstall with `openclaw skills install @<publisher>/playwright-browser --force`, which "Overwrites an existing workspace skill folder for the same slug."
Timeout on page load	Slow site or heavy JS	Increase `timeout` in config, use `domcontentloaded` instead of `networkidle`
Elements not found	Site structure changed	Update selectors, use text-based selectors for resilience
Screenshots are blank	Page not fully rendered	Add `waitForSelector` before screenshot
"Blocked by Cloudflare"	Bot detection triggered	Enable stealth mode, add delays, try Firefox engine
Memory issues	Too many pages open	Ensure pages are closed after use, set `concurrent_pages: 1`

#Getting Help

If you encounter issues not covered here, read the Gateway log. openclaw logs tails "Gateway file logs over RPC", and OpenClaw documents no per-skill log command:

$openclaw logs --follow

Run the failing task again while that is attached, and the browser errors and stack traces come through the stream. Most issues are related to selector changes on target websites or network timeouts.

#Next Steps

Now that you have browser automation set up, consider:

Combining with email automation: scrape data, then email reports automatically
Building data pipelines: scrape, process, store, visualize
Creating custom monitoring dashboards: aggregate scraped data into a single view
Exploring the Playwright MCP skill's advanced features: PDF generation, video recording, network interception

Browser automation is one of the most versatile tools in your OpenClaw toolkit. Start with simple tasks and build up to complex multi-site workflows as you gain confidence.

Frequently asked questions

Where do a browser skill's settings live?: In the main config file, not in a file of the skill's own. OpenClaw's skills config reference states that most skills configuration lives under skills in ~/.openclaw/openclaw.json, and a skill's own options sit on its entry at skills.entries.<key>.config, which the same page calls an optional bag for custom per-skill configuration fields.
How do I check that the skill installed correctly?: Run openclaw skills check, which the skills CLI documents as reporting missing prerequisites, and openclaw skills info <name> for one skill's resolved metadata. There is no openclaw skills test subcommand; the documented set is search, install, update, verify, list, info, check, library, curator and workshop.
Where are the logs when a browser run fails?: In the Gateway log. The openclaw logs command tails Gateway file logs over RPC and takes --follow to watch a run live. OpenClaw documents no per-skill log command, so run the failing task again while openclaw logs --follow is attached.

#Why Browser Automation Is a Game-Changer for AI Agents

#Playwright MCP Skill Setup

#What Is the Playwright MCP Skill?

#Installation

#Browser Selection

#Verify Installation

#Basic Browser Operations

#Navigating Pages

#Clicking Elements

#Filling Forms

#Taking Screenshots

#Web Scraping: Extracting Structured Data

#Basic Data Extraction

#Handling Pagination

#Dealing with Dynamic Content

#Extracting Tables

#Anti-Bot Protection and Stealth

#Common Anti-Bot Measures

#Stealth Mode Configuration

#Rate Limiting and Delays

#User-Agent Rotation

#Form Filling Automation

#Single-Page Forms

#Multi-Step Wizards

#Approval Workflow for Form Submissions

#Screenshot Automation

#Scheduled Website Screenshots

#Visual Regression Testing

#Capturing Evidence

#Practical Example: Price Monitoring

#Setup

#Configuration

#Cron Setup

#Sample Alert Output

#Practical Example: Job Board Scraping

#Setup

#Workflow Configuration

#Practical Example: Competitor Tracking

#Setup

#How It Works

#Sample Change Report

#Practical Example: Social Media Monitoring

#Setup

#Monitoring Configuration

#Data Extraction Workflows

#Saving to CSV

#Saving to JSON

#Piping Data to Other Tools

#Scheduling with Cron

#Common Scheduling Patterns

#Managing Scheduled Jobs

#Error Handling and Resilience

#Retry Logic

#Handling Site Changes

#Health Checks

#Ethics and Legal Considerations

#Respect robots.txt

#Rate Limiting Is Non-Negotiable

#Terms of Service

#Data Privacy

#Best Practices Checklist

#Complete Browser Automation Config Example

#Troubleshooting

#Common Issues

#Getting Help

#Next Steps

Frequently asked questions

Sources

Skip the trial and error