Web Scraping with Visual Workflows: A Beginner's Guide

What is Visual Web Scraping?

Traditional web scraping requires writing code — Python scripts with BeautifulSoup, Node.js with Puppeteer. SkillChat changes this by letting you build scrapers visually.

Instead of writing `page.goto(url)` and `page.$$eval(selector)`, you drag nodes onto a canvas and connect them.

Building Your First Scraper

Step 1: Create a Scraper Workflow

Click **New Workflow**, select "Web Scraper" category, and name it (e.g., "Product Price Tracker").

Step 2: Navigate to a Page

Drag a **Scrape Page** node and set the URL. Choose "Network Idle" for the wait condition — this ensures all content loads before scraping.

Step 3: Extract Data

Add an **Extract Data** node. Set the CSS selector to target the elements you want:

`.product-title` for product names

`.price` for prices

Check "Select multiple" to get all matching elements

Step 4: Handle Pagination

For multi-page scraping, use a **Loop** node:

Extract the "next page" link with **Extract Data**

Use a **Click Element** node to navigate

**Wait For Element** to ensure the new page loads

Extract data again

Step 5: Save Results

Connect an **Output Data** node to collect all extracted data. The results are saved with your execution history.

Advanced Techniques

Screenshots for Debugging

Add a **Screenshot** node after navigation to capture what the page looks like. Screenshots are stored in Supabase Storage and viewable in your execution results.

Typing into Search Fields

Use the **Type Text** node to interact with search forms:

**Wait For Element**: `input[name="search"]`

**Type Text**: Your search query with a realistic typing delay (50ms)

**Click Element**: The search button

**Wait For Element**: Results container

Scrolling for Lazy Content

Many sites load content as you scroll. Use the **Scroll Page** node:

Scroll down 500px

Wait 1 second for content to load

Extract newly loaded elements

Repeat with a **Loop** node

Best Practices

**Set reasonable timeouts** — 30 seconds for navigation, 10 seconds for element waits

**Add delays between actions** to avoid being rate-limited

**Use the browser pool wisely** — scraper workflows run on the worker service with a limited number of browsers

**Start small** — test with a single page before building pagination loops