Scraping pagination, infinite scroll and load-more buttons

Choosing the mode.

You set it in the side panel, from the Flow dropdown, before you export. Scroll to the bottom of the page you want and look at what is there:

• Numbered pages, or a Next arrow, and the URL changes when you click it — Pagination.
• More rows appear on their own as you approach the bottom — Infinite scroll.
• A button that says Load more, Show more or similar, and the rows appear when you press it — Load more button.
• The list simply ends — Single page.

Pagination and Load more need one extra pick: press Pick next/load-more button in the panel, then click the link or button on the page. That becomes NEXT_SELECTOR in the exported file.

Single page.

The base case, and the one worth understanding first because the other three are variations on it. open_start_page loads the URL and waits for the item selector to appear; rows_on reads every item on whatever is currently loaded.

def scrape(page):
    """Collect every item on the start page."""
    open_start_page(page)
    return rows_on(page)

Every other mode calls those same two functions. All that changes is what happens in between.

Pagination: following a next link.

Worked example. quotes.toscrape.com lists ten quotes a page with a Next link at the bottom. Pick the quote, the author and the tags; press Pick next/load-more button and click Next; set Flow to Pagination and Max pages to 5. The picker resolves the item to div.quote and the next link to li.next a, and the export contains this:

def scrape(page):
    """Walk every page via the next link, collecting items as we go."""
    open_start_page(page)
    rows = []
    pages_done = 0
    while True:
        rows += rows_on(page)
        pages_done += 1
        if MAX_PAGES is not None and pages_done >= MAX_PAGES:
            break
        next_link = page.query_selector(NEXT_SELECTOR)
        if next_link is None:
            break
        previous_url = page.url
        try:
            next_link.click()
            page.wait_for_load_state()
            page.wait_for_selector(ITEM_SELECTOR)
        except Exception as error:
            # Usually a consent banner over the link: a fresh profile has agreed to nothing, so
            # the overlay is there and it eats the click (see the storage_state note in main()).
            # The pages already walked are still yours; losing them to the last click is not.
            print(
                f"Stopped paginating: {error} — keeping the {len(rows)} rows collected so far.",
                file=sys.stderr,
            )
            break
        time.sleep(DELAY_SECONDS)
        # A next arrow that never disappears would loop forever, so stop if the URL
        # did not move. Site paginates without changing it? Delete this, set MAX_PAGES.
        if page.url == previous_url:
            break
    return rows

NEXT_SELECTOR

The link that goes to the following page. Yours from the pick; a string you can edit.

MAX_PAGES

The cap you set in the panel. Set it to None to walk every page there is.

DELAY_SECONDS

The pause after each page load. One second by default; raise it to be gentler on the site.

There are three ways out of that loop, and the third is the one people miss. It stops at the page cap; it stops when the next link is no longer on the page; and it stops when clicking the link did not change the URL. That last check exists because plenty of sites render a permanent Next arrow that simply does nothing on the final page — without it the script clicks the same dead arrow until you kill it. If your site paginates without changing the URL, that check will stop you after page one: delete it and use MAX_PAGES instead. The comment in the generated file says exactly that, so you do not have to work it out from the outside.

A click that raises — most often a cookie banner sitting over the link — is caught, reported on standard error, and ends the walk with every row collected so far intact. Losing four pages of data to a failure on the fifth is not a trade anybody wants.

Infinite scroll: reading a feed that grows.

Worked example. A feed with no pager and no button, where rows appear as you approach the bottom. Pick your fields, set Flow to Infinite scroll, and export. There is nothing else to pick — the item selector is the whole instruction.

def scrape(page):
    """Scroll until the item count stops growing (3 quiet rounds)."""
    open_start_page(page)
    quiet_rounds = 0
    seen = 0
    scrolls = 0
    while quiet_rounds < 3 and scrolls < MAX_SCROLLS:
        try:
            page.mouse.wheel(0, 20000)
        except Exception as error:
            # Whatever stopped the scroll, what has already loaded is on the page and is about
            # to be read. Ending here costs the rest of the feed, not the whole run.
            print(f"Stopped scrolling: {error} — reading what has loaded.", file=sys.stderr)
            break
        scrolls += 1
        time.sleep(DELAY_SECONDS)
        count = len(page.query_selector_all(ITEM_SELECTOR))
        quiet_rounds = quiet_rounds + 1 if count <= seen else 0
        # The highest count seen, not the latest one. A virtualized feed drops the items you
        # scrolled past, so the live count falls and then rises again; against the latest
        # count that rise looks like new content, resets quiet_rounds, and the loop never ends.
        seen = max(seen, count)
    return rows_on(page)

MAX_SCROLLS

A hard stop at 200 wheel events. A feed that never runs out would otherwise scroll forever.

DELAY_SECONDS

How long to wait after each scroll for the next batch to arrive. This is the knob to raise on a slow site.

If you have written this loop yourself in Selenium — scroll to document.body.scrollHeight, sleep, compare the height, repeat — you already know the shape. Two details in this version are worth stealing regardless of what you scrape with.

It counts items, not pixels. Page height is a proxy for content and a bad one: a sticky footer, a lazily-sized image or an expanding advert all change the height without adding a single row. The count of elements matching your item selector is the thing you actually care about, so that is what the loop watches.

It remembers the highest count, not the last one. This is the failure that turns a scroll loop into an infinite loop. A virtualised feed — the kind that keeps the DOM small by discarding rows you have scrolled past — reports fewer items than it did a moment ago, and then more again. Compared against the latest count, that rise looks like new content, resets the patience counter, and the loop never ends. Compared against the highest count ever seen, it correctly looks like nothing new.

The loop gives up after three consecutive rounds that added nothing, then reads the page once. Note the consequence: a virtualised feed that discards old rows will have discarded some by the time it finishes, and this mode will not see them. For those, pagination or a load-more button — if the site offers either — is the more reliable route.

Load more button: clicking until it gives up.

Worked example. A listing with a Load more button under it. Pick your fields, press Pick next/load-more button and click the button, then set Flow to Load more button.

def scrape(page):
    """Click the load-more button until it disappears or stops adding items."""
    open_start_page(page)
    quiet_rounds = 0
    seen = 0
    clicks = 0
    while quiet_rounds < 3 and clicks < MAX_CLICKS:
        button = page.query_selector(NEXT_SELECTOR)
        # Plenty of sites HIDE the button when the list runs out instead of removing it, so
        # query_selector still finds it. Clicking a hidden element waits for it to become
        # visible and never returns — 30s later the whole run dies with nothing written.
        if button is None or not button.is_visible():
            break
        try:
            button.click()
        except Exception as error:
            # A button that will not take a click is a button that is finished. Every row
            # collected up to here is still yours; losing them to the last click is not.
            print(f"Stopped loading more: {error}", file=sys.stderr)
            break
        clicks += 1
        time.sleep(DELAY_SECONDS)
        count = len(page.query_selector_all(ITEM_SELECTOR))
        quiet_rounds = quiet_rounds + 1 if count <= seen else 0
        # The highest count seen, not the latest one: a list that recycles the rows you have
        # scrolled past can report fewer items than a moment ago, which is not new content.
        seen = max(seen, count)
    return rows_on(page)

NEXT_SELECTOR

The button. Yours from the pick.

MAX_CLICKS

A hard stop at 200 clicks, for a button that never runs out.

DELAY_SECONDS

The pause after each click, waiting for the batch to arrive.

The is_visible() check is the whole reason this mode is not simply "click until the button is gone". Plenty of sites hide the load-more button when the list runs out rather than removing it from the DOM, so query_selector still finds it happily. Clicking a hidden element in Playwright waits for it to become visible, which it never will, and thirty seconds later the entire run dies with nothing written to disk. Asking whether it is visible before clicking turns that into a clean stop with all your rows.

Otherwise the stopping logic matches infinite scroll: three consecutive clicks that add nothing, and it is done.

Drill-down: fields from the detail page.

This one is not a flow mode — it combines with any of the four. Mark a link column to follow, then pick fields on the page it leads to; those columns join every row.

Worked example. books.toscrape.com shows a title and a price on the listing, and the description only on each book's own page. Pick the title, the price and the title's link, mark the link as the one to follow, then pick the description on the detail page that opens. The listing walk is unchanged — pagination, in this example — and two things are added to the file:

def extract_detail(page, url):
    """Visit one item's detail page and pull its extra fields."""
    # Same cheap wait the start page uses, and it matters more here because the cost is per ROW:
    # under the default "load" one hanging tracker on a product template costs thirty seconds on
    # every item, so a two-hundred-row drill-down spends over an hour waiting for ad pixels on
    # pages it had finished reading. The fields below are read straight off the document.
    response = page.goto(url, wait_until="domcontentloaded")
    # goto() raises on a TRANSPORT failure — refused connection, bad DNS — and an HTTP error is
    # not one of those: 404, 403 and 500 are successful navigations to a page that says no. Left
    # unchecked, the selectors below find nothing on that page and every detail cell records "",
    # which in the CSV is indistinguishable from a detail page that genuinely had no description.
    # Raising here routes it into the caller's per-row arm, so the row keeps its listing fields,
    # carries no detail ones, and says on the way past which URL failed and why.
    #
    # None is not a failure: goto() returns it for a navigation that did not produce a new
    # document, which is a statement about the navigation rather than about the server.
    if response is not None and not response.ok:
        raise RuntimeError(f"HTTP {response.status} at {url}")
    page.wait_for_load_state()
    return {
        "description": text(page, "#product_description + p"),
    }

        rows = scrape(page)
        for row in rows:
            url = row["link"]
            if url:
                try:
                    row.update(extract_detail(page, url))
                except Exception as error:
                    # One dead link must not cost you a finished scrape. This row keeps the
                    # fields read off the listing and simply has no detail ones. On stderr with
                    # every other warning this script emits: stdout carries the row count, and a
                    # skipped page is the kind of thing you want to see when that is piped away.
                    print(f"Skipped detail page {url}: {error}", file=sys.stderr)
                time.sleep(DELAY_SECONDS)

The whole listing is collected first, then each row's link is visited in turn. A detail page that fails costs you that page's fields and nothing else — the row keeps everything the listing gave it, a line goes to standard error naming the URL and the reason, and the run carries on. The HTTP status check is what makes that true: without it, a 404 would be a successful navigation to a page where every selector finds nothing, and the row would record empty strings that are indistinguishable in the CSV from a book that genuinely has no description.

Be aware of the cost. Drill-down turns one page load into one page load per row, with DELAY_SECONDS between each. Two hundred rows is two hundred extra visits, so this is the mode where the delay setting and a sensible MAX_PAGES earn their keep.

When the walk goes wrong.

• Only the first page came back. On pagination, either NEXT_SELECTOR no longer matches, or the URL-unchanged check fired. Test the selector in the browser console with document.querySelector("li.next a"); if it is fine, the URL check is your culprit.
• Zero rows, and the page obviously has items. Set HEADLESS = False and run again. A consent overlay, a login wall and a wrong ITEM_SELECTOR are indistinguishable until you look. If it is a banner or a login, the comment inside main() has the storage_state recipe for recording a session once and reusing it.
• Fewer rows than the feed has. On infinite scroll, raise DELAY_SECONDS first — three quiet rounds pass quickly on a site slower than the loop expects — then MAX_SCROLLS.
• It never finishes. Both hard stops exist for this, so lower MAX_SCROLLS or MAX_CLICKS and see how far it actually gets.
• Rows with every field empty. They are already dropped: an item whose columns all came back empty means ITEM_SELECTOR matched something that is not an item, so the script discards it rather than padding your CSV with blank lines.

All of these are edits to constants at the top of a file you own. The annotated walk-through covers the rest of the script — dependencies, how to run it, and putting it on a cron schedule.