Pagination with Interactive Extractors
Sections
Most websites that require pagination will update the URL which makes extraction easy. For websites that do not update the URL as you paginate, you can build an interactive extractor with pagination to extract data on each page.
An interactive extractor with pagination allows you to record the steps required to go to the next page, and the extractor will treat each page as an input for you to train.
Your account will be charged per page paginated using this method. For example, running a paginated extractor with 1 input with 20 pages and screenshots enabled should consume 40 queries (20 pages + 20 screenshots).
Adding Pagination
To add pagination to your interactive extractor, simply click Next Page Action.
Recording Pagination
Once you've added pagination to your interactive extractor, record the interaction steps required to navigate to the next page. For the example below, we're extracting reviews from https://www.walmart.com/reviews/product/14301816. We simply record a mouse click on the Next Page button and add a small delay for the page to load.
Since there are 6 total pages, the Next Page button needs to be clicked 20 times. By setting the pages variable to 20, the extractor will paginate 20 times and extract the data on each page.
If you're dealing with a link that changes every time it's being clicked, you may want to modify the selector to handle different variations of the link.
Once the pagination is done, you can train the extractor like you would normally. You can modify the number of pages to paginate by modifying the value in the settings for the extractor.
