your textthe website I’m trying to scrape has a load more button and infinite scrolling after clicking the button. My problem is that after clicking the button it just scrolls down to the bottom of the page resulting to an error. I want to be able scroll to each page and extract my data until all the products are extracted.

im trying to scrape the genius’ website under the ‘all songs’ page to create a list of all songs by a given artist. However, the issue I’m having is that the page loads dynamically as you scroll, so the scraper only takes the first couple results. I’ve been trying to use selenium and the wait feature, however can’t get it to work. Heres my current code, which prints instead of appending to a list for now. It also prints “None” before each of the items which I’m not sure how to clear.