I know the basic usage of
Selenium package in python to scrape webpages, however, since I am not familiar with web development, I am having trouble figuring out how to extract statistics from this webpage, which feels pretty strange to me.
When the page is loaded for the first time, some general information are displayed to the user. Then if we click NEXT button on the top, we will enter a page where we are supposed to select a few parameters like As At Month and Occupations. When the parameters are chosen, we have to click NEXT button again for the records to be shown. To make things worse, not all records are displayed, it seems that they are split into multiple pages and the user has to scroll up and down to view different pages. In the entire process, the url stays the same.
I have tried to understand the mechanism behind these behaviors through the developer tool, however, too many interactions are involved and I got confused.
I want to know how to set parameters and get all resulting records automatically through
Selenium package in python. To answer the question, probably I need to understand how the website works in the first place, so much appreciated if someone can explain it.