I’m using Node.js with Puppeteer for web scraping on LinkedIn Navigator. However, I’m getting blocked after scraping around 30 to 40 pages. My goal is to scrape at least 80-100 pages.
I’ve also built a front-end solution where I manipulate the DOM, take the URL, loop through the DOM elements, and handle pagination. To make it appear more human-like, I slowed down the DOM manipulation process. However, I still face the same blocking issue after around 35-40 pages.
I’ve tried using proxies as well, but the issue persists in Node.js. Additionally, most web scraping API and SDK solutions, such as Zenscrape, ScraperAPI, and many others, don’t support LinkedIn Navigator.
Does anyone have any tips or strategies to prevent getting blocked? Any advice on how to achieve this goal would be greatly appreciated. Thanks!