On this website , there are many cards which are accessible in multiple areas of the site. I am attempting to scrape the site to return all of its instances(in the form of its url) based on the title. For example, a card titled: AZ School Safety program, is found at: https://lawforkids.org/officers & https://lawforkids.org/educators . My goal is for scrapy to do this for me instead of manually finding every instance.

I am doing a Scrapy Spider to download images from a page, this is the Spider’s code

I am using scrapy to query an api with restricted access.

I want to fetch all URLs in the webpage without duplicate URLs using Python scrapy. I wanted to list only URLs starting with allowed_domains = en.wikipedia.org. In case links has external links, I don’t want to scan that external link.

I want to fetch all URLs in the webpage without duplicate URLs using Python scrapy. I wanted to list only URLs starting with allowed_domains = en.wikipedia.org. In case links has external links, I don’t want to scan that external link.

It feels like every scrapy.Request I make just doesn’t exist after the first one and the response argument is somewhy always the response to the first request even if the method was called by another request.

I have this website(https://www.nfrc.co.uk/search-members), I want to click on “Search for a roofing contractor now” then For a Domestic Property and then Search. A lot of pages appear, I just need links hidden under all the company names and navigate through all the pages. I have no coding background. It would be appreciated if someone provides me the python code to scrape these links.

I keep getting 303 redirected to some checkpoint page. I’ve checked the docs and tried to look for some similar problems but haven’t found much.

I am new to use python scrapy. I am now following some yt video and I stuck here, now I want to get data from a news website, I want to get some data from its content page (containing multiple news) and some more data from the news page. Since I have to yield data from different webpage and I can only use yield once, I planned to set a global dict variable “data” and I can add things within the dict. At the end I only need to yield the “data” once.

I have setup a few different scripts using selenium and scrapy and I’ve run into an issue where the add_xpath function fails on some websites.