Pls suggest some more ways other that using UserAgents and Rotaing IPs through proxies.

Written a spider code for scraping heading , url links and content in it.

I am doing a tutorial on scrapy scraping and want to scrap the data from a website. I created the scrapy project and opened the scrapy shell to test it. However, the requests gets redirected and gets status 301 and this causes an AttributeError: ‘Decompressor’ object has no attribute ‘process’.