I am using scrapy for a scrapying project with this url https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11
I tried to play with the url and open it in the shell, but it got 430 error, so i added some settings to the header like that:
scrapy shell -s COOKIES_ENABLED=1 -s USER_AGENT='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0'"https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11"
it got the page "200", but once i use view(response), it directed me to a page that say:Sorry!Your web browser is not accepting cookies.
here is a screenshot of the log:
