首页 > 代码库 > scrapy 修改URL爬取起始位置

scrapy 修改URL爬取起始位置

import scrapy
from Autopjt.items import myItem
from scrapy.http import Request

class AutospdSpider(scrapy.Spider):
    name = "fulong_spider"
    start_urls = [http://category.dangdang.com/pg1-cid4007379.html]
    url2 = (http://dangdang.com,http://jd.com,http://tianmao.com,)

    def start_requests(self):
        for url in self.url2:
            yield self.make_requests_from_url(url)

    def parse(self, response):
        item = myItem()
        item[name] =response.xpath(/html/head/title/text()).extract()
        print(item[name])
需要重写start_requests方法

scrapy 修改URL爬取起始位置