首页 > 代码库 > Python爬虫框架Scrapy 学习笔记 10.2 -------【实战】 抓取天猫某网店所有宝贝详情

Python爬虫框架Scrapy 学习笔记 10.2 -------【实战】 抓取天猫某网店所有宝贝详情

第二部分 抽取起始页中进入宝贝详情页面的链接


  1. 创建项目,并生成spider模板,这里使用crawlspider。

技术分享


2. 在中scrapy shell中测试选取链接要使用的正则表达式。

首先使用firefox和firebug查看源码,定位到要链接

技术分享


然后在shell中打开网页:

scrapy shell  http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2

技术分享

确定正则表达式为:http://detail.tmall.com/item.htm\?id=\d+&rn=\w+&abbucket=\d+

3. 在spider中测试上面的正则表达式

一开始我的spider是这样写的,但是parse_item函数总是没被调用

# -*- coding: utf-8 -*-
import scrapy
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule

from tmall.items import TmallItem


class TmallSpider(CrawlSpider):
    name = ‘tmallSpider‘
    allowed_domains = [‘shanhuijj.tmall.com‘]
    start_urls = [‘http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/‘]

    rules = (
        Rule(LinkExtractor(allow=r‘http://detail.tmall.com/item.htm\?id=\d+&rn=\w+&abbucket=\d+‘,
                           restrict_xpaths="//div[@class=‘J_TItems‘]/div/dl/dd[@class=‘detail‘]"),
             callback=‘parse_item‘, follow=False),
    )

    def parse_item(self, response):
        self.log("---------------------------------------------------------")
        item = TmallItem()
        item[‘url‘] = response.url
        return item

上面给规则加了一个 restrict_xpath参数,目的是限定寻找匹配的URL的范围

run spider得到的output是

C:\Users\Bo\PycharmProjects\tmall>scrapy crawl tmallSpider
2015-01-11 22:22:59+0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: tmall)
2015-01-11 22:22:59+0800 [scrapy] INFO: Optional features available: ssl, http11
2015-01-11 22:22:59+0800 [scrapy] INFO: Overridden settings: {‘NEWSPIDER_MODULE‘: ‘tmall.spiders‘, ‘SPIDER_MODULES‘: [‘tmall.spiders‘], ‘BOT_NAME‘: ‘tmall‘}
2015-01-11 22:22:59+0800 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-01-11 22:22:59+0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, Defau
ltHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-01-11 22:22:59+0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddlewar
e
2015-01-11 22:22:59+0800 [scrapy] INFO: Enabled item pipelines:
2015-01-11 22:22:59+0800 [tmallSpider] INFO: Spider opened
2015-01-11 22:22:59+0800 [tmallSpider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-11 22:22:59+0800 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2015-01-11 22:22:59+0800 [scrapy] DEBUG: Web service listening on 127.0.0.1:6081
2015-01-11 22:22:59+0800 [tmallSpider] DEBUG: Redirecting (302) to <GET http://jump.taobao.com/jump?target=http%3a%2f%2fshanhuijj.tmall.com%2fsearch.htm%3fspm%3da
1z10.3-b.w4011-3112923129.1.KjGkS2%2f%26tbpm%3d1> from <GET http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/>
2015-01-11 22:22:59+0800 [tmallSpider] DEBUG: Redirecting (302) to <GET http://pass.tmall.com/add?_tb_token_=usBqQxmPNGXG&cookie2=1f684ca608c88f56e0df3eec780bd223
&t=323bcd0702e9fe03643e52efdb9f44bc&target=http%3a%2f%2fshanhuijj.tmall.com%2fsearch.htm%3fspm%3da1z10.3-b.w4011-3112923129.1.KjGkS2%2f%26tbpm%3d1&pacc=929hsh3NK_
FxsF9MJ5b50Q==&opi=183.128.158.212&tmsc=1420986185422946> from <GET http://jump.taobao.com/jump?target=http%3a%2f%2fshanhuijj.tmall.com%2fsearch.htm%3fspm%3da1z10
.3-b.w4011-3112923129.1.KjGkS2%2f%26tbpm%3d1>
2015-01-11 22:22:59+0800 [tmallSpider] DEBUG: Redirecting (302) to <GET http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/&tbpm=1> fro
m <GET http://pass.tmall.com/add?_tb_token_=usBqQxmPNGXG&cookie2=1f684ca608c88f56e0df3eec780bd223&t=323bcd0702e9fe03643e52efdb9f44bc&target=http%3a%2f%2fshanhuijj
.tmall.com%2fsearch.htm%3fspm%3da1z10.3-b.w4011-3112923129.1.KjGkS2%2f%26tbpm%3d1&pacc=929hsh3NK_FxsF9MJ5b50Q==&opi=183.128.158.212&tmsc=1420986185422946>
2015-01-11 22:22:59+0800 [tmallSpider] DEBUG: Redirecting (302) to <GET http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/> from <GET
http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/&tbpm=1>
2015-01-11 22:22:59+0800 [tmallSpider] DEBUG: Crawled (200) <GET http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/> (referer: None)
2015-01-11 22:22:59+0800 [tmallSpider] DEBUG: Filtered offsite request to ‘detail.tmall.com‘: <GET http://detail.tmall.com/item.htm?abbucket=17&id=10910218037&rn=
732c60a3a253eff86871e55dabc44a61>
2015-01-11 22:22:59+0800 [tmallSpider] INFO: Closing spider (finished)
2015-01-11 22:22:59+0800 [tmallSpider] INFO: Dumping Scrapy stats:
        {‘downloader/request_bytes‘: 1864,
         ‘downloader/request_count‘: 5,
         ‘downloader/request_method_count/GET‘: 5,
         ‘downloader/response_bytes‘: 74001,
         ‘downloader/response_count‘: 5,
         ‘downloader/response_status_count/200‘: 1,
         ‘downloader/response_status_count/302‘: 4,
         ‘finish_reason‘: ‘finished‘,
         ‘finish_time‘: datetime.datetime(2015, 1, 11, 14, 22, 59, 808000),
         ‘log_count/DEBUG‘: 8,
         ‘log_count/INFO‘: 7,
         ‘offsite/domains‘: 1,
         ‘offsite/filtered‘: 8,
         ‘request_depth_max‘: 1,
         ‘response_received_count‘: 1,
         ‘scheduler/dequeued‘: 5,
         ‘scheduler/dequeued/memory‘: 5,
         ‘scheduler/enqueued‘: 5,
         ‘scheduler/enqueued/memory‘: 5,
         ‘start_time‘: datetime.datetime(2015, 1, 11, 14, 22, 59, 539000)}
2015-01-11 22:22:59+0800 [tmallSpider] INFO: Spider closed (finished)

输出中出现了 ‘offsite/domains‘:1  ‘offsite/filtered‘:8 说明已经匹配到了8个链接。可是为什么parse_item方法没有执行呢?


答案是,你自己来找改过的代码与原来的代码的区别吧

# -*- coding: utf-8 -*-
import scrapy
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule

from tmall.items import TmallItem


class TmallSpider(CrawlSpider):
    name = ‘tmallSpider‘
    allowed_domains = [‘shanhuijj.tmall.com‘, ‘detail.tmall.com‘]
    start_urls = [‘http://shanhuijj.tmall.com/search.htm?spm=a1z10.3-b.w4011-3112923129.1.KjGkS2/‘]

    rules = (
        Rule(LinkExtractor(allow=r‘http://detail.tmall.com/item.htm\?id=\d+&rn=\w+&abbucket=\d+‘,
                           restrict_xpaths="//div[@class=‘J_TItems‘]/div/dl/dd[@class=‘detail‘]"),
             callback=‘parse_item‘, follow=False),
    )

    def parse_item(self, response):
        self.log("---------------------------------------------------------")
        item = TmallItem()
        item[‘url‘] = response.url
        return item


Python爬虫框架Scrapy 学习笔记 10.2 -------【实战】 抓取天猫某网店所有宝贝详情