首页 > 代码库 > Scrapy爬虫 -- 03
Scrapy爬虫 -- 03
关于数据过滤，scrapy提供xpath和css两种过滤器（selector），一般xpath使用的较多，另外我对css也不算熟。这里主要是xpath。
关于xpath，是一种专门在 XML 文档中查找信息的语言。详细教程可以看这里：http://www.w3school.com.cn/xpath/，不过对于刚入门的人来说不用那么复杂。官网的tutorial给的一些示例足够基本入门的。
以下为示例：
/html/head/title: selects the <title> element, inside the <head>element of a HTML document （选择html头部的标题元素） /html/head/title/text(): selects the text inside the aforementioned<title> element.（选择上例中元素的文本） //td: selects all the <td> elements（选择所有td项目） //div[@class="mine"]: selects all div elements which contain an attribute class="mine"（选择class="mine"的div标签）
为了方便调式，scrapy提供了交互式的方式对网站进行抓取分析（注意一定要有""，否则。。。）：
scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"
正常的话会出现以下内容：
[ ... Scrapy log here ... ]
2014-01-23 17:11:42-0400 [default] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
[s] Available Scrapy objects:
[s] crawler <scrapy.crawler.Crawler object at 0x3636b50>
[s] item {}
[s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s] settings <scrapy.settings.Settings object at 0x3fadc50>
[s] spider <Spider ‘default‘ at 0x3cebf50>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
In [1]:
这样你就获得了crawler等六个类，现在我们只关注response类，正如字面意思，response代表返回的网页内容，使用view（response）就可以调用默认的浏览器查看抓取的页面（我的是firefox）。response.body存放着html的源代码，你可以在交互程序下查看。不过一般没有换行和对齐，十分惨烈。。。
response中包含了四个基本方法，用于数据过滤：
xpath(): returns a list of selectors, each of them representing the nodes selected by the xpath expression given as argument.
css(): returns a list of selectors, each of them representing the nodes selected by the CSS expression given as argument.
extract(): returns a unicode string with the selected data.
re(): returns a list of unicode strings extracted by applying the regular expression given as argument.
其中xpath()就是选择器，而extract()方法则会返回html标签之间的unicode内容，re()则是调用正则的接口（我恨正则）。
示例如下：
In [1]: response.xpath(‘//title‘) Out[1]: [<Selector xpath=‘//title‘ data=http://www.mamicode.com/u‘Open Directory - Computers: Progr‘>]> <p>response.xpath()将会返回selector的一个列表，你可以对其中的元素再次调用xpath, extract()等函数。</p> <p>就像这样：</p> <pre class="brush:python;toolbar: true; auto-links: false;">>>> response.xpath(‘//title‘)[0].xpath(‘text()‘).extract() [u‘DMOZ - Computers: Programming: Languages: Python: Books‘]</pre> <p>剩下的事就是把元素存入items之中了，然后通过pipeline存放到你想要的数据格式：</p> <pre class="shadow-none p-3 mb-5 bg-light rounded">import scrapyfrom tutorial.items import DmozItemclass DmozSpider(scrapy.Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response): for sel in response.xpath(‘//ul/li‘): item = DmozItem() item[‘title‘] = sel.xpath(‘a/text()‘).extract() item[‘link‘] = sel.xpath(‘a/@href‘).extract() item[‘desc‘] = sel.xpath(‘text()‘).extract() yield item</pre> <p>这里说一下yield这个关键字，yield在python中被称为generator，作用是将循环转化为iterator并返回，详情可以看这里：http://www.ibm.com/developerworks/cn/opensource/os-cn-python-yield/</p><p>Scrapy爬虫 -- 03</p></span> </div> <nav class="p-0 mb-4 bg-white text-left"> <a href="/itag/60/" title="des" class="tag" target="_blank">des</a> <a href="/itag/2/" title="http" class="tag" target="_blank">http</a> <a href="/itag/736/" title="io" class="tag" target="_blank">io</a> <a href="/itag/662/" title="os" class="tag" target="_blank">os</a> <a href="/itag/652/" title="使用" class="tag" target="_blank">使用</a> </nav> <div class="alert alert-secondary alert-dismissible fade show font-weight-light" role="alert"> <i class="bi bi-info-square-fill"></i> 声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉：<a class="badge badge-dark font-weight-light" data-toggle="modal" data-target="#exampleModal" data-whatever="@mdo" href="#"><i class="bi bi-envelope"></i> 投诉/举报</a> 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。 <button type="button" class="close" data-dismiss="alert" aria-label="Close"> <span aria-hidden="true">×</span> </button> </div> <div class="alert alert-secondary alert-dismissible fade show" role="alert"> <form name='feedback' method='post' enctype='multipart/form-data' action='#'> <input name='enews' type='hidden' value='AddFeedback'> <input type="hidden" name="bid" value="2" /> <input type='hidden' name='ecmsfrom' value='https://www.ouer.net/daima/75rs.html'> <div class="mb-3"> <label for="address"><h6><strong><i class="bi bi-chat-right-text-fill"></i> 看完仍有疑问？有类似问题直接问程序猿</strong></h6></label> <textarea class="form-control" aria-label="With textarea" name="saytext" style="height: 120px;" required placeholder="提问和评论都可以，用心的回复会被更多人看到和认可！"></textarea> </div> <button class="btn btn-primary btn-lg btn-block" type='submit' name='submit'>发布提问</button></form> <button type="button" class="close" data-dismiss="alert" aria-label="Close"> <span aria-hidden="true">×</span> </button> </div> <div class="modal fade" id="exampleModal" tabindex="-1" aria-labelledby="exampleModalLabel" aria-hidden="true"> <div class="modal-dialog"> <div class="modal-content"> <div class="modal-header"> <h5 class="modal-title" id="exampleModalLabel">投诉/举报</h5> <button type="button" class="close" data-dismiss="modal" aria-label="Close"> <span aria-hidden="true">×</span> </button> </div> <div class="modal-body"> <form name='feedback' method='post' enctype='multipart/form-data' action='../../e/enews/index.php'> <input name='enews' type='hidden' value='AddFeedback'> <input type="hidden" name="bid" value="2" /> <input type='hidden' name='ecmsfrom' value='https://www.u72.net/daima/75rs.html'> <input name='title' type='hidden' value='https://www.u72.net/daima/75rs.html'> <div class="mb-3"> <div class="input-group"> <div class="input-group-prepend"> <span class="input-group-text"><i class="bi bi-pencil-square"></i></span> </div> <input class="form-control" type="text" placeholder="Scrapy爬虫 -- 03..." readonly> </div> </div> <div class="mb-3"> <label for="username"><h5><i class="bi bi-person-lines-fill"></i> 您的姓名</h5></label> <div class="input-group"> <div class="input-group-prepend"> <span class="input-group-text"><i class="bi bi-person-check"></i></span> </div> <input class="form-control form-control" type="text" name="name"> </div> </div> <div class="mb-3"> <label for="address"><h5><i class="bi bi-chat-left-text"></i> 反馈内容</h5></label> <textarea class="form-control form-control" aria-label="With textarea" name="saytext" style="height: 130px;" required></textarea> </div> <div class="modal-footer"> <button type="button" class="btn btn-secondary" data-dismiss="modal">关闭</button> <button type="submit" name="submit" class="btn btn-primary">提交内容</button> </div> </form> </div> </div> </div> </div> <p class="social-share" data-disabled="google,twitter,facebook,tencent,diandian" ></p> <p class="text-left"><button type="button" class="itemCopy btn btn-light" id="TKLS" type="button" data-clipboard-text="https://www.u72.net/daima/75rs.html"><i class="bi bi-link-45deg"></i> https://www.u72.net/daima/75rs.html</button></p> </div> <br> <nav class="blog-pagination"> <p class="text-left"></p> </nav> </div> <aside class="col-md-4 blog-sidebar"> <div class="mb-3 bg-light rounded list-group"> <a href="#" class="list-group-item list-group-item-action active"> <div class="d-flex w-100 justify-content-between"> <h5 class="mb-1"><i class="bi bi-building"></i> 相关代码解决方案</h5> </div> </a> <a href="/daima/0r0r.html" title="Scrapy 爬虫" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> Scrapy 爬虫</h6></div></a><a href="/daima/6axv.html" title="爬虫--scrapy" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫--scrapy</h6></div></a><a href="/daima/nn17d.html" title="爬虫scrapy" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫scrapy</h6></div></a><a href="/daima/72wb.html" title="Scrapy爬虫 -- 02" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> Scrapy爬虫 -- 02</h6></div></a><a href="/daima/7ud3.html" title="Scrapy爬虫 -- 01" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> Scrapy爬虫 -- 01</h6></div></a><a href="/daima/nz8h7.html" title="scrapy爬虫框架" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> scrapy爬虫框架</h6></div></a><a href="/daima/ncsfe.html" title="Scrapy爬虫requests" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> Scrapy爬虫requests</h6></div></a><a href="/daima/nr5a9.html" title="python爬虫—使用scrapy爬虫框架" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> python爬虫—使用scrapy爬虫框架</h6></div></a><a href="/daima/z11d.html" title="爬虫：Scrapy1" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫：Scrapy1</h6></div></a><a href="/daima/fdf9.html" title="网页爬虫--scrapy入门" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 网页爬虫--scrapy入门</h6></div></a><a href="/daima/3xwd.html" title="爬虫框架--scrapy简介" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫框架--scrapy简介</h6></div></a><a href="/daima/28u3.html" title="Python——爬虫3（Scrapy）" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> Python——爬虫3（Scrapy）</h6></div></a><a href="/daima/7094.html" title="爬虫框架scrapy安装" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫框架scrapy安装</h6></div></a><a href="/daima/nkhav.html" title="Scrapy 豆瓣电影爬虫" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> Scrapy 豆瓣电影爬虫</h6></div></a><a href="/daima/ndx1d.html" title="爬虫框架scrapy安装" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫框架scrapy安装</h6></div></a><a href="/daima/nbam6.html" title="python scrapy爬虫入门" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> python scrapy爬虫入门</h6></div></a><a href="/daima/z2ab.html" title="爬虫：Scrapy2 - Items" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫：Scrapy2 - Items</h6></div></a><a href="/daima/hkd6.html" title="爬虫：Scrapy4 - Spiders" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫：Scrapy4 - Spiders</h6></div></a><a href="/daima/hmmz.html" title="python scrapy 简单的爬虫" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> python scrapy 简单的爬虫</h6></div></a><a href="/daima/h7n3.html" title="爬虫：Scrapy11 - Logging" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫：Scrapy11 - Logging</h6></div></a><a href="/daima/0xke.html" title="爬虫的入门以及scrapy" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫的入门以及scrapy</h6></div></a><a href="/daima/5294.html" title="爬虫框架Scrapy之详解" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫框架Scrapy之详解</h6></div></a><a href="/daima/53nf.html" title="爬虫框架Scrapy之CrawlSpiders" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫框架Scrapy之CrawlSpiders</h6></div></a><a href="/daima/53n8.html" title="爬虫框架Scrapy之Spider" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫框架Scrapy之Spider</h6></div></a><a href="/daima/nkrha.html" title="windows下Scrapy爬虫框架" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> windows下Scrapy爬虫框架</h6></div></a><a href="/daima/nz6wn.html" title="python爬虫之scrapy框架" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> python爬虫之scrapy框架</h6></div></a><a href="/daima/hc0u.html" title="爬虫：Scrapy8 - Item Pipeline" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫：Scrapy8 - Item Pipeline</h6></div></a><a href="/daima/kzz8.html" title="网络爬虫（蜘蛛）Scrapy,Python安装！" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 网络爬虫（蜘蛛）Scrapy,Python安装！</h6></div></a><a href="/daima/hev7.html" title="爬虫：Scrapy13 - 发送 email" class="list-group-item list-group-item-action" target="_blank"><div class="d-flex w-100 justify-content-between"><h6 class="mb-1"><img class="icon" src="/skin/law/images/icon-answer.svg" width="22" height="22"> 爬虫：Scrapy13 - 发送 email</h6></div></a></div> <div class="p-4 mb-3 bg-light rounded"> <h4>当日更新</h4> <ul class="list-group list-group-flush mb-0"><li class="list-group"><a href="/daima/nx9n1.html" title="怎样在log4j.xml配置文件中引入变量：小公司经验较多的我和阿里UC等大公司经验较多的Boss，一些技术交流和探讨" target="_blank"><i class="bi bi-question-square-fill"></i> 怎样在log4j.xml配置文件中</a></li> <li class="list-group"><a href="/daima/nx9n0.html" title="《Visual c++2013入门经典（第7版）》上机练习：为 Win32 控制台应用程序创建项目" target="_blank"><i class="bi bi-question-square-fill"></i> 《Visual c++2013入门经典（第</a></li> <li class="list-group"><a href="/daima/nx9nx.html" title="spring boot web项目在IDEA下热部署解决办法（四步搞定）" target="_blank"><i class="bi bi-question-square-fill"></i> spring boot web项目在IDEA</a></li> <li class="list-group"><a href="/daima/nx9nw.html" title="分布式缓存系统Memcached(九)——状态机之socket连接与派发" target="_blank"><i class="bi bi-question-square-fill"></i> 分布式缓存系统Memcached(九</a></li> <li class="list-group"><a href="/daima/nx9nv.html" title="MVC Filter" target="_blank"><i class="bi bi-question-square-fill"></i> MVC Filter</a></li> <li class="list-group"><a href="/daima/nx9nu.html" title="会话EJB系列(六)依赖注入" target="_blank"><i class="bi bi-question-square-fill"></i> 会话EJB系列(六)依赖注入</a></li> <li class="list-group"><a href="/daima/nx9ns.html" title="OpenCV3编程入门读书笔记2-core组件" target="_blank"><i class="bi bi-question-square-fill"></i> OpenCV3编程入门读书笔记2-c</a></li> <li class="list-group"><a href="/daima/nx9nr.html" title="开源Bug管理系统Redmine安装和使用心得" target="_blank"><i class="bi bi-question-square-fill"></i> 开源Bug管理系统Redmine安装</a></li> <li class="list-group"><a href="/daima/nx9nc.html" title="快速幂" target="_blank"><i class="bi bi-question-square-fill"></i> 快速幂</a></li> <li class="list-group"><a href="/daima/nx9nf.html" title="网站学习的些许感想" target="_blank"><i class="bi bi-question-square-fill"></i> 网站学习的些许感想</a></li> </ul> </div> </aside> </div> <div class="floatbar"> <div class="floatbar-item"> <a href="/fankui.html" target="_blank" class="floatbar-btn"> <i> <i class="bi bi-chat-right-text"></i> </i> <p> 联系<br> 我们 </p> </a> </div> <div class="floatbar-item floatbtn-item-top" style="display: none;"> <a href="javascript:" class="floatbar-btn backtotop"> <i> <i class="bi bi-chevron-double-up"></i> </i> <p> 回到<br> 顶部 </p> </a> </div> </div> <header> </header> </main> <!-- /.container --> <script src="/skin/law/top/js/toastr.min.js"></script> <script src="/skin/law/top/js/site.js"></script> <div class="container"> <nav class="navbar navbar-expand-lg navbar-light bg-light"> <a class="navbar-brand" href="#">友情链接：</a> <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false" aria-label="Toggle navigation"> <span class="navbar-toggler-icon"></span> </button> <div class="collapse navbar-collapse" id="navbarNavAltMarkup"> <div class="navbar-nav"> <a class="nav-link" href="https://www.xuebavip.cn">学霸VIP</a> </div> </div> </nav> </div> <div id="loginbox"></div> <div class="container"> <footer class="pt-4 my-md-3 pt-md-5 border-top text-center font-weight-light"> <p class="mb-1"><a href="/about/nbs.html" class="text-dark">关于我们</a> / <a href="/about/nbv.html" class="text-dark">广告服务</a> / <a href="/about/nbu.html" class="text-dark">免责声明</a></p> <p class="mb-1">若内容有误或涉及侵权不想在本站出现！</p> <p class="mb-1"><a class="badge badge-dark font-weight-light" href="/fankui.html"><i class="bi bi-envelope"></i> 请联系我们</a> 我们会及时处理和回复！</p> <p class="mb-1">Copyright © 2022 程序员工具箱 All Rights Reserved <a target="_blank" href="https://beian.miit.gov.cn" rel="nofollow" class="text-dark">蜀ICP备14004987号</a> </p> </footer> </div> <script src="/skin/law/bootstrap4.6.1/js/clipboard.min.js"></script> <script> var clipboard = new Clipboard('.itemCopy'); clipboard.on('success', function(e) { if (e.trigger.disabled == false || e.trigger.disabled == undefined) { e.trigger.innerHTML = "本问题链接复制成功"; e.trigger.disabled = true; setTimeout(function() { e.trigger.innerHTML = "本问题链接一键复制"; e.trigger.disabled = false; }, 2000); } }); clipboard.on('error', function(e) { e.trigger.innerHTML = "复制失败"; }); </script> <script src="/e/extend/DoTimeRepage/"></script> <script> var _hmt = _hmt || []; (function() { var hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?f2555e186a2326f28ced45c12daae1cb"; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); })(); </script> </body> </html>