首页 > 代码库 > 【scrapy】创建第一个项目
【scrapy】创建第一个项目
1)创建项目命令:
scrapy startproject tutorial
该命令将在当前目录下创建tutorial文件夹
2)定义Item
Items are containers that will be loaded with the scraped data;They are declared by creating a scrapy.Item class and defining its attibutes as scrapy.Field objects.
import scrapyclass DmozItem(scrapy.Item): title=scrapy.Field() link=scrapy.Field() desc=scrapy.Field()
3)定义spider
To create a spider,you must subclass scrapy.Spider and define the three main mandatory attributes:
name:identifies the spiders
start_urls:a list of urls where the spider will begin to crawl from.
parse():a method of the spider,which will be called with the downloaded Response object of each start url.The response is passed to the method as the first and only argument.
The parse() methods is in charge of processing the response and returning scraped data(as Item object) and more urls to follow(as Request object).
import scrapyclass DmozSpider(scrapy.Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response): filename = response.url.split("/")[-2] with open(filename, ‘wb‘) as f: f.write(response.body)
4)爬取
命令:scrapy crawl dmoz
5)storing the scraped data
命令:scrapy crawl dmoz -o items.json
that will generate a items.json file containing all scraped items,serialized in JSON
If you want to perform more complex things with the scraped items,you can write an Item Pipeline.
------------------------------------------------------------------------------------------------------------------------------------
Introduction to Selectors
There are several ways to extract data from web pages.Scrapy uses a mechanism based on XPath or CSS expressions called Scrapy Selectors.
You can see selectors as objects that represent nodes in the document structure.
Selectors have four basic methods:
xpath():returns a list of selectors
css():returns a list of selectors
extract():returns a unicode string with the selected data
re():returns a list of Unicode strings extracted by applying the regular expression given as argument.
Trying Selectors in the Shell:
Start a shell:
scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"
After the shell loads,you will have the response fetched in a local response variable,so if you type response.body() you will see the body of the response,or you can type response.headers to see its headers.