首页 > 代码库 > 关于python基础认证（用于爬虫）
关于python基础认证（用于爬虫）
转自http://www.voidspace.org.uk/python/articles/authentication.shtml
先转来，由于是python2的版本之后会翻译文章以及移植到python3
Introduction
This tutorial aims to explain and illustrate what basic authentication is, and how to deal with it from Python. You can download the code from this tutorial from the Voidspace Python Recipebook.
The first example, So Let‘s Do It, shows how to do it manually. This illustrates how authentication works.
The second example, Doing it Properly, shows how to handle it automatically - with a handler.
These examples make use of the Python module urllib2. This provides a simple interface to fetching pages across the internet, the urlopen function. It provides a more complex interface to specialised situations in the form of openers and handlers. These are often confusing to even intermediate level programmers. For a good introduction to using urllib2, read my urllib2 tutorial.
Basic Authentication
There is a system for requiring a username/password before a client can visit a webpage. This is called authentication and is implemented on the server. It allows a whole set of pages (called a realm) to be protected by authentication.
This scheme (or schemes) are defined by the HTTP spec, and so whilst python supports authentication it doesn‘t document it very well. HTTP documentation is in the form of RFCs [1] which are technical documents and so not the most readable 
.
The two normal [2] authentication schemes are basic and digest authentication. Between these two, basic is overwhelmingly the most common. As you might guess, it is also the simpler of the two.
A summary of basic authentication goes like this :
- client makes a request for a webpage
- server responds with an error, requesting authentication
- client retries request - with authentication details encoded in request
- server checks details and sends the page requested, or another error
The following sections covers these steps in more details.
Making a Request
A client is any program that makes requests over the internet. It could be a browser - or it could be a python program. When a client asks for a web page, it is sending a request to a server. The request is made up of headers with information about the request. These are the ‘http request headers‘.
Getting A Response
When the request reaches the server it sends a response back. The request may still fail (the page may not be found for example), but the response will still contain headers from the server. These are ‘http response headers‘.
If there is a problem then this response will include an error code that describes the problem. You will already be familiar with some of these codes - 404 : Page not found, 500 : Internal Server Error, etc. If this happens; an exception [3] will be raised by urllib2, and it will have a ‘code‘ attribute. The code attribute is an integer that corresponds to the http error code [4].
Error 401 and realms
If a page requires authentication then the error code is 401. Included in the response headers will be a ‘WWW-authenticate‘ header. This tells us the authentication scheme the server is using for this page and also something called a realm. It is rarely just a single page that is protected by authentication but a section - a ‘realm‘ of a website. The name of the realm is included in this header line.
The ‘WWW-Authenticate‘ header line looks like WWW-Authenticate: SCHEME realm="REALM".
For example, if you try to access the popular website admin application cPanel your browser will be sent a header that looks like : WWW-Authenticate: Basic realm="cPanel"
If the client already knows the username/password for this realm then it can encode them into the request headers and try again. If the username/password combination are correct, then the request will succeed as normal. If the client doesn‘t know the username/password it should ask the user. This means that if you enter a protected ‘realm‘ the client effectively has to request each page twice. The first time it will get an error code and be told what realm it is attempting to access - the client can then get the right username/password for that realm (on that server) and repeat the request.
HTTP is a ‘stateless‘ protocol. This means that a server using basic authentication won‘t ‘remember‘ you are logged in [5] and will need to be sent the right header for every protected page you attempt to access.
First Example
Suppose we attempt to fetch a webpage protected by basic authentication. :
req = urllib2.Request(theurl)
try:
handle = urllib2.urlopen(req)
except IOError, e:
if hasattr(e, ‘code‘):
if e.code != 401:
print ‘We got another error‘
print e.code
else:
print e.headers
print e.headers[‘www-authenticate‘]
Note
If the exception has a ‘code‘ attribute it also has an attribute called ‘headers‘. This is a dictionary like object with all the headers in - but you can also print it to display all the headers. See the last line that displays the ‘www-authenticate‘ header line which ought to be present whenever you get a 401 error.
A typical output from above example looks like :
WWW-Authenticate: Basic realm="cPanel"Connection: closeSet-Cookie: cprelogin=no; path=/Server: cpsrvd/9.4.2Content-type: text/htmlBasic realm="cPanel"
You can see the authentication scheme and the ‘realm‘ part of the ‘www-authenticate‘ header. Assuming you know the username and password you can then navigate around that website - whenever you get a 401 error with the same realm you can just encode the username/password into your request headers and your request should succeed.
The Username/Password
Lets assume you need to access pages which are all in the same realm. Assuming you have got the username and password from the user, you can extract the realm from the header. Then whenever you get a 401 error in the same realm you know the username and password to use. So the only detail left, is knowing how to encode the username/password into the request header 
. This is done by encoding it as a base 64 string. It doesn‘t actually look like clear text - but it is only the most vaguest of ‘encryption‘. This means basic authentication is just that - basic. Anyone sniffing your traffic who sees an authentication request header will be able to extract your username and password from it. Many websites like yahoo or ebay, use javascript hashing/encryption and other tricks to authenticate a login. This is much harder to detect and mimic from python ! You may need to use a proxy client server and see what information your browser is actually sending to the website [6].
base64
There is a very simple recipe base64 recipe over on the Activestate Python Cookbook (It‘s actually in the comments of that page). It shows how to encode a username/password into a request header. It goes like this :
import base64base64string = base64.encodestring(‘%s:%s‘ % (username, password))[:-1]req.add_header("Authorization", "Basic %s" % base64string)Where req is our request object like in the first example.
So Let‘s Do It
Let‘s wrap all this up with an example that shows accessing a page, extracting the realm, then doing the authentication. We‘ll use a regular expression to pull the scheme and realm out of the response header. :
import sys
import re
import base64
from urlparse import urlparse
theurl = ‘http://www.someserver.com/somepath/somepage.html‘
# if you want to run this example you‘ll need to supply
# a protected page with your username and password
username = ‘johnny‘
password = ‘XXXXXX‘ # a very bad password
req = urllib2.Request(theurl)
try:
handle = urllib2.urlopen(req)
except IOError, e:
# here we *want* to fail
pass
else:
# If we don‘t fail then the page isn‘t protected
print "This page isn‘t protected by authentication."
sys.exit(1)
if not hasattr(e, ‘code‘) or e.code != 401:
# we got an error - but not a 401 error
print "This page isn‘t protected by authentication."
print ‘But we failed for another reason.‘
sys.exit(1)
authline = e.headers[‘www-authenticate‘]
# this gets the www-authenticate line from the headers
# which has the authentication scheme and realm in it
authobj = re.compile(
r‘‘‘(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=[‘"]([^‘"]+)[‘"]‘‘‘,
re.IGNORECASE)
# this regular expression is used to extract scheme and realm
matchobj = authobj.match(authline)
if not matchobj:
# if the authline isn‘t matched by the regular expression
# then something is wrong
print ‘The authentication header is badly formed.‘
print authline
sys.exit(1)
scheme = matchobj.group(1)
realm = matchobj.group(2)
# here we‘ve extracted the scheme
# and the realm from the header
if scheme.lower() != ‘basic‘:
print ‘This example only works with BASIC authentication.‘
sys.exit(1)
base64string = base64.encodestring(
‘%s:%s‘ % (username, password))[:-1]
authheader = "Basic %s" % base64string
req.add_header("Authorization", authheader)
try:
handle = urllib2.urlopen(req)
except IOError, e:
# here we shouldn‘t fail if the username/password is right
print "It looks like the username or password is wrong."
sys.exit(1)
thepage = handle.read()
When the code has run the contents of the page we‘ve fetched is saved as a string in the variable ‘thepage‘. The regular expression used to match the authentication header in this example is r‘‘‘(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=[‘"]([^‘"]+)[‘"]‘‘‘. This doesn‘t work where there is a space in the realm, which you can fix by replacing \w+ with [^‘"]+. This gives us the regular expression:
r‘‘‘(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=[‘"]([^‘"]+)[‘"]‘‘‘
Warning
If you are writing an http client of any sort that has to deal with basic authentication, don‘t do it this way. The next example that shows using a handler is the right way of doing it.
Doing it Properly
In actual fact the proper way to do BASIC authentication with Python is to install an opener that uses an authentication handler. The authentication handler needs a passowrd manager - and then you‘re away 
.
Every time you use urlopen you are using handlers to deal with your request - whether you know it or not. The default opener has handlers for all the standard situations installed [7]. What we need to do is create an opener that has a handler that can deal with basic authentication. The right handler for our needs is calledurllib2.HTTPBasicAuthHandler. As I mentioned it also needs a password manager -urllib2.HTTPPasswordMgr.
Unfortunately our friend HTTPPasswordMgr has a slight problem - you must already know the realm you‘re fetching. Luckily it has a near cousin HTTPPasswordMgrWithDefaultRealm. Despite the keyboard busting name, it‘s a bit more friendly to use. If you don‘t know the name of the realm - then pass in None for the realm, and it will try the username and password you give it - whatever the realm. Seeing as you are going to specify a specific URL, it is likely that this will be sufficient. If you aren‘t convinced then you can always use HTTPPasswordMgr and extract the realm from the authentication header the first time you meet it.
This example goes through the following steps :
- establishes the top level url, username and password
- Create our password manager (with default realm)
- Gives the password to the manager
- Creates the handler with the manager
- Creates an opener with the handler installed
At this point we have a choice. We can either use the open method of the opener directly. This leavesurllib2.urlopen using the default opener. Alternatively we can make our opener the default one. This means all future calles to urlopen will use this opener. As all openers have the default handlers installed as well as the ones you pass it, it shouldn‘t break urlopen to do this. In the example below we install it, making it the default opener :
theurl = ‘http://www.someserver.com/toplevelurl/somepage.htm‘
username = ‘johnny‘
password = ‘XXXXXX‘
# a great password
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
# this creates a password manager
passman.add_password(None, theurl, username, password)
# because we have put None at the start it will always
# use this username/password combination for urls
# for which `theurl` is a super-url
authhandler = urllib2.HTTPBasicAuthHandler(passman)
# create the AuthHandler
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
# All calls to urllib2.urlopen will now use our handler
# Make sure not to include the protocol in with the URL, or
# HTTPPasswordMgrWithDefaultRealm will be very confused.
# You must (of course) use it when fetching the page though.
pagehandle = urllib2.urlopen(theurl)
# authentication is now handled automatically for us
Hurrah - not so bad hey 
.
A Word About Cookies
Some websites may also use cookies alongside authentication. Luckily there is a library that will allow you to have automatic cookie management without having to think about it. This is ClientCookie. In Python 2.4 it becomes part of the python standard library as cookielib. See my article on cookielib - for an example of how to use it.
Footnotes
| [1] | http://www.faqs.org/rfcs/rfc2617.html is the RFC that describes basic and digest authentication |
| [2] | There is also a M$ proprietary authentication scheme called NTLM, but it‘s usually found on intranets - I‘ve never had to deal with it live on the web. |
| [3] | An HTTPError, which is a subclass of IOError |
| [4] | Or at least state management is a separate subject. Using cookies the server may well have details of your session - but you will still need to authenticate each request. |
| [5] | See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for a full list of error codes |
| [6] | See this comp.lang.python thread for suggestions of several proxy servers that can do this. |
| [7] | See the urllib2 tutorial for a slightly more detailed discussion of openers and handlers. |