首页 > 代码库 > 如何抓取开了gzip的网页
如何抓取开了gzip的网页
有时候用 file_get_contents() 函数抓取网页会发生乱码现象。有两个原因会导致乱码,一个是编码问题,一个是目标页面开了Gzip。
编码问题好办,把抓取到的内容转下编码即可($content=iconv("GBK", "UTF-8//IGNORE", $content);),我们这里讨论的是如何抓取开了Gzip的页面。怎么判断呢?获取的头部当中有Content-Encoding: gzip说明内容是GZIP压缩的。用FireBug看一下就知道页面开了gzip没有。下面是用firebug查看我的博客的头信息,Gzip是开了的。繁峙县粮食局
view source print?
1 | 请求头信息原始头信息 |
2 | Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 |
3 | Accept-Encoding gzip, deflate |
4 | Accept-Language zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 |
5 | Connection keep-alive |
6 | Cookie __utma=225240837.787252530.1317310581.1335406161.1335411401.1537; __utmz=225240837.1326850415.887.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=%E4%BB%BB%E4%BD%95%E9%A1%B9%E7%9B%AE%E9%83%BD%E4%B8%8D%E4%BC%9A%E9%82%A3%E4%B9%88%E7%AE%80%E5%8D%95%20site%3Awww.nowamagic.net; PHPSESSID=888mj4425p8s0m7s0frre3ovc7; __utmc=225240837; __utmb=225240837.1.10.1335411401 |
7 | Host www.nowamagic.net |
8 | User-Agent Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0 |
下面介绍一些解决方案:
1. 使用自带的zlib库
如果服务器已经装了zlib库,用下面的代码可以轻易解决乱码问题。
view source print?
1 | $data = file_get_contents ( "compress.zlib://" . $url ); |
2. 使用CURL代替file_get_contents
view source print?
1 | function curl_get( $url , $gzip =false){ |
2 | $curl = curl_init( $url ); |
3 | curl_setopt( $curl , CURLOPT_RETURNTRANSFER, 1); |
4 | curl_setopt( $curl , CURLOPT_CONNECTTIMEOUT, 10); |
5 | if ( $gzip ) curl_setopt( $curl , CURLOPT_ENCODING, "gzip" ); // 关键在这里 |
6 | $content = curl_exec( $curl ); |
7 | curl_close( $curl ); |
8 | return $content ; |
9 | } |
3. 使用gzip解压函数
view source print?
001 | function gzdecode( $data ) { |
002 | $len = strlen ( $data ); |
003 | if ( $len < 18 || strcmp ( substr ( $data ,0,2), "\x1f\x8b" )) { |
004 | return null; // Not GZIP format (See RFC 1952) |
005 | } |
006 | $method = ord( substr ( $data ,2,1)); // Compression method |
007 | $flags = ord( substr ( $data ,3,1)); // Flags |
008 | if ( $flags & 31 != $flags ) { |
009 | // Reserved bits are set -- NOT ALLOWED by RFC 1952 |
010 | return null; |
011 | } |
012 | // NOTE: $mtime may be negative (PHP integer limitations) |
013 | $mtime = unpack( "V" , substr ( $data ,4,4)); |
014 | $mtime = $mtime [1]; |
015 | $xfl = substr ( $data ,8,1); |
016 | $os = substr ( $data ,8,1); |
017 | $headerlen = 10; |
018 | $extralen = 0; |
019 | $extra = "" ; |
020 | if ( $flags & 4) { |
021 | // 2-byte length prefixed EXTRA data in header |
022 | if ( $len - $headerlen - 2 < 8) { |
023 | return false; // Invalid format |
024 | } |
025 | $extralen = unpack( "v" , substr ( $data ,8,2)); |
026 | $extralen = $extralen [1]; |
027 | if ( $len - $headerlen - 2 - $extralen < 8) { |
028 | return false; // Invalid format |
029 | } |
030 | $extra = substr ( $data ,10, $extralen ); |
031 | $headerlen += 2 + $extralen ; |
032 | } |
033 |
034 | $filenamelen = 0; |
035 | $filename = "" ; |
036 | if ( $flags & 8) { |
037 | // C-style string file NAME data in header |
038 | if ( $len - $headerlen - 1 < 8) { |
039 | return false; // Invalid format |
040 | } |
041 | $filenamelen = strpos ( substr ( $data ,8+ $extralen ), chr (0)); |
042 | if ( $filenamelen === false || $len - $headerlen - $filenamelen - 1 < 8) { |
043 | return false; // Invalid format |
044 | } |
045 | $filename = substr ( $data , $headerlen , $filenamelen ); |
046 | $headerlen += $filenamelen + 1; |
047 | } |
048 |
049 | $commentlen = 0; |
050 | $comment = "" ; |
051 | if ( $flags & 16) { |
052 | // C-style string COMMENT data in header |
053 | if ( $len - $headerlen - 1 < 8) { |
054 | return false; // Invalid format |
055 | } |
056 | $commentlen = strpos ( substr ( $data ,8+ $extralen + $filenamelen ), chr (0)); |
057 | if ( $commentlen === false || $len - $headerlen - $commentlen - 1 < 8) { |
058 | return false; // Invalid header format |
059 | } |
060 | $comment = substr ( $data , $headerlen , $commentlen ); |
061 | $headerlen += $commentlen + 1; |
062 | } |
063 |
064 | $headercrc = "" ; |
065 | if ( $flags & 1) { |
066 | // 2-bytes (lowest order) of CRC32 on header present |
067 | if ( $len - $headerlen - 2 < 8) { |
068 | return false; // Invalid format |
069 | } |
070 | $calccrc = crc32( substr ( $data ,0, $headerlen )) & 0xffff; |
071 | $headercrc = unpack( "v" , substr ( $data , $headerlen ,2)); |
072 | $headercrc = $headercrc [1]; |
073 | if ( $headercrc != $calccrc ) { |
074 | return false; // Bad header CRC |
075 | } |
076 | $headerlen += 2; |
077 | } |
078 |
079 | // GZIP FOOTER - These be negative due to PHP‘s limitations |
080 | $datacrc = unpack( "V" , substr ( $data ,-8,4)); |
081 | $datacrc = $datacrc [1]; |
082 | $isize = unpack( "V" , substr ( $data ,-4)); |
083 | $isize = $isize [1]; |
084 |
085 | // Perform the decompression: |
086 | $bodylen = $len - $headerlen -8; |
087 | if ( $bodylen < 1) { |
088 | // This should never happen - IMPLEMENTATION BUG! |
089 | return null; |
090 | } |
091 | $body = substr ( $data , $headerlen , $bodylen ); |
092 | $data = "" ; |
093 | if ( $bodylen > 0) { |
094 | switch ( $method ) { |
095 | case 8: |
096 | // Currently the only supported compression method: |
097 | $data = gzinflate( $body ); |
098 | break ; |
099 | default : |
100 | // Unknown compression method |
101 | return false; |
102 | } |
103 | } else { |
104 | // I‘m not sure if zero-byte body content is allowed. |
105 | // Allow it for now... Do nothing... |
106 | } |
107 |
108 | // Verifiy decompressed size and CRC32: |
109 | // NOTE: This may fail with large data sizes depending on how |
110 | // PHP‘s integer limitations affect strlen() since $isize |
111 | // may be negative for large sizes. |
112 | if ( $isize != strlen ( $data ) || crc32( $data ) != $datacrc ) { |
113 | // Bad format! Length or CRC doesn‘t match! |
114 | return false; |
115 | } |
116 | return $data ; |
117 | } |
使用:
view source print?
1 | $html = file_get_contents ( ‘http://www.nowamagic.net/librarys/veda/‘ ); |
2 | $html =gzdecode( $html ); |
就介绍这三个方法,应该能解决大部分gzip引起的抓取乱码问题了。
如何抓取开了gzip的网页
声明:以上内容来自用户投稿及互联网公开渠道收集整理发布,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任,若内容有误或涉及侵权可进行投诉: 投诉/举报 工作人员会在5个工作日内联系你,一经查实,本站将立刻删除涉嫌侵权内容。