📦 fffonion / httplib2-exp

Some httplib2 scripts; using modified httplib2->https://github.com/fffonion/httplib2-plus

★ 8 stars ⑂ 4 forks 👁 8 watching

📥 Clone https://github.com/fffonion/httplib2-exp.git

HTTPS git clone https://github.com/fffonion/httplib2-exp.git

SSH git clone git@github.com:fffonion/httplib2-exp.git

CLI gh repo clone fffonion/httplib2-exp

ZIP Desktop VS Code

fffonion add few scripts 3cc014e 11 years ago 📝 History

📂 master View all commits →

📁 wordpress_bomb

📁 youiv

📄 _headers.py

📄 .gitmodules

📄 17ttt.com.py

📄 400gblist.py

📄 aspx_crawler.py

📄 baidu_qianxi.py

📄 count-torrents.py

📄 cproxy_appid.txt

📄 cproxy_crawler.py

📄 cproxy_paas.txt

📄 desktopsky_get.py

📄 gdajie.py

📄 gdajie.zip

📄 general_make.zip

📄 get_torrent.py

📄 hack.httpjiyhfvsd.2288.org.py

📄 hack.icbcaan.py

📄 hack.ziajz.ml.py

📄 hehe.txt

📄 httplib2

📄 HUST-WIRELESS_urllib2.py

📄 HUST-WIRELESS.py

📄 index.html

📄 itellyou_links.txt

📄 itellyoucn.py

📄 mnks.txt

📄 pcgames.py

📄 qq2013marathon.py

📄 query_4px.py

📄 README.md

📄 targ.txt

📄 test_jxyz.py

📄 wow.py

📄 youiv_crawler.py

📄 youiv.bat

📄 README.md

就……自从知道了httplib2之后抓站什么的基本都用httplib2了

urllib2不能通过head指定编码（so不能gzip,要自己做handler）

必须手动维护一个连接池才能Connection:keep-alive

以上两者（特别是后者）在处理大量请求的时候速度会很慢，而且，不怎么像正常的浏览器~ o(￣▽￣)o

之前用urllib2抓一个壁纸站的时候被IDC花现封ip了，呵呵0.0

另外，官方的httplib2有些缺陷

检查代理的时候没有扫描注册表(urllib, urllib2都是扫描的)

没有chunk_read功能，so无法实现下载进度

基于以上两点改了个httplib2plus

以上~ ＞▽＜