FreeOZ论坛

标题: [转帖] 世界著名搜索公司的爬虫清单 [打印本页]

作者: NEWGAY 时间: 7-9-2008 18:22
标题: [转帖] 世界著名搜索公司的爬虫清单
世界著名搜索公司的爬虫清单
AbachoBOT=Abacho.com
abcdatos_botlink=Abcdatos.com
http://www.abcdatos.com/botlink/=Abcdatos.com
AESOP_com_SpiderMan=Aesop.com
ah-ha.com crawler (crawler@ah-ha.com)=ah-ha.com
ia_archiver=Archive.org
Scooter=Altavista.com
Mercator=Altavista.com
Scooter2_Mercator_3-1.0=Altavista.com
roach.smo.av.com-1.0=Altavista.com
Tv_Merc_resh_26_1_D-1.0=Altavista.com
AltaVista-Intranet=Altavista.co.uk
jan.gelin@av.com=Altavista.co.uk
FAST-WebCrawler=alltheweb.com
crawler@fast.no=alltheweb.com
Acoon Robot=acoon.de
antibot=antisearch.net
Atomz=atomz.com
Buscaplus Robi=buscaplus.com
CanSeek/=canseek.ca
support@canseek.ca=canseek.ca
ChristCRAWLER=christcrawler.com
Crawler=crawler.de
admin@crawler.de=crawler.de
DaAdLe.com ROBOT/=daadle.com
RaBot=daum.net
Agent-admin/=daum.net
phortse@hanmail.net=daum.net
contact/jylee@kies.co.kr=kies.co.kr
DeepIndex=deepindex.com
DittoSpyder=ditto.com
Jack=domanova.co.uk
Speedy Spider=entireweb.com
ArchitextSpider=excite.com
ArchitectSpider=excite.com
Arachnoidea=euroseek.net
arachnoidea@euroseek.net=euroseek.net
EZResult=ezresults.com
Fast PartnerSite Crawler=fastsearch.net
FAST Data Search Crawler=fastsearch.net
KIT-Fireball=fireball.de
FyberSearch=fybersearch.com
GalaxyBot=galaxy.com
geckobot=geckobot.com
GenCrawler=gendoor.com
GeonaBot=geona.com
Googlebot=Google.com
googlebot@googlebot.com=Google.com
google=Google.com
moget/2.0=goo.ne.jp
moget@goo.ne.jp=goo.ne.jp
Aranha=girafa.com
Slurp.so/1.0=Yahoo
slurp@inktomi.com=Yahoo
Slurp/2.0j=Yahoo
www.inktomisearch.com=Yahoo
Slurp/2.0-KiteHourly=Yahoo
Slurp/2.0-OwlWeekly=Yahoo
spider@aeneid.com=Yahoo
Slurp/3.0-AU=Yahoo
Toutatis 2.5-2=hoppa.com
Hubater=hubat.com
IlTrovatore-Setaccio=iltrovatore.it
IncyWincy=incywincy.com
UltraSeek=infoseek.com
InfoSeek Sidewinder=infoseek.com
Mole2/1.0=intags.de
webmaster@intags.de=intags.de
MP3Bot=mp3bot.de
C-PBWF-ip3000.com-crawler=ip3000.com
ip3000.com-crawler=ip3000.com
kuloko-bot/0.2=kuloko.com
LNSpiderguy=lexis-nexis.com
NetResearchServer=look.com
MantraAgent=looksmart.com
NetResearchServer=loopimprovements.com
Lycos_Spider_(T-Rex)=lycos.com
JoocerBot=joocer.com
HenryTheMiragoRobot=mirago.co.uk
mozDex/=mozdex.com
MSNBOT/0.1=MSN
Gulliver=northernlight.com
ObjectsSearch/0.01=objectssearch.com
PicoSearch/=picosearch.com
PJspider=portaljuice.com
DIIbot=powerinter.net
nttdirectory_robot=navi.ocn.ne.jp
super-robot@super.navi.ocn.ne.jp=navi.ocn.ne.jp
griffon=super.navi.ocn.ne.jp
griffon@super.navi.ocn.ne.jp=super.navi.ocn.ne.jp
Spider/maxbot.com=maxbot.com
admin@maxbot.com=maxbot.com
gazz/1.0=Unknown Spider
gazz@nttrd.com=Unknown Spider
NationalDirectory-SuperSpider=nationaldirectory.com
dloader(NaverRobot)/=naver.com
dumrobo(NaverRobot)/=naver.com
Openfind piranha=openfind.com
Shark=openfind.com
robot-response@openfind.com.tw=openfind.com.tw
Openbot/=openfind.com.tw
psbot=picsearch.org
CrawlerBoy=pinpoint.com
ip3000.com=petersnews.com
AlkalineBOT=AlkalineBOT
Fluffy the spider=searchhippo.com
info@searchhippo.com=searchhippo.com
Scrubby/=scrubtheweb.com
asterias=singingfish.com
speedfind ramBot xtreme=speedfind.de
Kototoi/0.1=s.u-tokyo.ac.jp
Searchspider/=searchspider.com
SightQuestBot/=sightquest.com
Spider_Monkey/=spidermonkey.ca
Surfnomore Spider v1.1=surfnomore.com
Robot@SuperSnooper.Com=supersnooper.com
teoma_agent1=teoma.com
teoma_admin@hawkholdings.com=teoma.com
Teradex_Mapper=mapper.teradex.com
mapper@teradex.com=mapper.teradex.com
ESISmartSpider=travel-finder.com
Spider TraficDublu=traficdublu.ro
Tutorial Crawler=tutorgig.com
UK Searcher Spider=uksearcher.co.uk
Vivante Link Checker=vivante.com
appie=walhello.com
Nazilla=websmostlinked.com
www.WebWombat.com.au=webwombat.com.au
marvin/infoseek=webseek.de
marvin-team@webseek.de=webseek.de
MuscatFerret=webtop.com
WhizBang! Lab=whizbanglabs.com
ZyBorg=wisenut.com
WIRE WebRefiner=wire.co.uk
WSCbot=worldsearchcenter.com
Yandex=yandex.com
Yellopet-Spider=yellowpet.com
Iron33=verno.ueda.info.waseda.ac.jp/
ALink=Link Checkers
AMeta=Link Checker
ASPSearch URL Checker=Link Checker
BlogBot=Link Checker
BMChecker=Link Checker
Bookmark Buddy=Link Checker
Check&Get=Link Checker
CheckWeb=Link Checker
CNET_Snoop=Link Checker
CSE HTML Validator=Link Checker
DRKSpider=Link Checker
DISCo Watchman=Link Checker
DoctorHTML=Link Checker
Email Extractor=Email Extractor
EmailSiphon=Email Extractor
EmailWolf=Email Extractor
FavOrg=Link Checker
Favorites Sweeper=Link Checker
FreshLinks.exe=Link Checker
Funnel Web Profiler=Link Checker
Html Link Validator=Link Checker
The Informant=Link Checker
The Intraformant=Link Checker
InternetLinkAgent=Link Checker
InternetPeriscope=Link Checker
javElink=Link Checker
jdwhatsnew.cgi=Link Checker
JRTS Check Favorites Utility=Link Checker
Lambda LinkCheck=Link Checker
LinkLint-checkonly=Link Checker
LinkAlarm=Link Checker
Linkbot=Link Checker
Linkman=Link Checker
LinkProver=Link Checker
Links=Link Checker
LinkScan Server=Link Checker
LinkSweeper=Link Checker
Link Valet Online=Link Checker
LinkVerify Spider=Link Checker
LinkWalker=Link Checker
Morning Paper=Link Checker
MoveAnnouncer=Link Checker
NetLookout=Link Checker
NetMechanic=Link Checker
www.elsop.com=Link Checker
NetMind-Minder=Link Checker
NetMonitor=Link Checker
Netprospector JavaCrawler=Link Checker
online link validator=Link Checker
Rational SiteCheck=Link Checker
Robozilla=Link Checker
RPT-HTTPClient=Link Checker
SurfMaster=Link Checker
SyncIT=Link Checker
Watchfire WebXM=Link Checker
WatzNew Agent=Link Checker
WebSite-Watcher=Link Checker
WebTrends Link Analyzer=Link Checker
Weblink Scanner=Link Checker
Xenu's Link Sleuth=Link Checker
W3C_Validator=Link Validator
WDG_Validator/=Link Validator
Tooter=Link Validator
citenikbot/=citenik.co.uk
CLIPS-index=clips-index.imag.fr/
Computer_and_Automation_Research_Institute_Crawler=Research Bot
cosmos=xyleme.com
robot@xyleme.com=xyleme.com
DiaGem/=DiaGem
Digimarc WebReader=digimarc.com
EchO!/2.0=voila.com
FinaleRobot=expressus.com
robot-master@expressus.com=expressus.com
Ideare - SignSite=ideare.com
GentleSpider=research.att.com
Gulper Web Bot=Gulper Web Bot
larbin=Unknown Spider
sebastien.ailleret@inria.fr=inria.fr
ghi@lcs.mit.edu=Unknown Spider
MultiText=MultiText
NEC Research Agent=NEC Research Agent
OntoSpider=OntoSpider
sherlock_spider=sherlock.com.cn
Steeler=Steeler
ru-robot=rutgers.edu
0.1_hseo(at)cs.rutgers.edu=rutgers.edu
WebGather=WebGather
xyro=xyro
xcrawler@inria.fr=Unknown Spider
Zao/0.2=Zao
ADSARobot=ADSARobot
AnswerChase=AnswerChase
ASPSeek=ASPSeek
AVSearch=AVSearch
Checkbot=Checkbot
DaviesBot=DaviesBot
deepweb=deepweb.com
GigaBaz=brainbot.com
GigaBazVStheWeb=brainbot.com
crawler@brainbot.com=brainbot.com
Giskard=oralco.com
InternetSeer=InternetSeer
ipiumBot=ipiumBot
InsumaScout=InsumaScout
Katriona=Katriona
LEIA=LEIA
LexiBot=lexibot.com
metabot=metabot
NetCruiser=NetCruiser
NPBot=nameprotect.com
NetZippy=NetZippy
NZBot=navigationzone.com
Opencola=opencola.com
Oxxbot1=Oxxbot
Pansophica=Pansophica
Phoaks=Phoaks
PICgrabber=PICgrabber
PictureOfInternet=PictureOfInternet
erik@malfunction.org=Unknown Spider
PintaSpider=PintaSpider
PolyBot=PolyBot
Squid=Squid
Sqworm=Sqworm
TaWWWantula=TaWWWantula
TeraCrawl=TeraCrawl
TurnitinBot=turnitin.com
UCmore=ucmore.com
UdmSearch=mnoGoSearch
unlostBot=unlost.com
URLBlaze=urlblaze.net
UrlScope=UrlScope
Vagabondo=Vagabondo
vspider=vspider
WAVETools=WAVETools
Webbandit=Webbandit
Webclipping.com=Webclipping.com
webcollage=webcollage
WebCompass=WebCompass
WebGenie=WebGenie
Web Magnet=Unknown Spider
WebMiner=Unknown Spider
Webpush=Unknown Spider
WebSymmetrix=Unknown Spider
webrank=Unknown Spider
webwasher=Unknown Spider
WhosTalking=Unknown Spider
AnzwersCrawl/2.0=Anzwers
fido/1.0 Harvest/1.4.pl2=Planet Search
GAIS Robot/1.0B2=seednet
Googlebot/1.0=Google.com
Gulliver/1.2=Northern Light
Infoseek Sidewinder/0.9=Infoseek
KIT_Fireball/2.0=Fireball
lwp-trivial/1.27=Search 4 Free
Lycos_Spider_(T-Rex)/3.0=Lycos
Scooter/1.0=AltaVista
Scooter/1.0 scooter@pa.dec.com=AltaVista
Scooter/1.1 (custom)=AltaVista
Scooter/2.0 G.R.A.B. X2.0=AltaVista
Scooter/2.0 G.R.A.B. V1.1.0=AltaVista
search.at V1.2=search.at
inktomi=Inktomi Spider
SwissSearch V1.2=SwissSearch
The Informant=The Informant
Ultraseek=Infoseek
WebCrawler/3.0 Robot libwww/5.0a=WebCrawler
WebCrawler-AddURL/2.0=WebCrawler
WiseWire=WiseWire
WiseWire-Alpha-1.0=WiseWire
WiseWire-Alpha-Spider=WiseWire
WiseWire-Alpha12-Spider971219a=WiseWire
WiseWire-Alpha12-Spider(971223a)=WiseWire
WiseWire-HotSpider-1.0=WiseWire
WiseWire-Spider=WiseWire
WiseWire-Spider-1.0=WiseWire
WiseWire-Spider2=WiseWire
WiseWire-Widow-1.0=WiseWire
WiseWire-Widow-1.0r=WiseWire
WiseWire-Widow-1.0-ALPHA12=WiseWire
CherryPickerSE/1.0=Email Extractor
CherryPickerElite/1.0=Email Extractor
Crescent Internet ToolPak HTTP OLE Control v.1.0=Email Extractor
EmailCollector/1.0=Email Extractor
EmailWolf 1.00=Email Extractor
ExtractorPro=Email Extractor
ask jeeves=Ask Jeeves
lycos=Lycos.com
whatuseek=What You Seek
wisenutbot=Looksmart
msnbot=MSN
GigaBlast=Gigablast
Gigabot=Gigablast
archive_org=Archive.org
jeeves=Ask Jeeves
Asterias=Singingfish Spider
Slurp=Inktomi Spider
ZyBorg=LookSmart Bot
baiduspider=Baidu

作者: coredump 时间: 8-9-2008 01:19
网上居然这么多蜘蛛啊

作者: decisiontree 时间: 8-9-2008 01:54
爬虫的原理都大同小异.

搜完以后怎么组织存储,怎么更新比较复杂.

作者: ubuntuhk 时间: 8-9-2008 02:12
有些爬虫很没流氓，如腾讯的爬虫，多个进程不停地抓取内容，被很多论坛/网站ban掉，也在FreeOZ的黑名单中

Google的爬虫比较文明，遵守robots.txt的规矩，还设有webmaster tools让管理员查看自己网站的爬行记录，那些出错，出的什么错，一目了然

作者: decisiontree 时间: 8-9-2008 10:16

原帖由 ubuntuhk 于 8-9-2008 01:12 发表

 登录/注册后可看大图

有些爬虫很没流氓，如腾讯的爬虫，多个进程不停地抓取内容，被很多论坛/网站ban掉，也在FreeOZ的黑名单中

Google的爬虫比较文明，遵守robots.txt的规矩，还设有webmaster tools让管理员查看自己网站的爬行记 ...

我自己没试过,不过实验室的一个家伙曾经开着CROWLER然后人闪了, 结果后来收到几十封COMPLAIN邮件

,被无数网站列入黑名单.很多网站是更新很慢的,不需要那么频繁的扫.

作者: uspepper 时间: 9-9-2008 10:44
流着汗问一句：什么是爬虫？
一开始我还以为是生物意义上的爬虫

作者: decisiontree 时间: 10-9-2008 18:58
爬虫说白了就是一个程序。他的功能是在网上自动搜索和保存网页信息（因为它是自动工作的而且在不同的页面间跑来跑去，颇像个虫子。）。GOOGLE有世界上最大的COLLECTION靠的就是爬虫的辛勤工作。当然你是感觉不到它的，因为你GOOGLE的时候，是针对爬虫收集来的信息（已经被整理，和压缩过了）。

欢迎光临 FreeOZ论坛 (https://www.freeoz.org/bbs/)