User-agent:
для всех роботов
AltaVista
Google
MSN
Yandex
ABCdatos BotLink
Acme.Spider
Ahoy! The Homepage Finder
Alkaline
Anthill
Walhello appie
Arachnophilia
Arale
Araneo
AraybOt
ArchitextSpider
Aretha
ARIADNE
arks
ASpider (Associative Spider)
ATN Worldwide
Atomz.com Search Robot
AURESYS
BackRub
unnamed
BBot
Big Brother
Bjaaland
BlackWidow
Die Blinde Kuh
Bloodhound
Borg-Bot
BoxSeaBot
bright.net caching robot
BSpider
CACTVS Chemistry Spider
Calif
Cassandra
Digimarc Marcspider/CGI
Checkbot
ChristCrawler.com
churl
cIeNcIaFiCcIoN.nEt
CMC/0.01
Collective
Combine System
ConfuzzledBot
CoolBot
Web Core / Roots
XYLEME Robot
Internet Cruiser Robot
Cusco
CyberSpyder Link Test
CydralSpider
Desert Realm Spider
DeWeb(c) Katalog/Index
DienstSpider
Digger
Digital Integrity Robot
Direct Hit Grabber
DNAbot
DownLoad Express
DragonBot
DWCP (Dridus' Web Cataloging Project)
e-collector
EbiNess
EIT Link Verifier Robot
ELFINBOT
Emacs-w3 Search Engine
ananzi
esculapio
Esther
Evliya Celebi
nzexplorer
FastCrawler
Fluid Dynamics Search Engine robot
Felix IDE
Wild Ferret Web Hopper #1, #2, #3
FetchRover
fido
Hдmдhдkki
KIT-Fireball
Fish search
Fouineur
Robot Francoroute
Freecrawl
FunnelWeb
gammaSpider, FocusedCrawler
gazz
GCreep
GetBot
GetURL
Golem
Googlebot
Grapnel/0.01 Experiment
Griffon
Gromit
Northern Light Gulliver
Gulper Bot
HamBot
Harvest
havIndex
HI (HTML Index) Search
Hometown Spider Pro
Wired Digital
ht
//Dig
HTMLgobble
Hyper-Decontextualizer
iajaBot
IBM_Planetwide
Popular Iconoclast
Ingrid
Imagelock
IncyWincy
Informant
InfoSeek Robot 1.0
Infoseek Sidewinder
InfoSpiders
Inspector Web
IntelliAgent
I, Robot
Iron33
Israeli-search
JavaBee
JBot Java Web Robot
JCrawler
AskJeeves
JoBo Java Web Robot
Jobot
JoeBot
The Jubii Indexing Robot
JumpStation
image.kapsi.net
Katipo
KDD-Explorer
Kilroy
KO_Yappo_Robot
LabelGrabber
larbin
legs
Link Validator
LinkScan
LinkWalker
Lockon
logo.gif Crawler
Lycos
Mac WWWWorm
Magpie
marvin/infoseek
Mattie
MediaFox
MerzScope
NEC-MeshExplorer
MindCrawler
mnoGoSearch search engine software
moget
MOMspider
Monster
Motor
MSNBot
Muncher
Muninn
Muscat Ferret
Mwd.Search
Internet Shinchakubin
NDSpider
NetCarta WebMap Engine
NetMechanic
NetScoop
newscan-online
NHSE Web Forager
Nomad
The NorthStar Robot
ObjectsSearch
Occam
HKU WWW Octopus
OntoSpider
Openfind data gatherer
Orb Search
Pack Rat
PageBoy
ParaSite
Patric
pegasus
The Peregrinator
PerlCrawler 1.0
Phantom
PhpDig
PiltdownMan
Pimptrain.com's robot
Pioneer
html_analyzer
Portal Juice Spider
PGP Key Agent
PlumtreeWebAccessor
Poppi
PortalB Spider
psbot
GetterroboPlus Puu
The Python Robot
Raven Search
RBSE Spider
Resume Robot
RoadHouse Crawling System
RixBot
Road Runner
The ImageScape Robot
Robbie the Robot
ComputingSite Robi/1.0
RoboCrawl Spider
RoboFox
Robozilla
Roverbot
RuLeS
SafetyNet Robot
Scooter
Search.Aus-AU.COM
Sleek
SearchProcess
Senrigan
SG-Scout
ShagSeeker
Shai'Hulud
Sift
Simmany Robot Ver1.0
Site Valet
SiteTech-Rover
Skymob.com
SLCrawler
Inktomi Slurp
Smart Spider
Snooper
Solbot
Speedy Spider
spider_monkey
SpiderBot
Spiderline Crawler
SpiderMan
SpiderView(tm)
Spry Wizard Robot
Site Searcher
Suke
suntek search engine
Sven
Sygol
TACH Black Widow
Tarantula
tarspider
Tcl W3 Robot
TechBOT
Templeton
TitIn
TITAN
The TkWWW Robot
TLSpider
UCSD Crawl
UdmSearch
UptimeBot
URL Check
URL Spider Pro
Valkyrie
Verticrawl
Victoria
vision-search
void-bot
Voyager
VWbot
The NWI Robot
W3M2
WallPaper (alias crawlpaper)
the World Wide Web Wanderer
w@pSpider by wap4.com
WebBandit Web Spider
WebCatcher
WebCopy
webfetcher
The Webfoot Robot
Webinator
weblayers
WebLinker
WebMirror
The Web Moose
WebQuest
Digimarc MarcSpider
WebReaper
webs
Websnarf
WebSpider
WebVac
webwalk
WebWalker
WebWatch
Wget
whatUseek Winona
WhoWhere Robot
Weblog Monitor
w3mir
WebStolperer
The Web Wombat
The World Wide Web Worm
WWWC Ver 0.2.5
WebZinger
XGET
Nederland.zoek
Disallow:
индексировать всё
/
/..
/cgi-bin
/phpincludes
/css
/js
/images
/mail
Ваш файл Robots.TXT
Что должен робот сделать на странице:
проиндексировать эту страницу (если не часто обновляется)
прослеживать гиперссылки на странице (советуем выбрать)
Ваш тег МЕТА:
выделить