29 lines
629 B
Plaintext
Executable File
29 lines
629 B
Plaintext
Executable File
User-agent: *
|
|
Disallow: /directory
|
|
Disallow: /interests
|
|
Disallow: /tools/tell
|
|
Disallow: /tools/memadd
|
|
Disallow: /tools/search.bml
|
|
Disallow: /friends/
|
|
Disallow: /interface/
|
|
Disallow: /translate/
|
|
Disallow: /comments/
|
|
Disallow: /numreplies/
|
|
Disallow: /users/imp_
|
|
Disallow: /userinfo.bml?user=imp_
|
|
Disallow: /talk
|
|
Disallow: /stats/stats.txt
|
|
Disallow: /create
|
|
#
|
|
Crawl-delay: 1
|
|
|
|
Host: lj.rossia.org
|
|
|
|
#
|
|
# Blocked journals aren't listed here because robots.txt files
|
|
# can't be above 50k or so, depending on the spider.
|
|
#
|
|
# Instead, blocked journals have HTML inserted in them which
|
|
# should prevent behaved spiders from indexing it.
|
|
#
|