14 lines
389 B
Plaintext
Executable File
14 lines
389 B
Plaintext
Executable File
User-Agent: *
|
|
Disallow: /directory.bml
|
|
|
|
#
|
|
# Blocked journals aren't listed here because robots.txt files
|
|
# can't be above 50k or so, depending on the spider.
|
|
#
|
|
# Instead, blocked journals have HTML inserted in them which
|
|
# should prevent behaved spiders from indexing it.
|
|
#
|
|
# Note that http://username.livejournal.com journals have an
|
|
# autogenerated robots.txt, since it can be small.
|
|
#
|