ljr/livejournal/htdocs/robots.txt

14 lines
389 B
Plaintext
Raw Permalink Normal View History

2019-02-05 21:49:12 +00:00
User-Agent: *
Disallow: /directory.bml
#
# Blocked journals aren't listed here because robots.txt files
# can't be above 50k or so, depending on the spider.
#
# Instead, blocked journals have HTML inserted in them which
# should prevent behaved spiders from indexing it.
#
# Note that http://username.livejournal.com journals have an
# autogenerated robots.txt, since it can be small.
#