Saturday, November 10, 2001
A google spidersIn checking the log files for this site I've notived that Google has finally found it and has spent the past few days spidering through it.
There are a few thousand links for it to follow (out of what? A million potential URLs on this site? I know the Electric King James has over fifteen million URLs). For instance, there are three just for the years, 12 each for each year (okay, so there's only 11 for this year, but close enough) so that's now 39 URLs. Each day (for those days that have an entry) have at least one entry and while I may have skipped a day or two here and there, let's say there's an averave of 300 per year, so that's over 900 there. And if you assume an average of two entries per day (remember, you can retrieve the entire day, or just an entry) that's another 600 per year or 1,800 so we're now up to nearly 3,000 URLs that Google has to crawl through (with lots of duplication).
robots.txt for bible.conman.org
#----------------------------- # Go away---we don't want you # to endlessly spider this # site. #----------------------------- User-agent: * Disallow: /
There's a reason I don't allow web robots/spiders to the Electric King James—it would take way too long to index the site (if indeed, the spider in question was even aware of all the possible URLs) and my machine isn't all that powerful to begin with (it being a 33MHz 486 and all). But I feel that there is a research problem lurking here that some interprising Masters or Ph.D. candidate could tackle: how best to spider a site that allows multiple views per document.