You'd have to go to the site's Web server access logs to be sure. But, on gulker.com where I can read the server logs as well as referrers etc., I see that robots make up between 10% and 28% of traffic - the percentage is highest on the slowest days. (The Summary stats package I use lumps all spiders under a 'Known Robot' category).

That suggests that there is a relatively constant level of 'robot noise', which makes sense, because the 'bots have algorithms that regulate crawling, and they're unlikely to get excited by nude photos or radical technologies or links from Slashdot.

People are much more mecurial, and jump on things that catch their interest or are posted on influential sites. So one explanation of my 'standalone' and 'embedded' Web sites is that the former is getting mainly human traffic, and the latter is mostly a robot-visited site.

A correlation really needs to be done of the content of the two types of site and the type of referrer. A quick glance shows this isn't easy... sites that don't update often are hit wildly (celebrity) and daily-updated sites show the bot-only formula, and vice versa. Good thing I have the 2nd installment to re-think this...

The great thing about the Web, when it works, is that you can post an idea that is still less-than-fully-formed, and some smart person will help you move it along...
Comments 6:51:15 PM    

Chris Gulker's view from Silicon Valley - in words and pictures