 |
Thursday, December 19, 2002 |
|
Robot Noise: I'm getting ready to start the Robot Noise project as a way of trying to determine some of the technical aspects of how Web traffic is generated. I looked at the logs of 3 currently low-activity Web servers, and discovered that they had been hit only a couple times by search engines in the last 4 days. By comparison, www.gulker.com received some 2000 visits from robots in the same time period. So more active and more linked-to sites do indeed generate more crawls.
'Bots amounted to 20% to 27% of visits... lower on days when there was much linking and lots of human visitors... which leads me to suspect that 'bots are roughly constant and humans go up and down as whim and linkage dictate. Interesting that Hotbot crawls much more frequently than Googlebot... next step is to setup sites with different link and activity levels, and see how it affects crawl traffic.... |
Updated 4/16/04; 1:51:00 PM
|
Chris Gulker'sUpdated 4/16/04; 1:51:00 PM
| December 2002 | ||||||
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| 8 | 9 | 10 | 11 | 12 | 13 | 14 |
| 15 | 16 | 17 | 18 | 19 | 20 | 21 |
| 22 | 23 | 24 | 25 | 26 | 27 | 28 |
| 29 | 30 | 31 | ||||
| Nov Feb | ||||||
AlwaysOn Network
Columns (soon)
Dotcom Garden
Lone Genius Hackers
Picture Weblog
Theory & Strategy
Weblogging
gulker.com Cam
Interesting blogs et al.:
Natalie d'Arbeloff
Azeem Azhar
Ken Bereskin
Blogging Ecosysytem
Blogging Network
BlogStreet
Boing Boing
Tim Bray
Matt Croydon
DaveNet
Rael Dornfest
Esther Dyson
Dave Farber's IP
Dave Fitch
David Galbraith
John Getze
William Gibson
Dan Gillmor
James Gleick
Bernie Goldbach
Meg Hourihan
Joi Ito
Xeni Jardin
Jeff Jarvis
Linux Journal
Mitch Kapor
Kuro5hin
Gunnar Langemark
Joshua Levy
Scott Loftesness
Macintouch
Ross Mayfield
Hans Moravec
Rafe Needleman
Nonsense Verse
OS Opinion
Tim Porter
Recommended Reading
Reverse Cowgirl
Glenn Reynolds
Roger Ridey
Phil Ringnalda
John Robb
Scott Rosenberg
Anita Rowland
Brent Simmons
Robert Scoble
Doc Searls
Jessica Shea
Gavin Sheridan
Shifted Librarian
Stefan Smalla
Bruce Sterling
Scripting News
Slashdot
Dan Shafer
John Tringham
Jon Udell
Moicho Umeda
Philipp Weltentummler
Kevin Werbach
Amy Wohl
