Chris Gulker: Cyveillancebot

Thu, 05 Jun 2003 17:19:27 GMT

Brian Murray, Cyveillance VP of client services called this morning. We had a long, amicable chat. Brian made a couple points that I think are fair to point out:

- his firm does not currently work for RIAA

- the 'bot is not meant to harass sites that publish opinions that its clients may not favor. They are looking at ways to ameliorate the 'bot's 'DoS mode'

- their 'bots crawl either randomly or in an A to Z fashion, not in response to postings

- Cyveillance does not store or distribute materials downloaded from Web sites, except for materials that belong to their clients

I expressed my concerns that the behavior of the technology and their public messages are out of sync. It seems to me that a firm that wanted to be a good net citizen would fix the 'bot, observe robots.txt and otherwise be straightforward and forthcoming, insofar as that is consistent with their mission. Brian made reasonable responses, but said he would leave it to others to decide whether circumventing robots.txt amounts to 'circumventing a protection mechanism' per the DMCA.

Lastly, I asked what would happen if their detection mechanism misfired, and included my content along with some of the creepy stuff they sift through. I was concerned that a bug in their software would land my essays and articles in some sort of 'bad content line-up' that had the potential to sully my good name (such as it is). His response was that, while no one can guarantee that software won't misfire, everything that Cyveillance software flags is checked by humans (unlike RIAA's last batch of Cease-and-Desist orders). Brian was concerned that I had published his email. BTW my email to him was a public response to his public note on the spamcop-help lis. We are now in sync about expectations. I may go drop in on these guys in D.C. later on this year...

Wed, 04 Jun 2003 23:37:47 GMT

My note back to Brian at Cyveillance:

Brian,

Thank you very much for responding. I think the issue is that your 'bot assumes a wide pipe on the server: for those of us with meager resources, it's a nuisance. We get particularly cranky when the 'bot downloads a directory of 5 years worth of copyrighted columns, which it has done many hundreds of times now.

Which brings me to my other 2 concerns, and thank you for suggesting I bring them up directly with you:

Is Cyveillance not concerned that you are in at least technical violation of the DMCA when your 'bot ignores robots.txt? Can't this be construed as circumventing a protection mechanism?

Lastly, what do you do with my copyrighted materials? Your site suggests that you provide Internet content to clients (part of your 'early warning system'?) for a fee, which, in the case of my stuff, is pretty clearly a copyright violation, lacking prior authorization. Giving your clients a pointer is one thing, actually giving them the material is quite another.

Anyway, just curious...

Best, thanks again for looking into the bandwidth issue...

Chris

Somebody's got to ask... are the rules the same for corporations and individual citizens?

Wed, 04 Jun 2003 23:09:10 GMT

Cyveillance's Brian Murray responds:

Hi Chris,

Thank you very much for bringing this to my attention. I will speak with our IT folks tomorrow and ask them to investigate and also to exclude your domain from future crawls, just to be sure. We try very hard to prevent this type of thing from happening. It is true that our technology retrieves data by creating a single connection to your Web site and then downloading html files across this one connection, but you should know that this is done with the intention of eliminating the work of constantly building and dropping connections. It is also meant to minimize the impact on other users, though it leaves a distinct imprint on typical log files, especially if the server was not loaded at the time, allowing it to fill these serial requests very quickly. Since we do not download images-typically the largest files on a site-minimal bandwidth is required. We believe this low impact, high visibility technique is the most responsible technique for us to use under the circumstances. We are also working on ways to further minimize the impact. Based on your message, it appears the approach may not have worked as intended, and I will be sure to look into it. Hopefully, the data you have provided will help us identify what happened here.

Thanks again, and please feel free to contact me directly should you have any further issues.

Sincerely,

Brian Murray
Vice President of Client Services
Cyveillance, Inc.

Now I just have 2 more questions, Brian...

Wed, 04 Jun 2003 19:14:51 GMT

"Army looking for a few good blogs?" asks "Roger", who spots some unusual visitors to his blog, one of which has also been to gulker.com. It would seem that something called Land Information Warfare Activity, US Army Intelligence and Security Command (INSCOM), Fort Belvoir, Virginia came in on a Google search on 'Cyveillance'. Land Information Warfare Activity? Well, I guess they got a browserfull of my opinion about the quality of Cyveillance technology...

Wed, 04 Jun 2003 16:07:18 GMT

Cyveillance, redux: a number of readers have pointed to this comment by Brian Murray, VP of client services at Cyveillance:

http://news.spamcop.net/pipermail/spamcop-help/2003-June/034004.html

Brian writes:

"In terms of the other concerns expressed in this Forum relating to how Cyveillance gathers information from the Internet, we set the highest standards for our online activities... and we take great pains to ensure that our crawlers minimize the load on other sites’ servers"

My response:

Brian-

I read your comments on 'setting high standards' and 'taking pains to minimize your crawlers' load on servers' with great interest.

If the 'bot that usually identifies itself as some flavor of IE operating on IP addresses 63.148.99.xxx is indeed yours (as is widely reported on the Net), I would be grateful for your comments on the behavior I and others have noted in our server's logs. Here's a GREP of my Apache Web server's access log, Dec. 15 to present:

http://www.gulker.com/music_industry/63_148_99_log.txt

Please note that the 'bot in question connects repeatedly to long directories and downloads files sequentially without pausing - sometimes more than a hundred in a row - as fast as my relatively modest 144K net connection will allow. A number of other Webmasters have written me with similar experiences.

While this 'bot is connected, my server is all but inaccessible to others, and we at gulker.com are unable to access external sites easily. This 'bot is not well-behaved: it also ignores robots.txt.

So is it yours? If so, when will you apply your stated policy, and fix the darn thing?

Chris Gulker

http://www.gulker.com/

PS. This is probably redundant, since your firm specializes in knowing what's happening on the Net, but there is a category, complete with RSS feed, of information about the behavior of this 'bot:

http://www.gulker.com/categories/cyveillancebot/

Here's an essay I wrote describing my experience with this creature:

http://www.gulker.com/stories/2003/05/06/whatToThinkAboutCyveillanc

And the column I wrote for London's Independent 2 weeks ago:

http://news.independent.co.uk/digital/features/story.jsp?story=408191

And the article about same on Slashdot:

http://yro.slashdot.org/article.pl?sid=03/05/07/0120237

Awaiting a response with great interest...

Wed, 28 May 2003 02:47:23 GMT

"The business of popular music, today, is now, in some peculiarly new way, entirely about promotion. William Gibson, in a speech to the Director's Guild of America...

Wed, 21 May 2003 00:06:12 GMT

More 'bots o' interest: Blog Pulse, Breaking Blogs, IBM Almaden Research Center, Kototoi Zao, DeepIndex, NCSA VIAS, BlogBot... It's a zoo out there...

Tue, 20 May 2003 23:18:26 GMT

Interesting 'bots: Grub is a distributed bot, whose crawlers run as a background process on many machines; TurnItIn indexes your pages so teachers can see if students are plagiarizing them, Nutch is an Open Source 'bot. They're not all idiots...

Tue, 20 May 2003 20:45:10 GMT

RIAA spider 'tarpit' randomly generates hundreds of 'sites' just loaded with 'mp3' files, the better to keep stupid robot (and human) behavior in check. Complete with source code...

Sun, 18 May 2003 06:30:15 GMT

Darn, I thought Cyveillance had taken me off their list... but their idiot bot's been back, and woe be to those who try to access this site while Cyveillancebot is crawling the !@#$%!@ calendar links (of all things) you see on the right side of this page.

And, oh, yeah, we can't use our net connection while their stupid bot is hammering us, either. I can't believe a responsible company (they have good investors) lets technology that is this poor loose on the Web. You're embarrassing yourselves... and the people who think highly enough of you to actually pay your bills...

Thu, 15 May 2003 17:13:50 GMT

I'm a geek, I read referrer logs, along with my copy of the NY Times in the morning. And maybe that data point explains why Cyveillancebot hasn't visited ever since an apparent real, live human at Cyveillance read this Weblog a couple days ago.

Why would that cause the apparent (and welcome) cessation of Cyveillancebot activity? Well, one theory would be that Cyveillance is aware that they are in the business of, technically at least, infringing copyright, and are staying low, now that they know that I know.

One of the things I know, is that that access logs show a different pattern if a page is opened from my server, and if a copy of that page is opened from a file saved to a hard drive. So here's what it looks like when the page is opened from the server:

63.148.99.229 - - [13/May/2003:12:24:28 -0700] "GET / HTTP/1.0" 304 - "-" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

63.148.99.229 - - [13/May/2003:12:24:28 -0700] "GET /graphics/logo_blu_bg_shado_116.png HTTP/1.0" 304 - "http://www.gulker.com/" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

[snip - the full log sequence is here - snip]

63.148.99.229 - - [13/May/2003:12:24:29 -0700] "GET /graphics/right_bg.jpg HTTP/1.0" 304 - "http://www.gulker.com/" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

The first item in the sequence is a request for "/", the short way to request gulker.com's default home page, and all the rest are requests for the graphical bits and pieces that comprise the page. So what happens if someone saves my page to their hard drive, and then opens it in a browser?

What happens is that you get the same sequence in the access log, with one exception: there is no request for "/" - the browser already has it, it only needs the graphics and other 'furniture' to draw the page.

In a recent 2-day period, I noticed that the graphics-only sequence was requested 554 more times than the full sequence. So, more than 250 times a day, a browser somewhere in the world was pulling my page from a local file, rather than from my server.

One reason that this would happen is that the browser has cached my page, but not the graphics files associated with it. I've done some experiments with this (Mozilla, IE, Safari), and the behavior depends on the type of browser and how its cache prefs are set. I'm sure that some of the time, particular browser versions, set in just the right fashion, are causing this behavior, when people come back to my page before it's expired from their cache.

But there is another reason this could happen. If someone were to download my page, store it on their own hard drive or local server, and then open it from that server, you see the same sequence - no "/".

You might see this if, for example, Cyveillance - who have pulled more than a thousand files from my server including essays, articles, research, presos etc. without ever (until the Tuesday human visit) downloading the attendant graphics files - were to post my files to their internal, private network, and was allowing access to them by employees and clients.

If that's what they're doing, I think that is copyright infringement - many of the files they have pulled (and continue to pull down, over and over again) are copyrighted. It seems to me that if Cyveillance were to post something like "This imbecile is spouting anti-DCMA blasphemy at http://www.gulker.com/ " on their private or public servers, that would probably not be a copyright violation - their clients would be reading my opinions from my server. In this case, they are being paid to find inimical opinion on the Web.

But if they place my original work on their server, and then distribute it to the clients who pay them (large) fees, that probably is a copyright violation - they are being paid for distributing my copyrighted material without permission, which, of course, is exactly what they and their clients object to so strenuously. So, next step is a little detective work to figure out who owns the IP addresses that are pulling my stuff in this fashion... 63.148.99.229, BTW, is registered to Cyveillance according to arin.net, and is almost certainly one of their firewall machines...

Tue, 13 May 2003 22:01:08 GMT

RIAA now admits to sending dozens of erroneous cease-and-desist notices. "The errors represent a black eye for the RIAA's latest efforts against piracy, which rely on automated crawlers to scour the Internet in an attempt to find material that is being distributed in a way that violates federal copyright law." Ahem.. this was predicted here...

Tue, 13 May 2003 21:13:53 GMT

Cyveillance humans have arrived... for the first time, requests incoming from the Cyveillance IP block are pulling down graphics files linked to pages... a sign that a human-operated browser rather than a bot is looking. Enjoy the content™... and please respect the copyrights...

Tue, 13 May 2003 19:20:03 GMT

My 'interesting referrer' comes from AOL: "Roger" looked it up. Hmmmm... could it be? AOL-TW is well acquainted with copyright. They charge for their service, and they frequently cache content, including copyrighted material owned by others, on their servers to minimize bandwidth costs... so the claim could be made that they are selling copyrighted materials to others. Shows how loopy old-school copyright is in a networked world when an ISP is guilty of copyright infringement. So the referrer does indeed have a future with RIAA... heck, they are RIAA...

Tue, 13 May 2003 17:16:57 GMT

More referrer log amusement: this one from someone who followed links in robots.txt intended to catch bad 'bots:

172.174.227.122 - - [11/May/2003:01:46:34 -0700] "GET ***** HTTP/1.0" 200 4695 "-" "By Allowing Me Access You Waive All Rules Associated With It."

A unilateral, after-the-fact contract... Someone has a future with RIAA... You could probably write a script that would put the lines of a haiku or poem in successive referrers... I want to set my referrer to "All your Web are belong to us"...

Tue, 13 May 2003 17:02:24 GMT

Americans enjoy a political democracy, but not an economic democracy. Big law firms and corporations know this: they are well aware that almost no one has the resources - time and money - to prevail against a wealthy corporation and its phalanx of lawyers. Erin Brockovich notwithstanding, it is far more likely that corporations, not individuals will prevail in civil law matters.

Cyveillance, a company that is reported to be an agent of the RIAA, MPAA and dozens of large corporations with a mission to protect their copyrighted materials has downloaded more than a thousand files from my Web site in the last 5 months. Many hundreds of these files are copyrighted original essays, reports and presentations.

Responsible, law abiding publishers and corporations normally pay me from a few hundred to several thousand dollars to use the materials I have developed to formulate strategy and tactics, gauge markets, etc.

Cyveillance' Web site offers a fee-based service providing companies with information gleaned from the Web. So, since www.gulker.com is so amazingly popular with them, it is reasonable to suspect that they are providing my copyrighted materials (along with god knows how many others) to corporations for fees that are reported to be hundreds of thousands of dollars a year.

True, I place some of these files on the Web site so others can read them: it is my hope that they will spur debate and commentary, and advance the cause and utility of a global network. I would be delighted if Hilary Rosen logged on, read my work, and it helped her to find a way to serve, rather than sue, her customers.

However, I do not authorize anyone to resell my copyrighted materials for a profit without my permission. It is unreasonable, unethical and illegal for others to do this. The way in which my copyrighted material is delivered to RIAA executives or anybody else is important.

No one makes this point more loudly than RIAA and its lawyers: they recently sued 4 students for billions of dollars for distributing what they claimed were their members' copyrighted materials for free on private networks. Yet, here is at least one of their agents doing exactly the same thing (Cyveillance has a private network for its clients) for profit.

Sure I can sue. That involves paying an attorney or team of attorneys several hundred dollars an hour for god knows how long - likely months or years. RIAA, while arguably just as guilty as kids in college dorms, would likely easily outlast my resources.

The 'lawyer door' to RIAA's castle is thick, heavy and defended by large and hideous brutes. However, that doesn't mean that there is no recourse. RIAA's members are publicly-traded corporations whose executives are (or should be) highly sensitive to the profitability of those companies. When RIAA uses its resources to hurt customers by way of making an example of those whose listening and viewing preferences they don't like, it is playing with fire.

So let's approach via (or, better, stay away from) the 'customer door': if angry teenagers and young people and other good customers decided that it would be very cool not to buy any CDs or DVDs or go to a movie for, say, 90 days, the same corporations would likely undergo a rapid attitude, and tactics, adjustment.

Even better if the same customers took their disposable entertainment income and invested it in supporting local musicians, Indies et al. - or sent it along to responsible charities. I have refused to buy CDs or DVDs for 2 years now: the $1000 average that I spent annually on each of those products in the past now goes elsewhere... $1000 probably doesn't make an RIAA lawyer's car payment, but if enough of us choose this path, maybe RIAA will get the message...

Tue, 13 May 2003 04:09:19 GMT

RIAA apologizes for cease-and-desist order after they threaten an emeritus astronomy professor, Peter Usher, at Penn State . The report says that a cease-and-desist was issued after the prof's name "Usher" and an mp3 file were discoverd on the Penn State astronomy departments ftp server. The mp3 turned out to be an a capella ode to a satellite the department helped design, sung by the astronomers. RIAA apparently sends the orders without actually listening to the songs... A spider is supposed to have uncovered the file... I'm having a hard time believing it could be the work of the underpowered Cyveillance 'bot (or maybe it was and my prediction is coming true)... Naaaa... So which critter is doing the crawling for RIAA?

Mon, 12 May 2003 20:22:29 GMT

Cyveillancebot is quite busy on www.gulker.com today: 5:00 AM, 9:00 AM visits and more 'mini thrashes' at 11:00 and noon. Still hasn't crawled this, though blogging ecosystem has...

Mon, 12 May 2003 17:31:11 GMT

Cyveillance says the term RIAA shows up on 1620 'home pages', Googlebot says it shows up on 442,000. I'm beginning to think that Cyveillance's claim that it provides an Internet content "early warning system" for corporations is without basis in observed results.

When the RIAA failed to notice that their site had been hacked, by a relatively trivial mechanism, observers pointed out that RIAA seemed to be all but clueless about technology, particularly Internet technology.

I think that this jibes with my observations about Cyveillance: its technology does not to appear to be very good (in fact, is visibly very bad in the instance of its thrashing 'bot). The reason that it can command large fees from RIAA and others is that they are even more clueless than Cyveillance is. In the kingdom of the blind, a guy with one severely near-sighted eye can apparently do quite well selling Web services...

Mon, 12 May 2003 17:08:38 GMT

Looks like Cyveillancebot has thrashed someone else's site. But, how do you really feel...

Mon, 12 May 2003 16:44:08 GMT

Hit Song Science is another Web service that all the major music companies subscribe to. Interesting... record companies relying on robots to pick hits...

Mon, 12 May 2003 16:24:09 GMT

Cyveillancebot thrashed www.gulker.com (actually a mini-thrash this time) at around 4:30 AM on May 11, but it was snooping old directories, not the new directories loaded with commentary about it, its owners, its major clients and hot-button terms from DRM to file trading to terrorism. It does seem to have a predilection for downloading copyrighted material... in this case my copyrighted material...

Sun, 11 May 2003 06:55:16 GMT

Interesting: Cyveillance's 'bot ignores robots.txt (it hasn't accessed that file on gulker.com in nearly 1000 visits), but Cyveillance publishes a robots.txt on their site. Hmmm... so what's good for the goose is not good for us ganders?

Sun, 11 May 2003 02:05:56 GMT

Cyveillance will analyze your online brand for free: go to this page and enter a brand that you want to analyze (I tried my name) and it will tell you how many domains you have, how many pages you show up on (and breaks them out by conventional, violence, porn etc.), how often you turn up in links, etc.

Interesting... but I sense something's missing... here are the results of 3 searches on names in both Cyveillance and Google and the number of hits they return:

Search Name	Dave Winer	Hilary Rosen	Chris Gulker
Cyveillance	252	60	22
Googlebot	130,000	31,000	30,500

Hilary Rosen is CEO of RIAA, and much in the news lateley. Dave Winer is a Berkman Fellow at Harvard, and a much-read observer of technology. So, I'm either misunderstanding what Cyveillance means by 'home pages' or the universe that Cyveillancebot crawls is miniscule compared to Googlebot's. Googlebot touches gulker.com about 1000 times more often than Cyveillancebot: if that's any measure of the relative crawl-power of the two 'bots, then you'd expect to see 1000x fewer hits on a given name on Cyveillance, which is very roughly the case...

Sat, 10 May 2003 21:09:05 GMT

Phil Ringnalda comments that some 'bots treat robots.txt as a menu, not a fence. I've set gulker.com's version to help study this behavior...