Flutterby™! : Weblogs.com weirdness

Next unread comment / Catchup all unread comments User Account Info | Logout | XML/Pilot/etc versions | Long version (with comments) | Weblog archives | Site Map | | Browse Topics

Weblogs.com weirdness

2003-04-15 16:03:17.659455+00 by Dan Lyke 2 comments

I finally figured it out! Weblogs.com blocks wget. The error message is entirely non-intuitive,

Your crawler is hitting our servers too hard. Please slow down, it's hurting the service we provide to our customers. Thanks.

and I'd been trying to figure out why I was getting the error sometimes and not others, and from different IP addresses. Sigh. Oh well, lacking a clue bat I guess I'll have to code up something simple that gives a personalised client name in Perl.

[ related topics: Weblogs Perl Open Source Software Engineering ]

comments in ascending chronological order (reverse):

#Comment made: 2003-04-15 16:36:51.456214+00 by: Mark A. Hershberger

Thanks! I was getting this from doc.weblogs.com

Shouldn't you be able to cloak with "-U"?

#Comment made: 2003-04-15 17:06:11.091444+00 by: Dan Lyke

It was simple enough to use LWP::UserAgent[Wiki], and that's probably the right way to do things anyway. I didn't bother making my wget command line any more complex. Oddly, the LWP::UserAgent[Wiki] "GET" command, with a default string, worked just fine, but now I've got a unique user agent string and I don't hit the server more frequently than 2 hours.