Flutterby™! : Real syndication

Next unread comment / Catchup all unread comments User Account Info | Logout | XML/Pilot/etc versions | Long version (with comments) | Weblog archives | Site Map | | Browse Topics

Real syndication

2003-07-19 02:36:33.057438+00 by Dan Lyke 24 comments

I gotcher syndication right here, pal. None of this wussy "Uhhhm, excuse be, but we're going to entity encode HTML[Wiki] inside of XML[Wiki]" crap, none of wheel-reinvention of a format developed for freakin' headlines, none of the politics of new formats without applications, no standards bodies run amok, nobody's worried about who owns the copyright, that's what ya got tire irons and motorcycle chains for. There's no worrying about revision tracking here, when we want to make sure it's permanent, we'll tatoo it someplace that counts.

Dan has finally done the obvious: Flutterby via NNTP[Wiki], a protocol actually developed for delivering news. So, fire up a newsreader, point it to port 119 of your favorite server and subscribe to flutterby.weblogentries , use your email address from your Flutterby user info as your username, with your Flutterby password, and read some freakin' news.

Totally rawks in gnus, works nicely in Mozilla, Opera is a little funky, but nothing they won't fix, and it should work in all of the rest of your favorite newsreaders. Nope, not "news syndicators" or whatever that modern wussy crap is, we're talking industrial-strength time-tested terabytes-served chews alt.binaries.pictures.erotica.amateur.female for lunch blows your doors off and in the next county before you realize the light has changed technology.

This is the Bronco II to your Suzuki Samurai, a '67 SuperSport with a bored out block, glasspacks, and nitrous in the trunk to your mother's station wagon, baybee.

Limitations:

  • Read only right now. Working on it, but Flutterby doesn't thread, and I don't necessarily want those sorts of conversations coming to the front page.
  • Email and password changes might take up to 5 minutes to propagate, and the validity and duplication of email addresses in the user list is an issue. Email me if it doesn't work.
  • The "allow my email on the NNTP version" checkbox doesn't work. You can't give out your email yet. Deal.
  • I've noticed a lot of folks with spamtraps in the email list. If you ever get mail from one of those spamtraps, tell me immediately, 'cause it indicates something gone way wrong, and we'll have to leave the burning corpses of those responsible as a warning to others.

[ related topics: Web development Content Management Flutterby Meta Invention and Design Copyright/Trademark ]

comments in ascending chronological order (reverse):

#Comment made: 2003-07-19 02:44:41.435278+00 by: ghasty

Damn! You go boy!!!

Now can I link up via UUCP on my 300 baud to load into my BBS?

#Comment made: 2003-07-19 03:41:31.392884+00 by: John Anderson

A-frickin'-men.

#Comment made: 2003-07-19 11:33:22.624761+00 by: pharm

Ahem, leafnode he say:

check_date: www.flutterby.com: clocks of upstream and this computer are more than 10 minutes apart. Check your system clock.

Now, I sync by ntp from my isp, and the local clock looks right to me. Is flutterby's clock screwy, or is leafnode playing up?

#Comment made: 2003-07-19 15:41:55.058979+00 by: Mars Saxman

alright! I've always thought NNTP was underused... I've occasionally thought of building a peer-to-peer filesharing system based on it, actually.

maybe it's time to go download a newsreader.

-Mars

#Comment made: 2003-07-19 16:06:26.667447+00 by: Dan Lyke

pharm, your date is right. I've gotten this server a little closer, I'll see about diddling with ntp in a bit.

Mars, having looked at how this actually works, I'm now considering building something to move all these "modern" wheel reinventions back to NNTP. None of the dupliucated entries stuff that people are whining about, gnus gives all the filtering tools I'd ever want, and both gnus and Mozilla have solved the security issues that have recently plagued aggregators.

#Comment Re: [Entry #6357] Real syndication made: 2003-07-19 21:20:45.378678+00 by: Dan Lyke

And this one is a test to see if I can get updates feeding from the newsgroup back into the Flutterby pages.

Dan

#Comment Re: [Entry #6357] Real syndication made: 2003-07-20 00:16:03.455523+00 by: Dan Lyke

Okay, for those of you who have authorization worked out, this is now two-way:

All messages with a "References" header that references a front-page post will get attached to the appropriate front page post. No smarter threading than that. Best effort will be made to match user name and email back into your Flutterby user, but if not the post will simply appear as "anonymous from the NNTP side". I'll fix this when it becomes an issue.

Usual 5 minute latency applies. Might fix that if it becomes an issue, although as I watch the evolution of various online forums I'm beginning to think that enforced latency is a good thing.

For those of you who don't have authorization worked out, or who are still having problems, give me details on your setup. It's okay if that means "I use Internet Explorer for browsing and just get error X when I click on your link", just be as detailed as possible. Email is fine.

Thanks!

#Comment Re: [Entry #6357] Real syndication made: 2003-07-20 17:16:03.435241+00 by: Larry Burton [edit history]

Okay, are you going to put up a HOW-TO for those of use that might like to do the same thing with our 'blogs or would just a discussion about it here in the comments be a better way?

Dan Lyke wrote:
> Okay, for those of you who have authorization worked out, this is now
> two-way:
> 
> All messages with a "References" header that references a front-page
> post will get attached to the appropriate front page post. No smarter
> threading than that. Best effort will be made to match user name and
> email back into your Flutterby user, but if not the post will simply
> appear as "anonymous from the NNTP side". I'll fix this when it
> becomes an issue.
> 
> Usual 5 minute latency applies. Might fix that if it becomes an issue,
> although as I watch the evolution of various online forums I'm
> beginning to think that enforced latency is a good thing.
> 
> For those of you who don't have authorization worked out, or who are
> still having problems, give me details on your setup. It's okay if
> that means "I use Internet Explorer for browsing and just get error X
> when I click on your link", just be as detailed as possible. Email is
> fine.
> 
> Thanks!
> 
>

#Comment Re: [Entry #6357] Real syndication made: 2003-07-20 17:21:03.284037+00 by: Larry Burton

Oops, sorry about the quoting. I guess I'm going to have to re-train myself in this environment.

#Comment Re: Real syndication made: 2003-07-21 11:08:53.003472+00 by: meuon

Wow. Dan Rocks! - OK, we all knew that. - I'v tried to use NNTP for projects in the past, like Chatt. State Communiy College for Online Classes., with no success. This provides incredible functionality!

#Comment Re: Real syndication made: 2003-07-21 13:13:39.958707+00 by: topspin

Okay, the aggregators muddied the term "weblog," but putting content in a newsreader seems to change the name to "newslog" doesn't it? Not a complaint, mind you, just an observation. "Neblog....?"

I'm also naive enough to suggest some sorta "linking arrangement" be added somewhere (to each message as a footer?.. somehow as a "lead" message to the newsgroup?.... hey, I'm not a geek, I'm a guy with ideas.... stop snickering) which would open a browser to the web version of Flutterby, if not to the exact link found in the news version.

#Comment Re: [Entry #6357] Real syndication made: 2003-07-21 16:16:03.255827+00 by: Dan Lyke

Dang, Larry, I've tried a few tweaks to the formatter to make that quoting work (and now that I'm replying to this in Gnus I understand that format exactly), and I'm still missing something. I'll have to dig a bit.

On the HOW-TO for others: This is fairly heavily tied to the Flutterby system, but I was thinking that putting your average blog into NNTP should be trivial. I'm thinking I'll hack together an RSS-to-NNTP thingie just for giggles, although without a network of news servers it quickly becomes just another centralized aggregator.

Topspin, the coolest thing I could think of is that this becomes something more and other than a weblog. Weblogs are so 1998 now, and I got fired up recently about trying to be more 21st century.

And yes, I want to do a linking back, but I also wanted to stay away from muddying the nuggets of HTML goodness that end up on the news server. I was thinking about adding some "X-" headers, but then regular users wouldn't read it.

Maybe using style sheets? Will have to ponder, but not too long lest this become a standard.

#Comment Re: [Entry #6357] Re: Real syndication made: 2003-07-21 16:21:03.676358+00 by: Larry Burton

topspin wrote:

> Okay, the aggregators muddied the term "weblog," but putting content in 
> a newsreader seems to change the name to "newslog" doesn't it? Not a 
> complaint, mind you, just an observation. "Neblog....?"

Just another observation, I think the term would be 'slog.

How does this thing scale into including other 'blogs into the syndication? To replace RSS and Atom this thing is going to have to allow folks to use it like they are the other aggregators.

I'm just trying to understand where you are going with this.

-- Regards, Larry Burton

Now Listening to Patsy Cline - Crazy

#Comment Re: Real syndication made: 2003-07-21 17:59:12.879551+00 by: Mark A. Hershberger

Nice!

#Comment Re: Real syndication made: 2003-07-21 18:22:52.568027+00 by: Dan Lyke [edit history]

Oh yeah: Gary, if'n you wanna give me the lines for my newsfeeds file, I'll send it to you any way you want it.

#Comment Re: [Entry #6357] Re: Real syndication made: 2003-07-21 20:38:40.105076+00 by: Unknown, from NNTP

Larry Burton <larry@dallasbay.net> writes:

> topspin wrote:
>
>> Okay, the aggregators muddied the term "weblog," but putting content
>> in a newsreader seems to change the name to "newslog" doesn't it?
>> Not a complaint, mind you, just an observation. "Neblog....?"
>
> Just another observation, I think the term would be 'slog.

Following Dan's "be more 21st century", 'post-weblog' seems right. Short form? 'po'blog', natch. 

> How does this thing scale into including other 'blogs into the
> syndication? To replace RSS and Atom this thing is going to have to
> allow folks to use it like they are the other aggregators.

If everybody can run their own NNTP server (ha!), and everybody uses clients that can talk to more than one server from the same .newsrc file (I mean really, are there serious readers of netnews out there that aren't using Gnus?), where's the problem?

And I just noticed that Dan has proper threading going in the group, but the same linear flow on the web -- Dan, you rock.

john. -- "Faith which does not doubt is dead faith." - Miguel de Unamuno

#Comment Re: Real syndication made: 2003-07-21 20:45:28.937358+00 by: Dan Lyke

And now I've fixed the "what happens if we can't figure out who posted it" problem. Still need to handle NNTP replacement, and probably do something smarter about figuring out who posted it (email address given at Flutterby and of John's news client don't match).

#Comment Re: Real syndication made: 2003-07-22 12:17:38.837751+00 by: Steve Tibbett [edit history]

Ah looks like I'm not the only one.

news://food.dhs.org

Project is here (with source, C#):

http://www.gotdotnet.com/Commu...67a8-123d-4e11-ae71-a9d67aec8447

It makes so much sense to use a network of NNTP servers much like we have with Usenet, and have agents that can poll an RSS feed and deposit the contents into one of the servers on the NNTP network where it can propagate to all the others.

RSS is cool but it's designed to somewhat mitigate the problem of a user having to go and check for updates on a page constantly by having software do it.. but it really doesn't scale. Having a million aggregators polling your site is such a massive waste of bandwidth.

RSS basically punishes you for it. The more popular what you're saying is, the more you have to pay (in bandwidth) for saying it. :)

One issue with the a big distributed system is who gets to make up the group names and decide what goes where.. I guess some sort of moderation system would be required for who gets to do that.. but it would be cool if it was self-moderating somehow.

Difference between what you're doing and what I'm doing is you're feeding a real NNTP server, and I'm pretending to be one (when you connect to food.dhs.org:119, that's my server). I'm switching to feeding a real server.

Maybe we could combine the two efforts somehow - set up a real network of connected NNTP servers sharing the same feeds?

If we can come up with a way for users to add feeds themselves (with moderation) and let the network grow, it could turn into something big...

#Comment Re: Real syndication made: 2003-07-22 15:08:26.078048+00 by: Dan Lyke

Steve, I'm totally in to syndicating other weblogs via NNTP[Wiki]. Larry (of Larry's Log) and I are talking about ways to get his weblog online, and I'd really love to have another NNTP[Wiki] server to start echoing some of these groups to.

I too started out with writing my own NNTP server, but as I got into the intricacies of how each different client uses the server, and eventually had to just host a newsgroup for another project, inn2[Wiki] seemed like the easiest thing to do.

So, I'm happy to go start relearning what "newsfeeds" should look like if you've got an IP address for me to start ending stuff to.

One of the problems with NNTP[Wiki] is that it's based on a lot of trust, but I think if we grow carefully and get really nasty about cutting off anyone who violates that trust we should be able to accomplish some interesting things. And maybe we can figure out a way to do dynamic feeds, so that if I'm on a machine and fire up a server I can have inbound updates of my news without having to poll?

#Comment Re: Real syndication made: 2003-07-22 18:26:13.3059+00 by: Shawn [edit history]

I'll hack together an RSS-to-NNTP thingie just for giggles

Dan, are you aware there is already a product out there that does this?

#Comment Re: [Entry #6357] Re: Real syndication made: 2003-07-22 18:56:03.304248+00 by: Larry Burton

That isn't quite the same thing, Shawn. The product you reference allows one to read RSS feeds in a newsreader. Yeah the end result is you are reading the content in a newsreader, but Dan is bypassing RSS and just publishing straight to his very own newsgroup on a news server without producing any RSS in between. When he manages to put together a group of servers offering flutterby.* then he's dividing the load on his machine up by people being able to retrieve his content from other servers. Contrast this with RSS which can increase the network load on his server.

#Comment Re: Real syndication made: 2003-07-22 19:38:11.916355+00 by: Dan Lyke

Actually, Larry, the product Shawn pointed to was roughly what I was talking about hacking together. Of course for the purposes of what we've been hashing back and forth in email we're going to bypass RSS, and yes, I'm just using raw database queries back and forth for my stuff.

#Comment Re: Real syndication made: 2003-07-23 03:22:37.016984+00 by: anon123

See NNTP//RSS (http://www.queuing.com/methodize.org/nntprss/) - it's been doing this for a while. It allows you to reply to entries as well (if the blog supports the comments API)

Patches are available (http://www.mackmo.com/nick/blo...ermalink=nntprssc4javailable.txt) to do Bayesian classification of entries.

#Comment Re: Real syndication made: 2003-07-23 06:37:01.075912+00 by: Shawn [edit history]

I'm less familiar with the inner workings of NTTP than I am with other protocols, but it seems to me that even using RSS as a go-between would offer significant bandwidth/access advantages. Only Dan's news server would be accessing the RSS feed (in the vacuum of this model - yes, I realize that if an RSS feed is available, others will probably be using it directly). From that point on the distributed access model takes over. Right?

It seems to me that this is the real strength of XML (if there is one). Extract the raw data once and then let whatever service that wants it massage it into the look desired.

Obviously, there's some overhead involved in this approach, but one gains the advantage of modularity. It should then be fairly easy to build any interface on top of that framework.