NewsMonster 1.2 is now available . This build addresses improvements in usability, performance enhancements, and new bug fixes.

An important issue is now fixed that will enable us to expand on the NewsMonster reputation system. NewsMonster now tracks links on Windows (broken in 1.1) and records a lot more implicit information to share among peers. This should provide for a good bootstap of the reptutation system to allow us to move forward with our collaborative filtering and reputation functionality.

link to full image

Channel Browser in NewsMonster 1.2

Changelog

New channel browser in 1.2

NewsMonster 1.2 Beta1 is now available.

I wanted to get this out sooner rather than later in order to facilitate feedback. If no one has any problems I will probably do a 1.2 release late Monday evening. PLEASE REPORT YOUR RESULTS WITH THIS BUILD.

Significant new functionality includes:

Download

http://www.newsmonster.org/download/test-07-27-2003-1059369429.xpi

Changelog

It appears that the comments system has been broken on this site for a while and no one complained. Brad was nice enough to notice this and let me know. The problem was that a comment would fail to post but silently succeed. Very annoying.

Neil brings up something I have been thinking about lately :

If you look at a number of weblogs, you'll notice that many have a wide variety of badges (or buttons), indicating various things they support. Such badges include RSS, FOAF, Valid HTML, weblog tools, Creative Commons links, and so on. It's like being a boy scout. Got some RSS? Get an RSS badge. Passed the CSS validator? Get a Valid CSS badge. Then, people proudly display all their badges on their web site.

He then goes on to point out that iwe should be syndicating these badges in a "file" instead of presented on the users website.

I think he is onto something but doesn't take it to its inevitable conclusion.

Badges and sidebars (bookmarks, blogrolls, etc) should be syndicated. This could allow aggregators such as NewsMonster to integrate the badges into the presentation and provide enhanced options to the user.

For example Joi Ito's blog has sidebar options to subscribe via email, to view photo albums, monthly archives, etc. It would be cool to expose this functionality to aggregators.

Someone on Java Blogs obviously had a problem with their RSS generation (I won't name names). They kept increasing the size of displayed fonts until single characters were actually larger than my browser window.

Ouch...

link to full image

The other day in Aggregator Subscription Mechanisms I talked about using a feed:// or aggregator:// URI scheme to trigger manual subscription for RSS/Atom feeds.

The last few days I have given it some thought and reviewed IETF and W3C documentation on the subject and I believe that this URI scheme would be RFC 2718 compliant.

There is prior art on the subject with the view-source, mailto, javascript, and aim URI schemes.

The view-source scheme is probably the closest cousin for a feed URI scheme. In both cases we have a URI to an HTTP resource contained within a parent URI which has an associated action. In the case of the view-source scheme we are telling our browser to load up the source code for the given HTTP URL in a source code viewer of some sort. In the case of the feed scheme we would tell it to load the HTTP URL in our aggregator.

Section 2.3 (Demonstrated Utility) is probably the most apropos section:

URL schemes should have demonstrated utility. New URL schemes are expensive things to support. Often they require special code in browsers, proxies, and/or servers. Having a lot of ways to say the same thing needless complicates these programs without adding value to the Internet.

In the case of our feed scheme this is clearly demonstrated. There are thousands of "XML" links on websites right now. People clearly want to expose their feed URLs.

Another import note - HTML supports 'type' attributes for anchors .

This would allow you to specify the media type within your HTML.

<a href="http://www.peerfear.org/rss/index.rss" type="application/rss+xml"/>

There are some drawbacks here. If you had the ability to discover the media type within the URL we wouldn't have to rely on content producers to setup their webservers correctly.

Another downside is that I doubt this convention would take off.

I created a NewsMonster wiki the other day. There isn't much content right now but it should be a good way for NewsMonster users and developers to collaborate.

If you have new feature ideas, want to document a problem you are having, want to collaborate on logos, etc, this is probably the best place.

NewsMonster 1.1 final is now available. As usual if you are currently running NewsMonster you will automatically be prompted to upgrade.

This build includes all features of the first and second 1.1 betas.

New Features

Advertising in NewsMonster FREE

There is now a "sponsored channels/links" sidebar if you are running the free version of NewsMonster. Don't be alarmed, this is very unobtrusive. If you want to advertise here you should purchase NewsMonster PRO or contact me directly. Note that if you don't want to see advertisements you can always purchase NewsMonster PRO.

Next release

I wanted to put out another beta of 1.1. Earlier today I started working on manual feed updates and I fixed a few more bugs. This started out as a small change ended up but now I would feel

The major functionality I need testing is the manual feed update. This allows you to quickly update a site without having to aggregate all of your subscriptions. Right now I am pushing 200 subscriptions and waiting 10 minutes just to see an updated BoingBoing just isn't cool.

Ideally it would be nice to get some feedback by tomorrow morning as I would like to release 1.1 final in the afternoon. There are just too many important changes to 1.1 that I want to get this out the door ASAP.

Tests:

Download:

http://www.newsmonster.org/download/test-07-19-2003-1058682376.xpi

Shrook now has a feature called "Distributed Checking" which allows each version of Shrook to update a feed within a shorter interval (sometimes 5 minutes). This allows each Shrook instance to stay up to date without pounding the Internet connection of the feed host.

This is certainly an honorable goal. Delivering news to users quicker is something we all want.

To oversimplify: A central server maintains a database of when each channel was last updated. To keep it up to date, every so often, the server chooses a computer to check for new items and report back. The frequency of this varies from every 5 minutes for popular channels, to every half hour for channels with only one online subscriber, and it tries to use a different computer each time. At the other end, each copy of Shrook checks in with the server every 5 minutes, and if any of its channels are out of date, it reloads them.

In short this is a bad idea and I would like to explain why. (Note that since Shrook doesn't provide source (like NewsMonster) I had to conduct this analysis from the FAQ )

First off, this isn't distributed checking, this is centralized checking. The fact that Shook is using peers to perform the update check is irrelevant. The server could easily perform the update check and in fact this would be an optimization. Bandwidth for clients would be reduced and issues like NAT and proxy caches (which might return stale content) would not affect the update check.

The major downside is that for popular feeds this will literally become a Distributed Denial of Service attack once a significant number of peers has been deployed with distributed checking. Imagine you run a blog like Boing Boing and you have thousands of RSS readers hitting your site every hour. Now imagine you update your feed and within a five minute interval thousands of machines decide to download the same file? This would be a very bad thing. Your link would be saturated with RSS feed downloads

In the past we have had load balancing of feed download due to the natural update interval distribution due to the randomness with with aggregators are started and synchronized. With distributed checking this load balancing is destroyed.

This might not be the case. The Shrook distributed checking FAQ says "... each copy of Shrook checks in with the server every 5 minutes, and if any of its channels are out of date, it reloads them."

It doesn't specify where or how it reloads them. I highly doubt that it fetches the content from the centralized server as this would probably yield a hefty bandwidth invoice for the author.

Shrook also doesn't specify if it support gzip content encoding. If it doesn't support this feature (and any RSS reader worth a grain of salt needs to specify this on the webpage which Shrook does not) then this is a highly irresponsible change.

Not only would this essentially yield a "DDoS attack" of popular feeds but it would be downloading unnecessary content which could have been compressed.

Mozilla has an ego problem.

Want to view the source code of a web page?

Stuck using the Mozilla "view source" app.

Stuck using Mozilla Mail.

Stuck using Mozilla (and not IE, Konqueror, Safari, or any other browser)

The Mozex project exists but it needs some more work.

Scott brings up some interesting point about easy aggregator subscription mechanisms.

A quick update for those who don't understand what is going on here.

A number of aggregators provide a mechanism for their users to quickly subscribe to a feed. Usually this is used to bootstrap some type of wizard or configuration GUI or to redirect or a subscription URL.

In the case of NewsMonster we use a newsmonster-subscription: URI scheme with the feed URL appended to the end of the scheme prefix. This is almost the same function as the view-source URI that has been in Mozilla for years.

Why do we need this? Purists would argue that we in fact don't need this technology. The theory is that we currently have an RSS auto-subscription mechanism that detects the feed within a web page and automatically allows the user to subscribe with their favorite aggregator (which most aggregators seem to support).

There are a few issues here which mostly have to do with usability. What if you want to link to someone else's feed? What if you want to have a nice chicklet button for subscribing to a feed with a specific aggregator? What if you want to design your page around the fact that a feed is present - sometimes an in-browser UI metaphor can be very powerful.

None of this works with auto-subscription. None of this works with links to RSS files that don't have the correct media type set. Even if they do have the correct media type set I know of no feed reader that can bootstrap into a subscription UI based on the application/rss+xml media type.

This is a different type of subscription. This is manual-subscription.

Solutions

Using a custom URL rewritten to include the URL of the feed:

This is the mechanism Radio uses as well as Amphetadesk . Essentially they just load a URL on http://127.0.2.1:8080 and hope and pray that a server is running on that port. In most cases it is.

PROS:

Easy to support (just keep a port open)

Works well if you are a HTTP/browser oriented aggregator

CONS:

Terrible user experience. If the server isn't running the user gets a socket timeout.

Security?

Used with: Radio, Amphetadesk, Reptile, with Radio emulation supported by a few others.

Use application/rss+xml media type detection.

Use a custom URI scheme

Where do we go from here?