Fragmented Flat File RSS Construction
A month ago I thought I was being clever in how I was generating RSS feeds for my email centric rssCloud experiment called “nudges”. Simply, every post creates a static xml file that is an item element and which becomes a part of an RSS feed. I immediately enjoyed this simple approach to generating RSS. It was lightweight, easy to code and added flexibility while remaining sturdy and not prone to full data corruption by a single point of failure (a single file). Additionally, I had in mind ways to distribute the data to other feeds that could attach the item to itself across networks. Since the focus is on short micro-messages, this approach seemed to make sense to me and one of my rules as a developer is to not add sql database dependencies until it actually becomes logical to do so. Often, that means you can have fully functioning applications that do not need or use databases beyond a file system with static flat files.
I had then googled to see what else took similar approaches to generating RSS or other XML. I came across one little script that a student had written. Found some references regarding XSL that had similar methodologies. But overall barely anything. Until just now which is the spark for this blog post.
In my research on the commentAPI namespace and how I might want to re-purpose that for rssCloud @replies, I was hanging around bitworking.org. I’ve been there before and I have been familiar with commentAPI of course (it is silently pervasive) but what struck me was Joe Gregorio’s experiments with commentAPI which seem to be in line with what I want to do with rssCloud…. but also I was taken back by his blogging platform called RESTlog which is what bitworking.org runs on. This is what it sais in regards to RSS construction:
How does the server side work?
Items are stored in individual files with their name being their ID number. They are stored as xml files. Specifically they are stored as ‘item’ elements from RSS 2.0.Why? Well first note that this file is conveniently formatted so that it can be easily placed inside an RSS file directly as an item which will speed up the rebuilding of the index.rss file each time an item is modified. Also having each item in its own file makes the system more robust, if one of the files becomes corrupt I have lost as most one item not all of them. I ran into this problem when working on Pamphlet, my as yet unpublished weblogging tool. The items in Pamphlet are kept in monthly archives which has the weakness of potentially losing a whole months worth of posts if a single file gets corrupted. Luckily I keep good backups.
The HTML main page ‘index.html’ and the RSS file ‘index.rss’ are served up statically. That is, they are static files that are only updated by the server side script when items are added or changed. Each item has it’s own permalink and that permalink is served up dynamically through the RESTLog.cgi script, but this only uses a crude string substitution templating mechanism.
This work occurred back in 2001/2002. Here is the RESTLog Spec
Now I can discount this for being antiquated methods that have no place in today’s space. But that is never smart or accurate. In fact, the more I look back, the more I feel that we are not nec in a clean and clear time for technology and innovation. We see a lot of end results that appear innovative or mass adoption of services that in return make us assume that their is innovation… but I feel like their is a lot of bloat and black magic out there. It’s not nec a bad thing as it relates to a learning and experimentation phase on the Interwebs. Messes are inevitable. Meanwhile, their is a strong fortitude to push semantics and standards to the forefront of the most important and influential discourse happening today. A balance is needed between cool results and solid practices.
So back to this little gem. Well, Joe Gregorio is today celebrating 2 years at Google, Inc. So it’s safe to assume that a solid mind was behind RESTLog. And for now, i’ll take the idea of Fragmented Flat File RSS Construction and move forward with it 
Related posts: