Caching the Lingerie: Catastrophic Decline and Fall
I have been obsessing about what to get Phyllis for Valentine's day. My thinking is sexy lingerie. Is it OK to still be in love with your wife, and find her hot, after all these years, and three kids later?
Moving on. A few weeks back I wrote about Jared Diamond's book Collapse. I've now finished Collapse, and I find myself distressed by it. Certainly, the specter of our society succumbing to Malthusian and/or ecological disaster is frightening. But I think Collapse is disingenuous: it extrapolates from tiny, fringe societies (Viking Greenland, Henderson Island, Easter Island, and so on) to ours. It is completely clear why these societies catastrophically collapsed. For example, the Vikings in Greenland were dealing with an inhospitable climate without sufficient growing season, and then the climate got even worse. Yeah, maybe if they'd made like the Inuit they might have survived, but then again they might not have.
The much more interesting question to me is why do strong societies and institutions decline? What was the real cause of the fall of the Roman empire? Why do some companies create a culture that thrives, while others go through the motions, become hollow hulks that are "former glories," and eventually hit a catastrophic decline? What makes the Googles of the world Googles, and how long will they go on being Googles? At the opposite end of the spectrum, I used to work for Informix, a company that arguably had the best enterprise database technology. It has long since been swallowed by IBM, and is another tech wreck on the Ozmandius seas of time. What makes the difference?
I don't know the answers, but I want to go on thinking about the harder questions, not the easy questions (like why did a few thousand Norse in Greenland die after the winters got even colder and the ships stopped coming from Europe).
Generally, in a technology context, catastrophic failure results from a concatenation of unintended consequences rather than having a single cause. Most likely, technology failure means that a Wesbite is down, or that your site is generating HTTP 404 errors, not buildings and bridges falling down, or walls of water.
But the consquences are serious enough for the businesses involved, and it is worth thinking about a mindset that avoids problems. If cascading unintended consquences are the proximate causes of most system meltdowns, then the answer is to isolate, and to use mechanisms that provide elegant economy. Make sure that the wave of destruction can't spread too far, and keep things simple and not wasteful, even if it seems like there are resources to burn. (If this seems parallel to the right way to manage a society's ecosystem, that's the idea!)
Caching is a simple mechanism that fits the bill. If you've looked into how Google works, you'll know that the Google document servers make extensive use of caching. This is a straightforward mechanism: it takes less resources to grab a file from the file system than to generate the file. So the first time something(such as a list of search results) is requested, it is cached. Subsequent requests for the same information can use the (cheaper) cached version, provided nothing has changed.
Of course, nothing is ever free, so you have to worry about out of date (but cached) information, and you should think about the cost of caching (admittedly small) if no one ever requests the information again.
To bring this entry back to its beginning, I did a little bit of programming for Phyllis yesterday. She wanted a graphic on her High Risk pregnancy site displaying an estimate of the number of preemies born year-to-date in the US. (You can see what I came up with on the High Risk site.) This is something we care about, because our oldest son, Julian, was a 28-week preemie. Anything we can do to help parents who are in high-risk situations manage their pregnancies, we'd like to do.
The logic behind the numbers on the graphic is dirt simple: Roughly one a minute, time it from the beginning of the year. What I ended up doing was created a PHP program that "reads" the background graphic, and uses a value passed to it as a HTTP GET for display. You could actually call it with arbitrary text, for example, like this.
On the client-side, the Javascript code that generates the URL to the PHP program refreshes every 3 seconds. Off loading the time processing to the client-side takes away burden from our server, but I really wanted to cache the generated graphics. It's hard to see how to do so, since the numerical part of the graphic is constantly changing. This doesn't seem to cause a problem with only a few hits to HighRisk.org, but if there are thousands of hits, or someone malicious, you can see a potential vulnerability that might cause a cascading reaction.
I'll be thinking more in the days to come about how to avoid catastrophic declines. I'll also be thinking about Valentine's Day...
Posted by Harold Davis at February 4, 2005 10:37 AM
