Saturday, December 31, 2005
Reading List: Talk to the Hand
- Truss, Lynne. Talk to the Hand. London: Profile Books, 2005. ISBN 1-86197-933-9.
- Following the runaway success of Eats, Shoots & Leaves, one might have expected the author to follow up with another book on grammar, but instead in this outing she opted to confront the “utter bloody rudeness of everyday life”. Not long ago I might have considered these topics unrelated, but after the publication in July 2005 of Strike Out, and the subsequent discussion it engendered, I've come to realise that slapdash spelling and grammar are, as explained on page 23 here, simply one aspect of the rudeness which affronts us from all sides. As Bernard Pivot observed, “[spelling] remains a politeness one owes to our language, and a politeness one owes to those to whom one writes.” In this book Truss parses rudeness into six categories, and explores how modern technology and society have nearly erased the distinctions between private and public spaces, encouraging or at least reducing the opprobrium of violating what were once universally shared social norms. (Imagine, for example, how shocking it would have seemed in 1965 to overhear the kind of intensely personal or confidential business conversation between two fellow passengers on a train which it is now entirely routine to hear one side of as somebody obliviously chatters into their mobile phone.) Chapter 2, “Why am I the One Doing This?”, is 23 pages of pure wisdom for designers of business systems, customer relations managers, and designers of user interfaces for automated systems; it perfectly expresses the rage which justifiably overcomes people who feel themselves victimised for the convenience and/or profit of the counterparty in a transaction which is supposedly of mutual benefit. This is a trend which, in my opinion (particularly in computer user interface design), has been going in the wrong direction since I began to rant about it almost twenty years ago. A U.S edition is also available.
Friday, December 30, 2005
Reading List: Mommy Knows Worst
- Lileks, James. Mommy Knows Worst. New York: Three Rivers Press, 2005. ISBN 1-4000-8228-5.
- Why did we baby boomers end up so doggone weird? Maybe it's thanks to all the "scientific" advice our parents received from "experts" who seemed convinced that despite millennia of ever-growing human population, new parents didn't have the slightest clue what do with babies and small children. James Lileks, who is emerging as one of the most talented and prolific humorists of this young century, collects some of the very best/worst of such advice in this volume, along with his side-splitting comments, as in the earlier volumes on food and interior decoration. Flip the pages and learn, as our parents did, why babies should be turned regularly as they broil in the Sun (pp. 36-42), why doping little snookums with opiates to make the bloody squaller shut up is a bad idea (pp. 44-48), why everything should be boiled, except for those which should be double boiled (pp. 26, 58-59, 65-68), plus the perfect solution for baby's ears that stick out like air scoops (pp. 32-33). This collection is laugh-out-loud funny from cover to cover; if you're looking for more in this vein, be sure to visit The Institute of Official Cheer and other features on the author's Web site which now includes a weekly audio broadcast.
Tuesday, December 27, 2005
Reading List: Under the Banner of Heaven
- Krakauer, Jon. Under the Banner of Heaven. New York: Anchor Books,  2004. ISBN 1-4000-3280-6.
- This book uses the true-crime narrative of a brutal 1984 double murder committed by two Mormon fundamentalist brothers as the point of departure to explore the origin and sometimes violent early history of the Mormon faith, the evolution of Mormonism into a major mainstream religion, and the culture of present-day fundamentalist schismatic sects which continue to practice polygamy within a strictly hierarchical male-dominated society, and believe in personal revelation from God. (It should be noted that these sects, although referring to themselves as Mormon, have nothing whatsoever to do with the mainstream Church of Jesus Christ of Latter-day Saints, which excommunicates leaders of such sects and their followers, and has officially renounced the practice of polygamy since the Woodruff Manifesto of 1890. The "Mormon fundamentalist" sects believe themselves to be the true exemplars of the religion founded by Joseph Smith and reject the legitimacy of the mainstream church.) Mormonism is almost unique among present-day large (more than 11 million members, about half in the United States) religions in having been established recently (1830) in a modern, broadly literate society, so its history is, for better or for worse, among the best historically documented of all religions. This can, of course, pose problems to any religion which claims absolute truth for its revealed messages, as the history of factionalism and schisms in Mormonism vividly demonstrates. The historical parallels between Islam and Mormonism are discussed briefly, and are well worth pondering: both were founded by new revelations building upon the Bible, both incorporated male domination and plural marriage at the outset, both were persecuted by the existing political and religious establishment, fled to a new haven in the desert, and developed in an environment of existential threats and violent responses. One shouldn't get carried away with such analogies--in particular Mormons never indulged in territorial conquest nor conversion at swordpoint. Further, the Mormon doctrine of continued revelation allows the religion to adapt as society evolves: discarding polygamy and, more recently, admitting black men to the priesthood (which, in the Mormon church, is comprised of virtually all adult male members). Obviously, intertwining the story of the premeditated murder of a young mother and her infant committed by people who believed they were carrying out a divine revelation, with the history of a religion whose present-day believers often perceive themselves as moral exemplars in a decadent secular society is bound to be incendiary, and the reaction of the official Mormon church to the publication of the book was predictably negative. This paperback edition includes an appendix which reprints a review of a pre-publication draft of the original hardcover edition by senior church official Richard E. Turley, Jr., along with the author's response which acknowledges some factual errors noted by Turley (and corrected in this edition) while disputing his claim that the book "presents a decidedly one-sided and negative view of Mormon history" (p. 346). While the book is enlightening on each of the topics it treats, it does seem to me that it may try to do too much in too few pages. The history of the Mormon church, exploration of the present-day fundamentalist polygamous colonies in the western U.S., Canada, and Mexico, and the story of how the Lafferty brothers went from zealotry to murder and their apprehension and trials are all topics deserving of book-length treatment; combining them in a single volume invites claims that the violent acts of a few aberrant (and arguably insane) individuals are being used to slander a church of which they were not even members at the time of their crime. All of the Mormon scriptures cited in the book are available on-line. Thanks to the reader who recommended this book; I'd never have otherwise discovered it.
Monday, December 26, 2005
Friday, December 23, 2005
Thursday, December 22, 2005
The Einstein-Szilard Refrigerator Now OnlineAs the centennial of Einstein's miraculous year of 1905 draws to a close, I thought it would be appropriate to also commemorate the 75th anniversary of another of Einstein's contributions to human knowledge—this one decidedly more down to Earth than special and general relativity. Albert Einstein is often thought of as an archetype of the unworldly theoretical physicist, but in fact he was interested in and knowledgeable on a broad variety of topics far removed from his speciality. In 1930, along with fellow physicist Leo Szilard, he was granted U.S. Patent 1,781,541 for a refrigerator with no moving parts other than the refrigerant. At the time the patent was issued, Einstein and Szilard were residents of Berlin; both were soon to flee Germany when Hitler came to power. This document is a facsimile of the U.S. patent on the Einstein-Szilard refrigerator, filed on December 16, 1927, granted November 11, 1930, and licensed to the Electrolux Corporation. The document is based upon the Issued Patents Database published by the U.S. Patent and Trademark Office, which has full text for all patents issued since 1976 and scanned page images for patents all the way back to 1790. The images for older patents are not in a particularly Web-friendly format; they're large (2320×3408) monochrome bitmaps in TIFF format, which require a plug-in to view with most browsers, and may not scale well to a smaller screen. For this document I downloaded the original TIFF images (which are in the public domain), cropped borders, converted them to 16 bit per pixel grey-scale images, smoothed with a 3×3 convolution matrix, then resampled to 768×1202 image pixel images which were output in grey-scale JPEG encoding, wrapped in HTML documents with page navigation buttons.
Tuesday, December 20, 2005
Reading List: The New Paradigm
- Bockris, John O'M. The New Paradigm. College Station, TX: D&M Enterprises, 2005. ISBN 0-9767444-0-6.
As the nineteenth century gave way to the twentieth, the triumphs of
classical science were everywhere apparent: Newton's theories of
mechanics and gravitation, Maxwell's electrodynamics, the atomic
theory of chemistry, Darwin's evolution, Mendel's genetics, and the
prospect of formalising all of mathematics from a small set of logical
axioms. Certainly, there were a few little details awaiting explanation:
the curious failure to detect ether drift in the Michelson-Morley
experiment, the pesky anomalous precession of the perihelion of
the planet Mercury, the seeming contradiction between the
equipartition of energy and the actual spectrum of black
body radiation, the mysterious patterns in the spectral lines
of elements, and the source of the Sun's energy, but these seemed
matters the next generation of scientists could resolve by building
on the firm foundation laid by the last. Few would have imagined that
these curiosities would spark a thirty year revolution in physics
which would show the former foundations of science to be valid only
in the limits of slow velocities, weak fields, and macroscopic
At the start of the twenty-first century, in the very centennial
it is only natural to enquire how firm are the foundations of
present-day science, and survey the "little details and anomalies"
which might point toward scientific revolutions in this century.
That is the ambitious goal of this book, whose author's long career
in physical chemistry began in 1945 with a Ph.D. from Imperial
College, London, and spanned more than forty years as a full professor
at the University of Pennsylvania, Flinders University in Australia,
and Texas A&M University, where he was Distinguished Professor of
Energy and Environmental Chemistry, with more than 700 papers and
twenty books to his credit. And it is at this goal that Professor
Bockris utterly, unconditionally, and irredeemably fails.
By the evidence of the present volume, the author, notwithstanding his
distinguished credentials and long career, is a complete idiot.
That's not to say you won't learn some things by reading this
book. For example, what do
physicists Hendrik Lorentz, Werner Heisenberg, Hannes Alfvén,
Albert A. Michelson, and Lord Rayleigh;
chemist Amedeo Avogadro,
astronomers Chandra Wickramasinghe, Benik Markarian,
and Martin Rees;
the Weyerhaeuser Company;
the Doberman Pinscher dog breed;
Renaissance artist Michelangelo;
Cepheid variable stars;
Nazi propagandist Joseph Goebbels;
the Menninger Foundation and the Cavendish Laboratory;
evolutionary biologist Richard Dawkins;
religious figures Saint Ignatius of Antioch,
Bishop Berkeley, and Teilhard de Chardin;
parapsychologists York Dobyns and Brenda Dunne;
anomalist William R. Corliss;
Centreville Maryland, Manila in the Philippines,
and the Galapagos Islands
all have in common?
Their names are all misspelled in this book. Werner Heisenberg shares the distinction of having his name spelt three different ways, providing a fine example of Heisenberg uncertainty, although Chandra Wickramasinghe takes the prize with three different incorrect spellings within five pages: "Wickrisingam" (p. 146), "Wackrisingham" (p. 147), and "Wackrasingham" (p. 150). Even Bockris could not wackily rise to the challenge of misspelling the last names of statistician I. J. Good or physicist T. D. Lee—so he got their initials wrong! Evidently, the author's memory for names is phonetic, not visual, and none too accurate; when a citation is required, he just hits whatever keys resemble his recollection of the name, and never bothers to get up and check the correct attribution on his bookshelf.
Here, it is relevant to describe a corridor meeting with a mature colleague - keen on Quantum Mechanical calculations, - who had not the friends to give him good grades in his grant applications and thus could not employ students to work with him. I commiserated on his situation, - a professor in a science department without grant money. How can you publish I blurted out, rather tactlessly. "Ah, but I have Lili" he said (I've changed his wife's name). I knew Lili, a pleasant European woman interested in obscure religions. She had a high school education but no university training. "But" ... I began to expostulate. "It's ok, ok", said my colleague. "Well, we buy the programs to calculate bond strengths, put it in the computer and I tell Lili the quantities and she writes down the answer the computer gives. Then, we write a paper." The program referred to is one which solves the Schrödinger equation and provides energy values, e.g., for bond strength in chemical compounds.Now sit back, close your eyes, and imagine five hundred pages of this; in spelling, grammar, accuracy, logic, and command of the subject matter it reads like a textbook-length Slashdot post. Several recurrent characteristics are manifest in this excerpt. The author repeatedly, though not consistently, capitalises Important Words within Sentences; he uses hyphens where em-dashes are intended, and seems to have invented his own punctuation sign: a comma followed by a hyphen, which is used interchangeably with commas and em-dashes. The punctuation gives the impression that somebody glanced at the manuscript and told the author, "There aren't enough commas in it", whereupon he went through and added three or four thousand in completely random locations, however inane. There is an inordinate fondness for "e.g.", "i.e.", and "cf.", and they are used in ways which make one suspect the author isn't completely clear on their meaning or the distinctions among them. And regarding the footnote quoted above, did I mention that the author's wife is named "Lily", and hails from Austria? Further evidence of the attention to detail and respect for the reader can be found in chapter 3 where most of the source citations in the last thirty pages are incorrect, and the blank cross-references scattered throughout the text. Not only is it obvious the book has not been fact checked, nor even proofread; it has never even been spelling checked—common words are misspelled all over. Bockris never manages the Slashdot hallmark of misspelling "the", but on page 475 he misspells "to" as "ot". Throughout you get the sense that what you're reading is not so much a considered scientific exposition and argument, but rather the raw unedited output of a keystroke capturing program running on the author's computer. Some readers may take me to task for being too harsh in these remarks, noting that the book was self-published by the author at age 82. (How do I know it was self-published? Because my copy came with the order from Amazon to the publisher to ship it to their warehouse folded inside, and the publisher's address in this document is directly linked to the author.) Well, call me unkind, but permit me to observe that readers don't get a quality discount based on the author's age from the price of US$34.95, which is on the very high end for a five hundred page paperback, nor is there a disclaimer on the front or back cover that the author might not be firing on all cylinders. Certainly, an eminent retired professor ought to be able to call on former colleagues and/or students to review a manuscript which is certain to become an important part of his intellectual legacy, especially as it attempts to expound a new paradigm for science. Even the most cursory editing to remove needless and tedious repetition could knock 100 pages off this book (and eliminating the misinformation and nonsense could probably slim it down to about ten). The vast majority of citations are to secondary sources, many popular science or new age books. Apart from these drawbacks, Bockris, like many cranks, seems compelled to personally attack Einstein, claiming his work was derivative, hinting at plagiarism, arguing that its significance is less than its reputation implies, and relating an unsourced story claiming Einstein was a poor husband and father (and even if he were, what does that have to do with the correctness and importance of his scientific contributions?). In chapter 2, he rants upon environmental and economic issues, calls for a universal dole (p. 34) for those who do not work (while on p. 436 he decries the effects of just such a dole on Australian youth), calls (p. 57) for censorship of music, compulsory population limitation, and government mandated instruction in philosophy and religion along with promotion of religious practice. Unlike many radical environmentalists of the fascist persuasion, he candidly observes (p. 58) that some of these measures "could not achieved under the present conditions of democracy". So, while repeatedly inveighing against the corruption of government-funded science, he advocates what amounts to totalitarian government—by scientists.
Friday, December 16, 2005
JavaScrypt Compatibility Fix for Mozilla Firefox 1.5The recent release of the Mozilla Firefox 1.5 browser broke the JavaScrypt browser-based cryptography package due to an obscure change in the handling of a String method when applied to an object which is actually an array of Numbers. The reason JavaScrypt (or, more precisely, the AES encryption code upon which it is based) was doing such an odd thing is that the straightforward way to make such a test, the instanceof operator, does not work in browsers prior to Internet Explorer 5 and Netscape 6, so what amounts to a dirty trick ("heuristic", to be polite) was used instead. While I'll admit I was tempted to rewrite the code in question to use instanceof, I strongly dislike torpedoing older hardware and software unless there is absolutely no alternative. In this case, a simple, albeit ugly, fix sufficed, and the new version runs on every browser with which I've tested it, from Netscape 4.7 through current versions of Firefox, Internet Explorer, and Opera. I've also moved the Web pages associated with JavaScrypt closer to HTML 4.01 (Transitional) compliance, but we're not all the way there yet. The problem is that several of the pages use the wrap="off" specification in <textbox> fields to keep long lines (for example, lengthy pass phrases) from being wrapped. You should be able to achieve the same effect with the CSS 2.1 "white-space: pre" specification but, alas, it doesn't seem to work within a <textbox> on most current browsers, so we're stuck with the non-compliant wrap attribute for the time being.
Sunday, December 11, 2005
New Firewall in Production at FourmilabOn Thursday, December 8th at about 15:45 UTC, Fourmilab's Internet connection was switched over to a new firewall configuration. This followed several short-duration live tests in which problems were identified to be fixed in offline testing on a "toy network" with a retired laptop simulating the external Internet. It seems to be in the nature of these projects that, although embarked upon with great hope and enthusiasm, as the time of cut-over to live production approaches, one is afflicted with affright and apprehension. This is particularly the case with the firewall, because once it's installed, to restore full functionality it is necessary to change the IP address of every machine on the local network, and even remembering how to do that on some of the ancient hardware in production here (for example, a Sun SPARCstation 2 with a serial number of less than 10,000 running SunOS 5.5) can be a challenge. Since one of the devices on the local network is a printer shared by all the machines, every machine's printer configuration must be changed as well. All of this means that if the new firewall collapses in some way which requires reverting to the old one, there's a lot more involved than just swapping a few cables. By late last week, I had exhausted all pretexts for further cunctation, so there was nothing for it but to take a deep breath, throw the switch, and see what happened. Here is the configuration as it presently stands, or rather sits on the floor in front of the communications rack to permit easy switching of patch cables back and forth. (Click the image to view an enlargement.) The new firewall and the hardware used to build the toy network for testing it sit atop the cardboard box. At the bottom of the stack are the twin Nokia IP265 security appliances, mounted side by side in a single 1U rack chassis. The IP265 runs an operating system derived from FreeBSD which Nokia calls IPSO, under which the Check Point VPN-1 NG firewall software runs. The Nokia boxes use flash memory instead of a hard drive; the only moving part is the fan. Each has four 10/100 full-duplex Ethernet interfaces, permitting them to support physically separated external, LAN (inside), and DMZ (server farm) networks, while using the fourth network interface for heartbeat and state synchronisation between the active and backup firewalls. Atop the firewalls are the network hubs and switches--mostly retired or spare gear--that were used to set up the toy networks for testing and are now connected to production machines by the cables running up to the patch panel. An old wireless access point on the floor to the right of the box permitted testing of its interaction with the DHCP server running on the firewalls. The laptop on the table simulated a server on the DMZ during the test period. Visible on screen is this spreadsheet, which I've been using to monitor mean HTTP server hits per second on an hourly basis for a week before and after (cells with a blue background) cutover to the new firewall. (There has been a modest decline in hit rate over this period which has nothing to do with the new firewall: I have been deploying increasingly effective versions of Gardol to detect and block packets from the ever growing tsunami of referrer pollution attacks, and dumping the packets from these bozos into the bit bucket before they hit the HTTP server reduces the hit rate computed from its log file.)
The two firewalls, named XL5 and XL6 (yes, I'm a big Gerry Anderson fan, and have every episode on DVD) use the Virtual Router Redundancy Protocol (VRRP) to provide full redundancy. Firewall XL5 is designated the primary and normally responds to the virtual IP addresses of the firewall cluster. If it crashes and ceases to respond correctly to heartbeats from the backup, XL6, or if it detects a fault (for example, loss of link on one of its mission-critical network interfaces) and declares itself down, XL6 immediately takes over the virtual IP addresses and becomes primary. This isn't just a "hot spare" configuration--it's a "sizzling spare" because even while XL6 is serving as the backup, its copy of the firewall software receives continuous state synchronisation messages from XL5, so a fail-over from primary to backup firewall does not interrupt active TCP connections; this can be a real lifesaver if you're running a lengthy full backup from a server on the DMZ to a backup host on the LAN when a hiccup occurs. The propensity of the "3Con" 3CR16110-95 firewalls I'm replacing for crashing during large, high-speed data transfers among networks, combined with their dropping all TCP connections when a fail-over occurs, made full backups a nightmare, forced me to write Valve, and was the primary motivation for the present firewall migration project. The internal network has been reconfigured into physically distinct LAN, DMZ, and external segments. The LAN and DMZ machines are given addresses on the private networks 10.1.x.x and 10.2.x.x respectively, and only servers and a few special purpose machines with specific needs have addresses visible from the outside. All other machines "hide" behind a single address with NAT, and cannot accept connections of any kind initiated from the outside. The only devices connected to the external network are the router on the leased line, the firewalls, and an ePowerSwitch I can use in extremis to power cycle firewalls and switches when I'm off site. The above configuration sounds pretty simple, but when you combine some of the odd things which are done around here with the Byzantine complexity of the Check Point firewall software, which occasionally brings to mind Joe Costello's remark about CATIA: "I've never met a human being who would want to read 17,000 pages of documentation, and if there was, I'd kill him to get him out of the gene pool.", you end up with 18 firewall rules, 27 NAT table entries, and two months of development and pre-production testing. Apart from a few minor speed bumps, however, everything has gone smoothly so far and I anticipate moving the server and load balancer presently held in reserve in case a fall-back is required to the new firewall early next week, restoring full redundancy to the site before my absence during the holidays.
Wednesday, December 7, 2005
Reading List: Friendship 7
- Godwin, Robert ed. Friendship 7: The NASA Mission Reports. Burlington, Ontario, Canada: Apogee Books, 1999. ISBN 1-896522-60-2.
- This installment in the Apogee NASA Mission Reports series contains original pre- and post-flight documents describing the first United States manned orbital flight piloted by John Glenn on February 20th, 1962, including a complete transcript of the air-to-ground communications from launch through splashdown. An excerpt from the Glenn's postflight debriefing describing his observations from space including the "fireflies" seen at orbital sunrise is included, along with a scientific evaluation which, in retrospect, seems to have gotten everything just about right. Glenn's own 13 page report on the flight is among the documents, as is backup pilot Scott Carpenter's report on training for the mission in which he describes the "extinctospectropolariscope-occulogyrogravoadaptometer", abbreviated "V-Meter" in order to fit into the spacecraft (p. 110). A companion CD-ROM includes a one hour NASA film about the mission, with flight day footage from the tracking stations around the globe, and film from the pilot observation camera synchronised with recorded radio communications. An unintentionally funny introduction by the editor (complete with two idiot "it's"-es on consecutive lines) attempts to defend Glenn's 1998 political junket / P.R. stunt aboard socialist space ship Discovery. "If NASA is going to conduct gerontology experiments in orbit, who is more eminently qualified . . . ." Well, a false predicate does imply anything, but if NASA were at all genuinely interested in geezers in space independent of political payback, why didn't they also fly John Young, only nine years Glenn's junior, who walked on the Moon, commanded the first flight of the space shuttle, was Chief of the Astronaut Office for ten years, and a NASA astronaut continuously from 1962 until his retirement in 2004, yet never given a flight assignment since 1983? Glenn's competence and courage needs no embellishment--and the contrast between the NASA in the days of his first flight and that of his second could not be more stark.
Separated at Birth? Saddam and Mad Mel
Saturday, December 3, 2005
Reading List: The New New Left
- Malanga, Steven. The New New Left. Chicago: Ivan R. Dee, 2005. ISBN 1-56663-644-2.
- This thin book (or long essay--the main text is less than 150 pages), argues that urban politics in the United States has largely been captured by an iron triangle of "tax eaters": unionised public employees, staff of government funded social and health services, and elected officials drawn largely from the first two groups and put into office by their power to raise campaign funds, get out the vote, and direct involvement in campaigns due to raw self-interest: unlike private sector voters, they are hiring their own bosses. Unlike traditional big-city progressive politics or the New Left of the 1960s, which were ideologically driven and motivated by a genuine desire to improve the lot of the disadvantaged (even if many of their policy prescriptions proved to be counterproductive in practice), this "new new left" puts its own well-being squarely at the top of the agenda: increasing salaries, defeating attempts to privatise government services, expanding taxpayer-funded programs, and forcing unionisation and regulation onto the private sector through schemes such as "living wage" mandates. The author fears that the steady growth in the political muscle of public sector unions may be approaching or have reached a tipping point--where, albeit not yet a numerical majority, through their organised clout they have the power to elect politicians beholden to them, however costly to the productive sector or ultimately disastrous for their cities, whose taxpayers and businesses may choose to vote with their feet for places where they are viewed as valuable members of the community rather than cash cows to be looted. Chapter 5 dismantles Richard Florida's crackpot "Creative Class" theory, which argues that by taxing remaining workers and businesses even more heavily and spending the proceeds on art, culture, "diversity", bike paths, and all the other stuff believed to attract the golden children of the dot.com bubble, rust belt cities already devastated by urban socialism can be reborn. Post dot.bomb, such notions are more worthy of a belly laugh than thorough refutation, but if it's counter-examples and statistics you seek, they're here. The last three chapters focus almost entirely on New York City. I suppose this isn't surprising, both because New York is often at the cutting edge in urban trends in the U.S., and also because the author is a senior fellow at the Manhattan Institute and a contributing editor to its City Journal, where most of this material originally appeared.
Friday, December 2, 2005
Internet Slum: Referrer Pollution AttacksThanks to a wonderfuly insightful feedback message from Christopher Masto, who read the original report of what appeared at the time to be an odd distributed denial of service, attack, I now believe I understand what is going on. The symptom remains the same: large numbers of identical Web server hits from a variety of sites (disproportionately in locations known to be the home of many spammers), each requesting the same Webalizer server status page and specifying as the referrer (the URL of the page supposedly containing the link to the document being fetched) what appear to be pages constructed to attract search engines to lists of commercial sites which, in fact, contain no link to the status page being requested. As I'm writing this, for example, two sites, one in Russia and the other in the Ukraine, are pumping in such HTTP requests at the rate of several packets per second, even though a version of Gardol configured to recognise the attack is is dispatching them directly to the bit bucket with iptables. (I've edited the following iptables status report to make it fit on the page, and hidden the second byte of the offending IP addresses to avoid giving them free publicity.)
Chain INPUT (policy ACCEPT 2335M packets, 200G bytes) pkts bytes target prot in out source 53014 2629K DROP tcp * * 81.x.8.26 35965 1774K DROP tcp * * 195.x.176.138What these sites appear to be doing is to crawl sites with a high page rank in Google and other search engines (Fourmilab has a very high page rank for some common and hence presumably valuable queries such as "earth", "sky", and "diet") looking for Webalizer-generated statistics pages. If they find them, they start blasting in zillions of hits on those pages, sometimes (but not always) not even bothering to read all the result data from the TCP connection, which causes the out of state TCP messages from the firewall which alerted me to this in the first place. The hits on the statistics pages, in turn, all specify as a referrer "Search engine poison" pages such as:
barbie-com.natobr.org.grapewhich are filled with obviously mechanically generated keywords and links to other pages which go to advertisers. (Like everything on the Internet, many such pages are pornographic or drug pedlars; I have listed only G rated exemplars here, although you never know where you may end up if you follow the links in them. To avoid giving these scum precisely what they're looking for—a link from my site, I have not provided a link or complete URL, and I have added the name of a fruit after each top level domain name. If you wish to see the content of one of these sites, remove the fruit and add the HTTP protocol specifier at the start.) But why all the hits on the status pages, you ask? Well, most sites that run Webalizer use the default configuration, which includes a list of the top 30 referrers by URL. If, by flooding your site with requests, they can work their way into that list, then when your site status page is crawled by Googlebot, the referer link will be seen as a link from a highly ranked site to their trash, which is believed to drastically boost their own page ranking. The more highly ranked sites they pollute this way, the higher their own rank rises. You can see the sites they've hit by doing a Google search for these URLs and noting that they're almost all in referrer statistics. Now, like the more conventional E-mail spammers, these guys are vandals who don't care in the least how much network bandwidth they squander and how much congestion they create on the outbound Internet connections of the victim sites they pound. They don't have to hit the statistics pages themselves, which tend to be large for active sites (mine are about 100K)—they could request something tiny and have the same effect on the referrer statistics. But ever more moronic is that before piling on a site they don't seem to check /robots.txt for a Web crawler restriction on the statistics pages such as mine:
User-agent: * Disallow: /serverstats/or for a document-specific exclusion in the statistics page itself, as the one I include:
<meta name="robots" content="noindex,nofollow">either of which will cause search engine Web crawlers to ignore the page. So all of the pounding on my site is of absolutely no benefit whatsoever to the arthropod apes who are doing it—even if they did manage to get their cretinous URLs into the list of top referrers (which they did before I configured Gardol to drop their packets), it won't help the page rank of their sites because Google, Yahoo, MSN and the rest don't index the statistics pages due to the robots exclusion. Webalizer does not by default, however, include the "noindex" declaration, so to deter these bozos and reduce the probability of an attack, it's wise to include statistics pages in /robots.txt and/or add the requisite <meta> tag to each of them them with a declaration like:
HTMLHead <meta name="robots" content="noindex,nofollow">in the webalizer.conf file for the site. All of this goes to show that when adding automatically generated content to a Web site, you should be as paranoid as Perl in exposing potentially "tainted" data from the outside. Even something as obscure as a list of top referrers in a server statistics page may be used as a billboard to promote somebody else's site, and your site subjected to callous abuse in order to so pollute it.