Books by Ward, Jonathan H.

Leinbach, Michael D. and Jonathan H. Ward. Bringing Columbia Home. New York: Arcade Publishing, [2018] 2020. ISBN 978-1-948924-61-0.

Author Michael Leinbach was Launch Director at the Kennedy Space Center when space shuttle orbiter Columbia was lost during its return to Earth on February 1st, 2003. In this personal account, he tells the story of locating, recovering, and reconstructing the debris from the orbiter, searching for and finding the remains of the crew, and learning the lessons, technical and managerial, from the accident.

April 2020 

Ward, Jonathan H. Countdown to a Moon Launch. Cham, Switzerland: Springer International, 2015. ISBN 978-3-319-17791-5.

In the companion volume, Rocket Ranch (December 2015), the author describes the gargantuan and extraordinarily complex infrastructure which was built at the Kennedy Space Center (KSC) in Florida to assemble, check out, and launch the Apollo missions to the Moon and the Skylab space station. The present book explores how that hardware was actually used, following the “processing flow” of the Apollo 11 launch vehicle and spacecraft from the arrival of components at KSC to the moment of launch.

As intricate as the hardware was, it wouldn't have worked, nor would it have been possible to launch flawless mission after flawless mission on time had it not been for the management tools employed to coordinate every detail of processing. Central to this was PERT (Program Evaluation and Review Technique), a methodology developed by the U.S. Navy in the 1950s to manage the Polaris submarine and missile systems. PERT breaks down the progress of a project into milestones connected by activities into a graph of dependencies. Each activity has an estimated time to completion. A milestone might be, say, the installation of the guidance system into a launch vehicle. That milestone would depend upon the assembly of the components of the guidance system (gyroscopes, sensors, electronics, structure, etc.), each of which would depend upon their own components. Downstream, integrated test of the launch vehicle would depend upon the installation of the guidance system. Many activities proceed in parallel and only come together when a milestone has them as its mutual dependencies. For example, the processing and installation of rocket engines is completely independent of work on the guidance system until they join at a milestone where an engine steering test is performed.

As a project progresses, the time estimates for the various activities will be confronted with reality: some will be completed ahead of schedule while other will slip due to unforeseen problems or over-optimistic initial forecasts. This, in turn, ripples downstream in the dependency graph, changing the time available for activities if the final completion milestone is to be met. For any given graph at a particular time, there will be a critical path of activities where a schedule slip of any one will delay the completion milestone. Each lower level milestone in the graph has its own critical path leading to it. As milestones are completed ahead or behind schedule, the overall critical path will shift. Knowing the critical path allows program managers to concentrate resources on items along the critical path to avoid, wherever possible, overall schedule slips (with the attendant extra costs).

Now all this sounds complicated, and in a project with the scope of Apollo, it is almost bewildering to contemplate. The Launch Control Center was built with four firing rooms. Three were outfitted with all of the consoles to check out and launch a mission, but the fourth cavernous room ended up being used to display and maintain the PERT charts for activities in progress. Three levels of charts were maintained. Level A was used by senior management and contained hundreds of major milestones and activities. Each of these was expanded out into a level B chart which, taken together, tracked in excess of 7000 milestones. These, in turn, were broken down into detail on level C charts, which tracked more than 40,000 activities. The level B and C charts were displayed on more than 400 square metres of wall space in the back room of firing room four. As these detailed milestones were completed on the level C charts, changes would propagate down that chart and those which affected its completion upward to the level A and B charts.

Now, here's the most breathtaking thing about this: they did it all by hand! For most of the Apollo program, computer implementations of PERT were not available (or those that existed could not handle this level of detail). (Today, the PERT network for processing of an Apollo mission could be handled on a laptop computer.) There were dozens of analysts and clerks charged with updating the networks, with the processing flow displayed on an enormous board with magnetic strips which could be shifted around by people climbing up and down rolling staircases. Photographers would take pictures of the board which were printed and distributed to managers monitoring project status.

If PERT was essential to coordinating all of the parallel activities in preparing a spacecraft for launch, configuration control was critical to ensure than when the countdown reached T0, everything would work as expected. Just as there was a network of dependencies in the PERT chart, the individual components were tested, subassemblies were tested, assemblies of them were tested, all leading up to an integrated test of the assembled launcher and spacecraft. The successful completion of a test established a tested configuration for the item. Anything which changed that configuration in any way, for example unplugging a cable and plugging it back in, required re-testing to confirm that the original configuration had been restored. (One of the pins in the connector might not have made contact, for instance.) This was all documented by paperwork signed off by three witnesses. The mountain of paper was intimidating; there was even a slide rule calculator for estimating the cost of various kinds of paperwork.

With all of this management superstructure it may seem a miracle that anything got done at all. But, as the end of the decade approached, the level of activity at KSC was relentless (and took a toll upon the workforce, although many recall it as the most intense and rewarding part of their careers). Several missions were processed in parallel: Apollo 11 rolled out to the launch pad while Apollo 10 was still en route to the Moon, and Apollo 12 was being assembled and tested.

To illustrate how all of these systems and procedures came together, the author takes us through the processing of Apollo 11 in detail, starting around six months before launch when the Saturn V stages, and command, service, and lunar modules arrived independently from the contractors who built them or the NASA facilities where they had been individually tested. The original concept for KSC was that it would be an “operational spaceport” which would assemble pre-tested components into flight vehicles, run integrated system tests, and then launch them in an assembly-line fashion. In reality, the Apollo and Saturn programs never matured to this level, and were essentially development and test projects throughout. Components not only arrived at KSC with “some assembly required”; they often were subject to a blizzard of engineering change orders which required partially disassembling equipment to make modifications, then exhaustive re-tests to verify the previously tested configuration had been restored.

Apollo 11 encountered relatively few problems in processing, so experiences from other missions where problems arose are interleaved to illustrate how KSC coped with contingencies. While Apollo 16 was on the launch pad, a series of mistakes during the testing process damaged a propellant tank in the command module. The only way to repair this was to roll the entire stack back to the Vehicle Assembly Building, remove the command and service modules, return them to the spacecraft servicing building then de-mate them, pull the heat shield from the command module, change out the tank, then put everything back together, re-stack, and roll back to the launch pad. Imagine how many forms had to be filled out. The launch was delayed just one month.

The process of servicing the vehicle on the launch pad is described in detail. Many of the operations, such as filling tanks with toxic hypergolic fuel and oxidiser, which burn on contact, required evacuating the pad of all non-essential personnel and special precautions for those engaged in these hazardous tasks. As launch approached, the hurdles became higher: a Launch Readiness Review and the Countdown Demonstration Test, a full dress rehearsal of the countdown up to the moment before engine start, including fuelling all of the stages of the launch vehicle (and then de-fuelling them after conclusion of the test).

There is a wealth of detail here, including many obscure items I've never encountered before. Consider “Forward Observers”. When the Saturn V launched, most personnel and spectators were kept a safe distance of more than 5 km from the launch pad in case of calamity. But three teams of two volunteers each were stationed at sites just 2 km from the pad. They were charged with observing the first seconds of flight and, if they saw a catastrophic failure (engine explosion or cut-off, hard-over of an engine gimbal, or the rocket veering into the umbilical tower), they would signal the astronauts to fire the launch escape system and abort the mission. If this happened, the observers would then have to dive into crude shelters often frequented by rattlesnakes to ride out the fiery aftermath.

Did you know about the electrical glitch which almost brought the Skylab 2 mission to flaming catastrophe moments after launch? How lapses in handling of equipment and paperwork almost spelled doom for the crew of Apollo 13? The time an oxygen leak while fuelling a Saturn V booster caused cars parked near the launch pad to burst into flames? It's all here, and much more. This is an essential book for those interested in the engineering details of the Apollo project and the management miracles which made its achievements possible.

January 2016 

Ward, Jonathan H. Rocket Ranch. Cham, Switzerland: Springer International, 2015. ISBN 978-3-319-17788-5.

Many books have been written about Project Apollo, with a large number devoted to the lunar and Skylab missions, the Saturn booster rockets which launched them, the Apollo spacecraft, and the people involved in the program. But none of the Apollo missions could have left the Earth without the facilities at the Kennedy Space Center (KSC) in Florida where the launch vehicle and space hardware were integrated, checked out, fuelled, and launched. In many ways, those facilities were more elaborate and complicated than the booster and spacecraft, and were just as essential in achieving the record of success in Saturn and Apollo/Saturn launches. NASA's 1978 official history of KSC Apollo operations, Moonport (available on-line for free), is a highly recommended examination of the design decisions, architecture, management, and operation of the launch site, but it doesn't delve into the nitty-gritty of how the system actually worked.

The present book, subtitled “The Nuts and Bolts of the Apollo Moon Program at Kennedy Space Center” provides that detail. The author's research involved reviewing more than 1200 original documents and interviewing more than 70 people, most veterans of the Apollo era at KSC (many now elderly). One thread that ran through the interviews is that, to a man (and almost all are men), despite what they had done afterward, they recalled their work on Apollo, however exhausting the pace and formidable the challenges, as a high point in their careers. After completing his research, Ward realised he was looking at a 700 page book. His publisher counselled that such a massive tome would be forbidding to many readers. He decided to separate the description of the KSC hardware (this volume) and the operations leading up to a launch (described in the companion title, Countdown to a Moon Launch, which I will review in the future).

The Apollo/Saturn lunar flight vehicle was, at the time, the most complex machine ever built by humans. It contained three rocket stages (all built by different contractors), a control computer, and two separate spacecraft: the command/service modules and lunar module, each of which had their own rocket engines, control thrusters, guidance computers, and life support systems for the crew. From the moment this “stack” left the ground, everything had to work. While there were redundant systems in case of some in-flight failures, loss of any major component would mean the mission would be unsuccessful, even if the crew returned safely to Earth.

In order to guarantee this success, every component in the booster and spacecraft had to be tested and re-tested, from the time it arrived at KSC until the final countdown and launch. Nothing could be overlooked, and there were written procedures which were followed for everything, with documentation of each step and quality inspectors overseeing it all. The volume of paperwork was monumental (a common joke at the time was that no mission could launch until the paperwork weighed more than the vehicle on the launch pad), but the sheer complexity exceeded the capabilities of even the massive workforce and unlimited budget of Project Apollo. KSC responded by pioneering the use of computers to check out the spacecraft and launcher at every step in the assembly and launch process. Although a breakthrough at the time, the capacity of these computers is laughable today. The computer used to check out the Apollo spacecraft had 24,576 words of memory when it was installed in 1964, and programmers had to jump through hoops and resort to ever more clever tricks to shoehorn the test procedures into the limited memory. Eventually, after two years, approval was obtained to buy an additional 24,000 words of memory for the test computers, at a cost of almost half a million 2015 dollars.

You've probably seen pictures of the KSC firing room during Apollo countdowns. The launch director looked out over a sea of around 450 consoles, each devoted to one aspect of the vehicle (for example, console BA25, “Second stage propellant utilization”), each manned by an engineer in a white shirt and narrow tie. These consoles were connected into audio “nets”, arranged in a hierarchy paralleling the management structure. For example, if the engineer at console BA25 observed something outside acceptable limits, he would report it on the second stage propulsion net. The second stage manager would then raise the issue on the launch vehicle net. If it was a no-go item, it would then be bumped up to the flight director loop where a hold would be placed on the countdown. If this wasn't complicated enough, most critical parameters were monitored by launch vehicle and spacecraft checkout computers, which could automatically halt the countdown if a parameter exceeded limits. Most of those hundreds of consoles had dozens of switches, indicator lights, meters, and sometimes video displays, and all of them had to be individually wired to patchboards which connected them to the control computers or, in some cases, directly to the launch hardware. And every one of those wires had to have a pull ticket for its installation, and inspection, and an individual test and re-test that it was functioning properly. Oh, and there were three firing rooms, identically equipped. During a launch, two would be active and staffed: one as a primary, the other as a backup.

The level of detail here is just fantastic and may be overwhelming if not taken in small doses. Did you know, for example, that in the base of the Saturn V launch platform there was an air conditioned room with the RCA 110A computer which checked out the booster? The Saturn V first stage engines were about 30 metres from this delicate machine. How did they keep it from being pulverised when the rocket lifted off? Springs.

Assembled vehicles were transported from the Vehicle Assembly Building to the launch pad by an enormous crawler. The crawler was operated by a crew of 14, including firemen stationed near the diesel engines. Originally, there was an automatic fire suppression system, but after it accidentally triggered and dumped a quarter ton of fire suppression powder into one of the engines during a test, it was replaced with firemen. How did they keep the launcher level as it climbed up the ramp to the pad? They had two pipes filled with mercury which ran diagonally across the crawler platform between each pair of corners. These connected to a sight glass which indicated to the operator if the platform wasn't level. Then the operator would adjust jacking cylinders on the corners to restore the platform to level—while it was rolling.

I can provide only a few glimpses of the wealth of fascinating minutæ on all aspects of KSC facilities and operations described here. Drawing on his more than 300 hours of interviews, the author frequently allows veterans of the program to speak in their own words, giving a sense of what it was like to be there, then, the rationale for why things were done the way they were, and to relate anecdotes about when things didn't go as planned.

It has been said that one of the most difficult things NASA did in Project Apollo was to make it look easy. Even space buffs who have devoured dozens of books about Apollo may be startled by the sheer magnitude of what was accomplished in designing, building, checking out, and operating the KSC facilities described in this book, especially considering in how few years it all was done and the primitive state of some of the technologies available at the time (particularly computers and electronics). This book and its companion volume are eye-openers, and only reinforce what a technological triumph Apollo was.

December 2015