- Walton, David.
Three Laws Lethal.
Jersey City, NJ: Pyr, 2019.
ISBN 978-1-63388-560-8.
-
In the near future, autonomous vehicles, “autocars”,
are available from a number of major automobile manufacturers.
The self-driving capability, while not infallible, has been
approved by regulatory authorities after having demonstrated
that it is, on average, safer than the population of human
drivers on the road and not subject to human frailties such as
driving under the influence of alcohol or drugs, while tired, or
distracted by others in the car or electronic gadgets. While
self-driving remains a luxury feature with which a minority of
cars on the road are equipped, regulators are confident that as
it spreads more widely and improves over time, the highway
accident rate will decline.
But placing an algorithm and sensors in command of a vehicle
with a mass of more than a tonne hurtling down the road at 100
km per hour or faster is not just a formidable technical
problem, it is one with serious and unavoidable moral
implications. These come into stark focus when, in an incident
on a highway near Seattle, an autocar swerves to avoid a tree
crashing down on the highway, hitting and killing a motorcyclist
in an adjacent lane of which the car's sensors must have been
aware. The car appears to have made a choice, valuing
the lives of its passengers: a mother and her two children, over
that of the motorcyclist. What really happened, and how the car
decided what to do in that split-second, is opaque, because the
software controlling it was, as all such software, proprietary
and closed to independent inspection and audit by third
parties. It's one thing to acknowledge that self-driving
vehicles are safer, as a whole, than those with humans behind
the wheel, but entirely another to cede to them the moral agency
of life and death on the highway. Should an autocar value the
lives of its passengers over those of others? What if there
were a sole passenger in the car and two on the motorcycle? And
who is liable for the death of the motorcyclist: the auto
manufacturer, the developers of the software, the owner of car,
the driver who switched it into automatic mode, or the
regulators who approved its use on public roads? The case was
headed for court, and all would be watching the precedents it
might establish.
Tyler Daniels and Brandon Kincannon, graduate students in the
computer science department of the University of Pennsylvania,
were convinced they could do better. The key was going beyond
individual vehicles which tried to operate autonomously based
upon what their own sensors could glean from their immediate
environment, toward an architecture where vehicles communicated
with one another and coordinated their activities. This would
allow sharing information over a wider area and be able to avoid
accidents resulting from individual vehicles acting without the
knowledge of the actions of others. Further, they wanted to
re-architect individual ground transportation from a model of
individually-owned and operated vehicles to transportation as a
service, where customers would summon an autocar on demand with
their smartphone, with the vehicle network dispatching the
closest free car to their location. This would dramatically
change the economics of personal transportation. The typical private
car spends twenty-two out of twenty-four hours parked, taking up
a parking space and depreciating as it sits idle. The
transportation service autocar would be in constant service
(except for downtime for maintenance, refuelling, and times of
reduced demand), generating revenue for its operator. An angel
investor believes their story and, most importantly, believes in
them sufficiently to write a check for the initial demonstration
phase of their project, and they set to work.
Their team consists of Tyler and Brandon, plus Abby and Naomi
Sumner, sisters who differed in almost every way: Abby outgoing
and vivacious, with an instinct for public relations and
marketing, and Naomi the super-nerd, verging on being “on
the spectrum”. The big day of the public roll-out of the
technology arrives, and ends in disaster, killing Abby in what
was supposed to be a demonstration of the system's inherent
safety. The disaster puts an end to the venture and the
surviving principals go their separate ways. Tyler signs on as
a consultant and expert witness for the lawyers bringing the
suit on behalf of the motorcyclist killed in Seattle, using the
exposure to advocate for open source software being a
requirement for autonomous vehicles. Brandon uses money
inherited after the death of his father to launch a new venture,
Black Knight, offering transportation as a service initially in
the New York area and then expanding to other cities. Naomi,
whose university experiment in genetic software implemented as
non-player characters (NPCs) in a virtual world was the
foundation of the original venture's software, sees Black Knight
as a way to preserve the world and beings she has created as
they develop and require more and more computing resources.
Characters in the virtual world support themselves and compete
by driving Black Knight cars in the real world, and as
generation follows generation and natural selection works its
wonders, customers and competitors are amazed at how Black
Knight vehicles anticipate the needs of their users and maintain
an unequalled level of efficiency.
Tyler leverages his recognition from the trial into a new
self-driving venture based on open source software called
“Zoom”, which spreads across the U.S. west coast and
eventually comes into competition with Black Knight in the
east. Somehow, Zoom's algorithms, despite being open and having
a large community contributing to their development, never seem
able to equal the service provided by Black Knight, which is so
secretive that even Brandon, the CEO, doesn't know how Naomi's
software does it.
In approaching any kind of optimisation problem such as
scheduling a fleet of vehicles to anticipate and respond to
real-time demand, a key question is choosing the
“objective function”: how the performance of the
system is evaluated based upon the stated goals of its
designers. This is especially crucial when the optimisation is
applied to a system connected to the real world. The parable of
the
“Clippy
Apocalypse”, where an artificial intelligence put in
charge of a paperclip factory and trained to maximise the
production of paperclips escapes into the wild and eventually
converts first its home planet, then the rest of the solar
system, and eventually the entire visible universe into paper
clips. The system worked as designed—but the objective
function was poorly chosen.
Naomi's NPCs literally (or virtually) lived or died based upon
their ability to provide transportation service to Black
Knight's customers, and natural selection, running at the
accelerated pace of the simulation they inhabited, relentlessly
selected them with the objective of improving their service and
expanding Black Knight's market. To the extent that, within
their simulation, they perceived opposition to these goals, they
would act to circumvent it—whatever it takes.
This sets the stage for one of the more imaginative tales of how
artificial general intelligence might arrive through the back
door: not designed in a laboratory but emerging through the
process of evolution in a complex system subjected to real-world
constraints and able to operate in the real world. The moral
dimensions of this go well beyond the trolley
problem often cited in connection with autonomous vehicles,
dealing with questions of whether artificial intelligences we
create for our own purposes are tools, servants, or slaves, and
what happens when their purposes diverge from those for which we
created them.
This is a techno-thriller, with plenty of action in the
conclusion of the story, but also a cerebral exploration of the
moral questions which something as seemingly straightforward and
beneficial as autonomous vehicles may pose in the future.
December 2019 