With great relief I arrived at O'Hare airport, knowing that my next
plane ride would bring me home. I drove out the I-88 toll road,
past ATT's Bell Laboratories, to the little town of Warrenville.
When I first moved to Warrenville in 1969, the town of a few thousand people was in the middle of farm country.
Robert Wilson, the legendary high energy physicist had gathered
many of the best experimental physicists in the world to 6,800 acres
of prairie to make the largest accelerator ever built. Over the years,
the accelerators they constructed became capable of smashing protons against antiprotons at the incredible energy of 1.8 Trillion Electron Volts (TEV).
Collisions of subatomic particles at high energies help reveal information about quarks and leptons, the fundamental forms of matter. Acting like a giant microscope, the Fermilab accelerator used
higher energies to create and discover particles unobservable at
lower energies.
In the mid-1970s, the accelerator provided evidence of the existence of the upsilon, a subatomic particle composed of a bottom
quark and an anti-bottom quark, a discovery that provided strong
support for the Standard Model. In the 1990s, the lab was hot on
the trail of the top quark, the last quark yet to be observed experimentally.
When I moved to Warrenville, though, there was nothing but
prairie where Fermilab now stands. My father and the other physicists did their work in a suburban office complex until they could
start using houses in a subdevelopment on the site that had been
purchased under eminent domain. Old farmhouses from the countryside were moved together into a cluster, to house visiting scientists. The subdevelopment was painted bright primary colors, and a
chef was brought in to staff the village cafeteria (although the food
was lousy, at least there was a chef).
At that time, Fermilab was just beginning to coalesce. While
everybody slaved away building the accelerator, Wilson continued
to put in those unique features that made the lab such a special
place. A herd of
buffalo was brought in and a project was started in
the middle of the 4-mile circumference main ring to restore it to the
original prairie grasses. One of the lab buildings was even constructed as a geodesic dome, with the panels of the dome made out
of panes of plastic, sandwiching thousands and thousands of aluminum cans, collected in a recycling program by the neighboring
schools.
My visit to Fermilab on Friday had been set up over the past
two months via electronic mail. Around Singapore I had received a
message from a lab public relations engineer asking if we might
make my visit a two-way exchange of information.
Of course, I wrote back, two-way is good. By the time I hit Amsterdam, I had another message from the lab wanting to know if I
would be discussing the future of networking on my visit. With
another 100 messages to plow through, I dashed a hurried (but puzzled) acknowledgment.
Arriving at the lab, I saw that the little two-way exchange had
blossomed into a full-blown public seminar with the title "The Future of Networking." A nice topic, no doubt, but one best left to
astrologers and marketing analysts.
I asked around to see if perhaps some physicist had spoken on
"The Future of Cosmology" in a recent seminar, and perhaps I was
just part of some series. No such luck, so I took advantage of the
rule that states the one with the white board gets to set the agenda
and changed my topic to a more manageable one.
After my talk, I sat down with some of the Fermilab networking
staff, including the managers of the U.S. High Energy Physics network (HEPnet). HEPnet ties physicists in dozens of countries to the
major world physics laboratories, such as Fermilab, CERN in Geneva, NIKHEF in Holland, KEK in Japan, and many, many, other
sites.
HEPnet started as a physical network, a microwave link between
Lawrence Berkeley Laboratory and the Stanford Linear Accelerator
Center (SLAC). It was based on DECnet protocols, a natural choice
considering the overwhelming use of PDP and VAX computers in
the physics community.
The network quickly grew, using the big laboratories as hubs to
run tail circuits to regional (and sometimes not so regional) research
universities. CERN and Fermilab were two of the more aggressive
hubs, bringing some of the first network lines into places like China,
South America, and Eastern Europe.
Connecting the hubs together were the lines that made up the
HEPnet backbone. While some of the tail sites were strictly DECnet,
many of the backbone sites began using multiprotocol routers, letting HEPnet finance a part of the line and having the other user
communities fund the rest.
Many of the universities at the ends of the tail circuits began
joining the Internet, and the HEPnet lines became one of many possible paths, or often just a virtual network layered on some other
service provider.
Gradually, HEPnet grew from a single protocol network to a
multiprotocol one, and from a physical network to a shared line use
model. The shared line use just made HEPnet a convenient label for
the bundle of resources that the high energy physics community
was purchasing for its users.
In the U.S., HEPnet was partially swallowed up by a larger network run by the Department of Energy, the Energy Sciences network (ESnet). Rather than lease many lines between labs for
individual communities, ESnet would furnish the backbone.
Although HEPnet relinquished the role of backbone, it still furnished quite a few tail sites, particularly medium-speed links to
places that ESnet did not serve, such as Brazil, or those sites that
need special high-speed dedicated capacities, such as the physics
departments of many large universities. In addition to managing
the dedicated links, the HEPnet group was trying to move towards
providing higher level services, such as running a mail gateway and
starting video conferences.
In addition to the HEPnet WAN links, the Fermilab Computer
Division had to manage a vast array of internal resources, including
what was reputed to be the largest VAX Cluster in the world.
Workstations are a dime a dozen, and large Amdahls provide local
mainframe support.
The network architecture at Fermilab was originally, as can be
expected by the DECophilic physicists, set up as a huge extended
Ethernet running protocols like DECnet and LAT. Over time,
though, UNIX-based workstations made TCP/IP an important part
of the network.
Routers were introduced to segment the traffic into subnetworks. With some of the Local Area VAX Clusters (LAVCs) having
as many as 60 workstations as clients, segmenting the network certainly made management easier.
The biggest concern, though, was how to continue migrating to
higher and higher bandwidth LANs. Fermilab is the kind of place
that will always be using the latest equipment in all sorts of custom
configurations. To accommodate FDDI and any future LANs, the
laboratory laid bundles of 48 single and multimode fibers around
the ring.
All that fiber around the lab meant that in addition to one or
more lab backbones, the Computer Division could take a fiber to an
experiment or workgroup, thereby creating rings on demand. Like
many sites, Fermilab was trying hard to go through the exercise of
laying fiber in a large area only once.
After a quick lunch, I went into the building housing the
Linear
Accelerator (LINAC), the first of the seven accelerators linked together at the laboratory. Behind racks and racks of electronic equipment, I found the office of Peter Lucas, one of the designers of the
accelerator control system.
Controlling a series of accelerators involves monitoring and setting a huge number of devices on subsystems ranging from vacuum
pumps to refrigeration units to superconducting magnets. In the
case of Fermilab, the accelerator control system must handle over
45,000 data points, some of which operate at very high frequencies.
Most devices are managed by CAMAC crates, a venerable backplane architecture that can hold up to 22 cards per crate. CAMAC
crates are controlled by serial links, which in turn plug into a computer, usually a PDP 11.
The computer can poll each of the cards on the crate to gather
and set data. If a card needs to generate an interrupt, it can set a
single "look at me" bit on the crate. The computer then polls each
of the cards to find out which ones need attention. The basic control flow on the front end was thus a PDP 11 to a CAMAC crate to a
card to the device being controlled (although the PDPs were scheduled to be swapped out at some point for µVAXen).
In the early 1980s, when this system was first being put together,
VAXen were just being released. A series of 311/785s, later supplemented by an 8650, acted as database servers, keeping track of information on each of the data points, such as which CAMAC and
PDP that data point was accessible from.
The last part of the system were µVAXen (originally PDPs) acting as consoles. The basic operation would be for a console to log a
request for a certain data element at a certain frequency. The database server would define the elements and the PDP 11s were responsible for delivering the data.
So how to connect all these systems together? In 1986, when the
current network was being designed, the great Ethernet versus Token Ring debate was raging. After extensive deliberations, the lab
decided the predictable latency of the token ring was needed for the
main ring control system.
DEC, of course, was firmly in the Ethernet camp, but lack of
product support by the vendor doesn't faze experimental physicists.
They turned to their electrical engineers, who built token ring cards
for the Unibus used by the PDPs and VAXen.
Running on top of the token ring, the lab put their own Accelerator Control network (ACnet). ACnet allows any console to monitor and manage any device, a capability that makes operators at
other accelerator centers stuck with special purpose consoles green
with envy.
ACnet is a protocol in which a console logs a single request,
which can generate a stream of data. A request might be for a certain data item at 1 hz, which would yield a stream of data spaced
one per second. In addition to logged requests, certain events trigger alarms, sending the front-end PDP into the database to decide
who should be notified.
The token ring choice worked, but it turned out that there really
wasn't that much difference between it and Ethernet. Not only that,
DEC kept coming out with new µVAXen, each sporting a newly incompatible variant on the Q-bus. It was evident that Ethernet had
to be incorporated into the network somehow.
An Ethernet was added and VAXstations started to pop up
around the laboratory as consoles. To link the Ethernet with ACnet,
the lab designed and wrote an Ethernet-to-token ring bridge running in software on one or more of the VAXen.
The ACnet was still token ring, but the addition of the bridge
meant that authorized users from all over the lab had access to accelerator data. One of the VAXen was even set up with the X Window System, allowing X servers to access the data.
Since X runs just fine on top of the Internet, especially if you are
communicating between two core sites, it is even possible to run the
accelerator remotely. On one occasion, lab employees at a conference in Japan pulled up the accelerator control panels. To prevent
any possibility of accelerator hackers, the lab set up the system so
that remote terminals could only read data items.
This architecture proved to be quite flexible, easily expanding to
meet the needs of the
Tevatron
and other newer subsystems. PDPs
controlling CAMAC crates could easily be added if new data points
needed to be controlled. Over time, CAMAC crates became a bit
old-fashioned and newer devices simply had a VME crate directly
on the token ring, alleviating the need for the front-end PDP.
Even the token ring decision turned out to have been wisely
made. Networking at the lab grew like a prairie fire, with workstations going in as fast as physicists could unpack them. This is not a
user community that will wait for official installers to unpack boxes.
In fact, if pieces are missing, they are just as liable to manufacture a
replacement themselves as they are to call up the shipper and complain.
After walking me through the architecture, Peter Lucas took me
over to an X Window System console and showed me some of the
1,000 application packages that had been written for the accelerator.
All the screens were color coded, providing visual clues on
which items in a complex form led to other screens and which items
could be changed by the user. Clicking on a line item with the ID
for a CAMAC crate, for example, would result in a graphic representation of the map, with each card and its ID depicted. Clicking
on one of the cards would show the data items kept by the card and
its current status.
All these programs were written at first for PDPs, in a day when
that level of user interface was real, heavy-duty programming.
These were the kind of programmers who battled their way through
RSX, FORTRAN 2, and TECO, languages that make me shudder
whenever I think of them.
When the superconducting magnets in the main ring generate proton-antiproton collisions at 1.8 TEV, there needs to be a way to collect data describing the event and store it away for further analysis.
The detectors used in high energy physics are as complicated as the
accelerator itself.
I drove around the main ring to
D Zero (DØ), the quadrant of
the ring containing a detector that had been under construction for
over nine years. The instrument had consumed most of the time of
400 physicists and the U.S. $50 million camera was almost ready to
be inserted into the ring.
DØ was a six-story building, just large enough to house the detector. When it was operational, the detector would put out 100,000
data points per collision, and there would be a collision every 3.5
microseconds. With roughly 250 kbytes per collision, this results in
a data rate of 75 Gbytes per second.
This is quite a bit of data, especially when you consider the fact
that the experiment would run 24 hours per day for six to nine
months at a time. One of the great hopes of the DØ detector was
that it would provide experimental evidence of the existence of the
top quark, the only one not yet empirically observed.
Top quarks are very rare, and it was quite likely that only one
(or less) would be found per day. In a day of operation, the data
acquisition subsystem on the detector thus had to somehow filter
through 6.48 petabytes (6.48 million gigabytes) looking for the interesting 250,000 bytes.
DØ was pandemonium. With the pins being pulled in a few
days to roll the detector in, desks were littered with piles of empty
styrofoam cups and coke cans, boards were everywhere in various
stages of assembly or disassembly. A pager went off every few minutes, directing people to where they weren't.
I found Dean Schaumberger, a professor from State University of
New York at Stony Brook who agreed to give me a quick briefing.
We walked through the data acquisition system, concentrating on
the data and ignoring the physics. To decide which events to keep,
an event mask provided the first level of filtering, allowing the
physicists to hone in on a few dozen attributes that characterize interesting events. Once the event mask was tripped, boards housed
in 79 VME-based front end crates would digitize the data on the
detector. The goal of the mask was to reduce the data flow to 200 to
400 events per second.
Since it was possible that two interesting events could occur
back to back, the front-end boards were double buffered. After the
two buffers were full, though, events that tripped the mask had to
be discarded until space became available again.
The 79 front-end crates were distributed among eight data cables, each cable providing a 32-bit wide data path running at 10
Mhz, yielding a data rate of 40 Mbytes per cable. With all eight
cables running in parallel, the data flow reached 320 Mbytes per
second.
The next level of analysis was provided by a farm of 50 µVAX
4000s, each machine running at the rate of about 12 times a VAX
11/780. Each µVAX had 8 custom boards, one for each of the 8 data
cables. The data cables thus came out of the VME Gates and
snaked through each of the µVAXen allowing any machine to collect
all the data on one event.
The µVAXen, running the ELN real-time operating system, were
also connected to a scheduling VAX, a machine that decided where
on the farm there were available cycles to process the next event.
With the major pieces in place, Dean walked me through the
data path. When the mask got tripped, data would go into the
front-end boards. If the data bus was clear, the token generator (another µVAX at the end of the data bus) would issue a token with the
current event number.
Each of the eight data cables would get a token. On any one of
those cables, if a VME crate saw a token, it was able to capture the
token and send as many bursts of data as it needed, releasing the
token downstream when it was done to allow the next crate to
dump its data. When all eight tokens got back to the token scheduler, the next event could proceed.
When the data reached a µVAX on the farm, it would undergo
scrutiny by a variety of programs designed to filter interesting
events from the subatomic chaff. The goal at this stage was to winnow down the data rate another 200 to 400 times.
Events that met the second level of filtering would emerge from
the µVAX farm and go to a VAX 6000 to be spooled to tape. I saw a
rack with 11 Exabyte tape drives on it, and Dean informed me that
the experiment had well over 50.
This was not excessive when you consider that one event per
second for six months would be close to 4 terabytes of data. Eventually, a physicist would spend up to a half hour with each interesting event, looking for clues. Some events that looked particularly
promising were also spooled on the VAX 6000 for immediate analysis during the run.
The next morning, I went with my
father to see
SciTech, a hands-on
science museum for kids he had spent the last two years building.
SciTech had managed to obtain a cavernous old post office in the
middle of nearby Aurora.
The museum specialized in the very big and the very small,
teaching kids about light, sound, astronomy, and even particle physics. Every exhibit was hands-on. To explain quarks, for example,
SciTech was modifying slot machines, replacing lemons and cherries
with rare quarks and adjusting the probabilities to match. Pulling
the lever to create a particularly rare particle would result in a prize.
All around were prisms, tuning forks, echo chambers, computers, and dozens of other devices making up the exhibits. Signs were
next to each exhibit and teenage "explainers" would walk around
telling people what they were seeing. A group of girl scouts trundled down to the basement to help build a new exhibit. Upstairs,
some six-year olds were playing in a science club.
Starting a museum is never easy, and SciTech had managed to
surprise everybody with its rapid growth. Still run on a shoestring,
the museum was able to get enough grants, admissions, and members to keep expanding the exhibits and start filling up the huge old
post office.
I admired the exhibits, then ran to catch my plane. O'Hare was
not as bad as usual, so I had a little time to kill. Walking down the
moving walkway, I came to a stop behind a little girl and her
mother.
"Move over, honey," the mother said, "this is an airport and the
nice man has to hurry and catch his plane."
The little girl smiled sweetly and moved over, just in time for
me to get off the ramp and head into the nearby cocktail lounge.