Chapter 3 of my book A History of the Internet and the Digital Future has just been published by Ars Technica. This is one of the 3 chapters (of the 13 in the book) that are being published for free. Here it is, or read at Ars.
Chapter 3: The Essence of the Internet
The Internet is a loose arrangement of connected but autonomous networks of devices. Each device, a “host” in networking jargon, uses a “protocol” to communicate with other devices on the network. These protocols tie together diverse networks and govern communication between all computers on the Internet. Not only are the protocols elemental to the Internet and how it works, but the unique collaboration between their designers was the formative event of Internet culture. In as much as any single element of the whole can be, these protocols are the essence of the Internet. The remarkable manner in which a team of young collaborators developed these protocols set the tone for the future development of Internet culture. As their work on the protocols proceeded they began to establish the informal conventions that would characterize the tone of collaboration and discussion on the Internet thereafter. The process began in a bathroom, late on the night of April 7, 1969.
As BBN started building the IMPs for the ARPANET in 1969, an important piece of the network was missing: the software that would govern how computers would communicate. Graduate students at various facilities funded by the US Department of Defense Advance Research Projects Agency (ARPA) had been given the task in 1969 of developing the missing communication protocols. They formed an informal “network working group.” Finding themselves working in a vacuum, the students connected to ARPANET, who had been given the task in 1969 of developing the technical protocols, also began to establish the informal protocols that would influence interpersonal communications on the Internet in general.
Uncertain of their positions within the hierarchy of the ARPANET project, the students issued notes on their protocols under the title “Request for Comments” (RFC). Steve Crocker, a graduate student who had received his bachelor’s degree at UCLA only a year before, used the title Request for Comments to make the invitation to participate as open as possible, and to minimize any claim to authority that working on so crucial an aspect of the network as its protocols might imply. The first RFC document, which set the tone for the next half century of Internet culture and initiated the process to define the protocols that govern virtually all data exchange on the planet, was composed in humble circumstances. Its author recalls: “I had to work in a bathroom so as not to disturb the friends I was staying with, who were all asleep.” The tone in which the RFCs were typed was distinctive.
Crocker was the de facto leader of the small group of six. He and two others of the group had been at the same high school in Los Angeles, Van Nuys High, and were graduate students of Leonard Kleinrock. (Kleinrock was under contract with ARPA to run the network measurement center at UCLA.) Crocker was writing a document that outlined some broad ideas on how the students would pass around ideas through “temporary, informal memos.” Even as he drafted the document, the prospect of disapproval from far above in the academic hierarchy weighed heavily upon him:
In my mind, I was inciting the wrath of some prestigious professor at some phantom East Coast establishment. I was actually losing sleep over the whole thing.
Crocker was eager to open up the process to as many of his peers as possible:
Closely related to keeping the technical design open was keeping the social process around the design open as well. Anyone was welcome to join the party.
Vint Cerf, an early participant in the informal networking group (and now vice president of Google), sums up the approach and context:
Keep in mind that the original developers of the host level protocols were mostly graduate students. We adopted a humble and inclusive posture and a mantra that Dave Clark ultimately coined as “rough consensus and running code”—that means we don’t really vote exactly, we just try to assess rough consensus among the group trying to agree on proposed standards.
RFC 3, released in April 1969, elaborated on the character and objectives of the RFCs (note that the word “Host” here refers to a connected computer):
These standards (or lack of them) are stated explicitly for two reasons. First, there is a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition.
RFC 3 continues in the counter-hierarchical vein, establishing the principle that no text should be considered authoritative and that there is no final edit. This is a pivotal element of the “perpetual beta” described in the next chapter. Also implicit was that authority was to be derived from merit rather than fixed hierarchy. A later elaboration of this principle was:
We reject kings, presidents and voting.
We believe in rough consensus and running code.
Crocker’s RFC, though penned in humble circumstances, set the open, inviting tone of the next half century of Internet culture and initiated the process to define the protocols that govern virtually all data exchange on the planet. Since Crocker’s RFC there have been almost six thousand RFCs published, which maintain an open, collaborative approach in Internet-engineering circles. The meritocracy of the RFCs was exemplified by a generation of delinquent programmers at MIT from the late 1950s to the late 1960s, who in turn created the “hacker” culture that influenced much of what was to follow. The first fruit of the graduate students’ labour was the NCP, the Network Control Protocols, which governed communications between machines on the Internet. The NCP, however, was merely the first protocol that allowed communications on the ARPANET. An “internetworking” protocol that could tie different machines and networks together was yet to come.
Radio and satellite networks
San Francisco features disproportionately in the history of the digital age. Little attention, however, has been given to one of its acknowledged landmarks: a public house called Zott’s. Zott’s (named “The Alpine Inn” since the mid 1950s) is a small, wood-panelled tavern and a historic focal point for the ne’er-do-wells of Silicon Valley. Its founder was Felix Buelna, a Mexican, who moved from Santa Clara in the wake of the gold rush when that area became crowded by would-have-been gold diggers in the mid 1800s. He built the inn on the site of a pony trail that had been used by rancheros and settlers to reach the coast. Buelna’s inn was a place of gambling with a colorful clientele and, in the words of the US National Park Service’s official survey, “a long string of colorful owners.”
Regulars in the 1880s included the construction workers building Stanford University, whose entrepreneurs and technologies would propel the dot-com boom a century later. The inn also became the regular haunt of the new university’s students. In 1908 the editors of the Stanford Sequoia lambasted their immoderate peers, writing that the student body had been “held up to the world as a community composed largely of drunkards.” In January the following year the president of the university wrote in vexed mood to the county supervisors requesting that they not renew the inn’s liquor licence because it was “unusually vile, even for a roadhouse, a great injury to the University and a disgrace to San Mateo County.” Yet the humble wood-panelled structure remained a landmark through the twentieth century as the digital industry evolved around it. By early 2001 its car park accommodated the expensive sports cars of the young Silicon Valley millionaires. It was fitting, then, that more than a century after its establishment Zott’s should be the site for an important event in the history of the Internet.
On 27 August 1976 a van parked in Zott’s beer garden. It was part of the Stanford Research Institute’s (SRI) packet radio experiment, conducted under contract for ARPA. The SRI team removed a computer terminal from the van and placed it on a wooden table in Zott’s beer garden. A wire connected the terminal to the van, and radio equipment in the van connected it to ARPA’s new packet radio network, PRNET, which in turn was connected to ARPANET. The team at Zott’s sent a message from their terminal across the PRNET and thence to a distant machine connected to ARPANET. This was one of the more momentous events to have happened in any beer garden: it was the first ever packet data transmission across two networks using the new “internet” protocol.
The discoveries that made this transmission possible arose as part of an earlier project at the University of Hawaii in 1970. Norman Abramson, the Professor of Electrical Engineering and Computer Science, had faced a difficult problem. He wanted to network the University of Hawaii’s seven campuses. This posed three problems. First, the campuses were physically spread across four islands. Second, the leased telephone lines that connected ARPANET facilities to each other were too expensive for his budget. Third, the line quality of the Hawaiian telephone system was too poor to carry networking data. The answer, Abramson decided, was to use radio. Thus from 1970 ARPA began to fund Abramson’s attempt to develop a packet radio network.
Radio signals travel differently to electric signals across telephone lines. While telephone signals travel from point to point in an orderly sequence, radio transmits indiscriminately to all receivers within its broadcast range. Signals broadcast by different nodes to the receiver at the same time can collide and be destroyed. Abramson’s team developed an elegant solution to this problem: when any node sent a packet but did not receive confirmation of successful delivery from the receiving node it would wait for a random period and then resend the message. Since all nodes would wait a random period before resending, the odds of repeat collisions were slight. Thus the network would quickly correct itself when it lost packets. Using this method Abramson’s team built a functioning network called the AlohaNet that linked Hawaii University’s campuses to each other and to the ARPANET. This method of dealing with collision between messages was called the “Aloha method,” and ARPA used its example to build its own packet radio network, PRNET.
The discovery of the Aloha method for packet radio networking was particularly timely since the political tides in which ARPA swam had become slightly more turbulent. In 1969 Senate Majority Leader Mike Mansfield had signalled his intention to cut $400 million from the defence research budget. He was the author of Section 203 of the Military Procurement Authorization Act for Fiscal Year 1970, the so-called “Mansfield Amendment,” which stipulated that all funded research must have a “direct and apparent relationship to a specific military function or operation.” Packet radio was just such a project. The power of massive, expensive mainframe computers could be relayed to the battlefield by networks of radio, cable and, as ARPA was beginning to prove, satellite.
***
Thus by the mid 1970s ARPA had built three functioning networks: ARPANET, PRNET and SATNET, using cable, radio and satellite. Now it remained to network the different networks.
By 1973 a new protocol was required to internetwork the ARPANET, PRNET and SATNET. After organizing the 1972 International Conference on Computer Communication at which the ARPANET was demonstrated in Washington, Robert Kahn had joined ARPA (now named DARPA) where, following some unrelated projects, he resumed his work on packet networking. In the spring of 1973 he approached Vint Cerf, one of the group of graduate students that had developed the NCP protocol for the ARPANET, and outlined the need for a new internetworking protocol that would allow computers to communicate together across cable, radio and satellite networks. Cerf had just become an Assistant Professor at Stanford and ran a series of seminars to tease out the problem. He drew together a group of researchers who would later hold key positions in the networking and computer industries. Participants included Robert Metcalfe, who was representing the Xerox PARC research center, and Gerard Lelann, who was visiting Cerf’s lab from Cyclades, the French packet network project. Cerf’s group continued the inclusive, open approach of drafting RFCs.
They were influenced by the example of Cyclades, which had adopted a centrifugal approach in which data transmission was not regulated by the equipment of the network itself but by the computers sending and receiving the data at its edges. At the Xerox company’s PARC research center, Robert Metcalfe was working on something similar. Xerox had just unveiled a revolutionary new computer called the Alto. The Alto had a mouse, graphical display, a desktop and system of windows and folders for storing files. The machine was two decades ahead of its time and represented a paradigm shift in computing that the senior management of Xerox failed spectacularly to capitalize upon. Metcalfe was working on a network to connect many Altos in an office, developing “Ethernet” and Local Area Networking (LAN). He grew impatient with the consensus approach that Cerf, Lelann and others were taking and decided to move ahead on his own. In 1973 Metcalfe’s PhD thesis had refined the Hawaiian Aloha method. He now applied these mathematical improvements to develop a system informally named Alto Aloha, which became PUP (PARC Universal Packet). PUP adopted the same centrifugal datagram approach of Cyclades, and its network was dumb to the extent that it was merely a system of cables. Unlike the ARPANET, where the IMP machines controlled many of the network’s functions, the PUP network had no capability to control transmission or flow of data, or to verify delivery or repeat transmission of lost or partially delivered packets. Instead the software protocols running on the connected host computers would control the network. As a later PARC memo on the specifications of the PUP noted:
Pup communication is end-to-end at the packet level. The inter-network is required only to be able to transport independently addressed Pups from source to destination. Use of higher levels of protocol is entirely the responsibility of the communicating end processes.
This moved control over the operation of the network from the connecting infrastructure to the actual devices participating in the network themselves. This was a centrifugal approach, and it suited the requirements of the network of networks that ARPA had in mind.
The NCP (Network Control Protocol) that controlled communications on the original landline ARPANET was not appropriate for radio and satellite networking. Instead, a new internetworking protocol would give each connected “host” computer a far greater degree of responsibility for control of the network. The new protocol, which would run on each host computer connected to the network, would not only establish connections between hosts, it would assume the functions that the dedicated Interface Message Processor (IMP) computers had performed: verifying safe delivery of packets, retransmitting them where necessary and controlling the rate of data flow. Simply put, to allow data to flow across a network that included landline, satellite and radio connections, a new protocol would take a much more flexible approach to communication control. In May 1974, Cerf and Kahn published an outline of the new Transmission Control Protocol (TCP):
a simple but very powerful and flexible protocol which provides for variation in individual network packet sizes, transmission failures, sequencing, [and] flow control.
This internetworking protocol is, in a technical sense, the essence of the Internet, and in its priorities and functions can be discerned the cardinal characteristics of the new medium.
TCP is centrifugal by necessity, as one of its designers notes:
We wanted as little as possible at the center. Among other reasons, we knew quite well that it’s much easier to scale a system that doesn’t have choke points in the middle.
Some of the enthusiasm for the centrifugal approach of relying on the host computers themselves and abandoning the IMPs may have arisen from social rather than technical reasons. The IMP machines connecting host computers at each participating facility to the ARPANET were controlled by BBN, ARPA’s main contractor, which gave BBN control over the network itself. From their central offices BBN engineers could remotely update, repair and monitor the use of IMPs across the network. Increasingly, researchers preferred the idea of a dumb network controlled by a community of computers using a common protocol without the IMP standing between the host and the network. It is a small irony that the IMPs were originally introduced in 1969 not to control the network but to convince the ARPA-funded researchers that connecting to the ARPANET would not directly impose a burden on the processing power of their host computers. Support for the removal of the IMPs only five years later was a sign of how far networking had come.
Much as Paul Baran’s original, decentralized network prioritized survivability over other considerations, the TCP prioritized robustness over accountability and control. Billing and accounting, which would have been foremost in the minds of commercial designers, were entirely absent from the ARPA internetworking protocol. TCP was also heterogeneous by nature. It was designed so that machines on different networks using different technologies could seamlessly communicate as though they were on the same network. Various networks were bridged by so-called “gateway” machines that maintained routing tables with the addresses of computers on their own local networks. TCP underwent several revisions, and following a meeting in January 1978 between Cerf and two researchers, Jon Postel and Danny Cohen, at the University of South California, it was split into two parts to streamline the functions of the gateway computers. TCP would handle communication between computers and an additional Internet Protocol (IP) handled internetwork connections between networks. The combination of TCP and IP would avoid the gateway computers from duplicating functions already performed by host computers within local networks. What remained was to make sure that internetworking actually worked in real world conditions.
Cerf, who had joined ARPA in 1976, oversaw a series of practical internetworking tests. A particularly ambitious test was conducted on 22 November 1977. As the SRI packet radio van drove along the California freeway, the radio equipment onboard broadcast data packets via PRNET to a gateway machine that connected to ARPANET. Travelling across the ARPANET by cable, the packets sent from the van reached a gateway machine on the East Coast of the United States that connected to SATNET. From this point the packets were relayed by orbiting satellite to Goonhilly Downs in the UK, and thereafter back via ARPANET to California. To monitor the fidelity of the network’s transmission, a screen in the van generated patterns from the data it was receiving. Errors in the data transmission would be immediately clear from flaws in the pattern. Yet the system performed so well that whenever the signal was blocked by bridges and other objects that the van’s radio could not penetrate the pattern would only pause and then resume when the signal returned. There were no errors. This test had spanned three networks and two continents. Cerf recalls, “the packets were travelling 94,000 miles round trip . . . We didn’t lose a bit!” Another test put network computers aboard aircraft from the Strategic Air Command to simulate wartime conditions:
. . . airborne packet radio in the field communicating with each other and to the ground using airborne systems to sew together fragments of the Internet that had been fragmented by nuclear attack.
Here was proof that internetworking was possible between radio, satellite and landline networks across the globe and under adverse conditions. An optimistic observer might have thought that now, after proof had been offered, the telephone companies would embrace TCP/IP and bring the Internet to the masses. Instead, however, the telephone industry rushed to develop its own standard. It was keen to maintain its central control over the network.
How the TCP/IP Internet suite of protocols differed from and eventually overcame the alternative put forward by the telephone companies says much about the character of the Internet and the nature of enterprises that are best suited to prosper upon it.
Centrifugal defeats Centripetal (TCP v X.25)
ARPA’s networking endeavors, like those at RAND previously, were totally at odds with the ethos of the telephone industry. The reason for this conflict extended back to the industry’s very origins. In 1877 Alexander Graham Bell was granted a second patent for his telephone and formed the Bell Telephone Company. For a brief period, only Bell Telephone and its licensees could legally operate telephone systems in the United States. Then, when his patent expired at the end of 1893, a glut of competitors rushed into the market. Between 1894 and 1904, over six thousand independent telephone companies went into business in the country. This had two effects. First, it dramatically accelerated the proliferation of telephone systems across the United States. Second, it resulted in a chaotic mess of incompatible telephone networks. Telephone subscribers might indeed find themselves newly connected, but they were also unable to use their telephones to communicate with people who did not happen to be on the same system they subscribed to. In the 1908 annual report of AT&T (as Bell was known from 1899 when it came under the ownership of its subsidiary, AT&T), the president of the company, Theodore Vail, warned that only through universal and consistent service could reliable telephony be assured. The solution, AT&T argued, was monopoly. As its 1908 marketing slogan put it: “one policy, one system, universal service.” Vail’s argument prevailed and in 1913 AT&T became a government-sanctioned monopoly under the regulation of the Federal Communications Commission (FCC).
From this point on, AT&T’s control over the United States’ telecommunications network was so absolute that, until the late 1960s, homeowners in the United States were even forbidden from modifying their telephone sets in any way. While Bell Labs, a research unit within AT&T, was a hive of innovation, the company had no strategic interest in transforming its network. AT&T was firmly wedded to the orthodoxies of central control and knew with the benefit of long experience that the circuit switching of telephone calls, the technology that Alexander Graham Bell had patented in the late 1870s, worked as it was. Thus Paul Baran had been rebuffed in the mid 1960s when he approached AT&T with the opportunity to use the packet-switched networking concept he had developed at RAND. Even so, in 1968 ARPA had anticipated that the successful demonstration of four functioning nodes in a data network would convince a telephone company of the merits of digital networking. The original planning document for the ARPANET project included the optimistic objective of transferring the technology so that a utility could provide “digital message transmission as a tariffed service.” Yet even in 1975, after proof of concept and extensive testing of the packet-switched distributed network concept had been successfully conducted at the taxpayers’ expense in the form of the ARPANET, AT&T refused to pursue the Internet idea.
Shortly thereafter, in the mid to late 1970s, telecommunications companies in Europe, Canada, Japan and the US did begin to appreciate that the potential of digital packet-switched networks should no longer be overlooked. While some public data networks had been established in the late 1960s and early 1970s, such as the Broadband Exchange Service in Canada from 1969, many telephone companies found the digital network revolution challenging. This was not least because they had been beaten to this conclusion by the computer manufacturers. While the ARPANET used open, non-proprietary standards to which any manufacturer’s devices could connect, the commercial computer manufacturers began to offer proprietary equipment and networks that were incompatible with machines supplied by their competitors. Thus telephone companies would not be able to choose suppliers on the basis of competitive prices if they built a network using proprietary equipment from IBM, Digital Equipment Corporation (DEC) or some other supplier’s technology. Thus, even as they became interested in data networks, the telephone carriers grew concerned that they might become in thrall to the equipment manufacturers.
The solution, the telecommunications giants realized, was to create an open standards network, the equipment for which could be produced by a plurality of suppliers, but over which they could retain the strict central control that they had always exercised over the traditional telephone and telegraph networks. Though they were keen to avoid monopolies in networking equipment, many of the telecommunications carriers were monopolies themselves. Outside the United States, telecommunications services were often offered either by regulated monopolies along the AT&T model or by the so-called “PPTs” (governmental ministries of posts, telegraph and telephone) or by nationalized companies. Within the US, as ARPA and the networking researchers it funded were veering off in the direction of multiple networks, decentralized control and open standards, AT&T remained fixed to the Vailian ideas that had helped to make it one of the largest corporations in the world. In the late 1970s, J.C.R. Licklider had cautioned that different networks would be unable to communicate with one another unless common standardized protocols were introduced. This is indeed what began to happen.
Between the mid 1970s and the late 1980s, a pivotal period in the expansion of networking to the public, ARPA and its allies fought a standards war against the telecommunications industry. Representing the interests of the telephone industry was the Consultative Committee on International Telegraphy and Telephony (CCITT), a body of the International Telecommunications Union (ITU). Within the CCITT a group began working on the question of standards in 1975. The resulting protocol, X.25, was adopted by the CCITT in September the following year. Despite their dawning awareness of data networks as an emerging market, the telephone companies had not fully made the leap to the concept of distributed networking that lay at the heart of Baran’s work at RAND in the early 1960s. Nor had the telephone companies embraced ARPA’s approach towards diversity. Foremost among Baran’s considerations had been the need to provide a robust communications infrastructure using unreliable, redundant equipment. The telephone companies took the view that digital networks, like the analogue networks for telephone and telegraph before them, could only offer reliable service if every aspect of the network were controlled by the operating company. The imperative for ARPA had been to enable “a very broad class of interactions” between a diverse array of incompatible computers and networks that served different purposes. Thus where TCP/IP enabled diversity, X. 25 required consistency and conformity.
In contrast, TCP/IP, like the French network Cyclades, used “datagram” packets. Datagrams are simple, elemental packets that can be combined as needed by host computers to scale up or down the level of complexity in their communications. X.25, however, was built to provide consistently high-quality communication in all cases—whether necessary or not—which made it inappropriate for many uses. This distinction between TCP/IP, which used datagrams, and X.25, which rejected them in favor of consistent and rigidly controlled communications, was an ideological division between two generations of engineers, centripetal on one hand and centrifugal on the other, as much as it was technical. Vint Cerf, one of the chief architects of TCP, recalls that he had offered TCP/IP to the CCITT but had been rebuffed because the protocol had come from the Department of Defense, and because “they thought they could not ‘sell’ datagrams.”
TCP/IP did not only accommodate different devices, but accommodated many different types of networks too. TCP/IP could be used inside a network, or to connect many networks, or both. Not so for the telephone companies. Their interest was in national public data networks. The diverse array of networks of varying sizes, architectures and purposes that TCP/IP could support, and which the Internet eventually became, was beyond their view. According to Cerf, TCP/IP “eventually won out because it was so general.” Ultimately, homogenous X.25, the expression of the centripetal bent of the telephone industry, was defeated by TCP/IP, the open, diverse and untidy offering of the research community. As TCP/IP spread across the globe, communications began to take on the centrifugal character that had previously been the preserve of participants on the ARPANET. Moreover, new opportunities beckoned for underground, amorphous communities drawn together by common interest rather than proximity. The growth spurt that the early Internet enjoyed in the late 1980s was partly due to the ability of TCP/IP to work at both the macro and micro levels. This proved to be enormously fortuitous for the development of the Internet.
Dr Johnny Ryan FRHistS 