Interview with internet pioneer Steve Crocker

I interviewed Steve Crocker, Chairman of ICANN and author of RFC1.

This interview was published on on 18 July 2013.

In a bathroom, at three in the morning in April 1969, a graduate student named Steve Crocker started to write one of the most important documents of the last century. Though drafted in humble circumstances Crocker’s document would set the open, inclusive tone of the next half century of Internet engineering culture, and initiate the process of defining the rules that govern virtually all data exchange on the planet.

He was de facto leader of a small band of graduate students working on the problem of how computers at their four universities should talk to each other on an experimental network called the ARPANET. He stood in a secluded bathroom as he drafted the students’ first written output, for fear of waking the friends he was staying with.

He had no idea of the enormity of what he was beginning. “Request For Comments”, now known as RFC 1, would become the first of thousands in a series that continues to define internet standards today.

Crocker and his fellow graduate students worked on the network because most of their superiors viewed it as “a sideshow, an intrusion”. The ARPANET had been foisted on several key research centres by the US Department of Defence’s Advanced Research Projects Agency (DARPA), a major funding body.

DARPA believed that networking the enormously expensive computers at major research laboratories across the country would enable new laboratories to share these machines instead of buying their own, and save the taxpayer millions of dollars in the process. Once DARPA’s contractors had built the physical hardware for the network the professors passed off responsibility for connecting their own computers to each other across it downward, to their graduate students. “In the US”, Crocker says, “one of the common sayings is that graduate students are the second lowest form of life on the university campus”.

However, he remembers, “nobody had told us we were in charge”. The graduates started to work without any formal authority, visiting each other’s laboratories from August 1968 onward. Ironically for a computer networking project intended to reduce the need for physical meetings they quickly realised that they’d need to set aside a large budget for travel. By March 1969 when they began to write down their ideas, Crocker, who had volunteered to organise the notes, started worrying about RFC 1.

Being at the bottom of the academic hierarchy and fearing opprobrium from the top made it difficult to draft authoritative documentation. Crocker had always found writing difficult (although he is a pleasure to read), but this particular document, he tells me 44 years later, made him “very nervous that we were going to get yelled at, that we were going to get criticized for being presumptuous”. Crocker started loosing sleep over the problem.

But what emerged as a result was an open and inclusive mode of collaboration that set the tone for later movements such as open source, Linux, and Wikinomics. To hedge against opprobrium from above he gave his text a humble title, “Request For Comments”, and made its tone as inclusive as possible. In RFC 3 he elaborated on the rules for future RFCs. In effect, there were none. An RFC could be contributed by anyone, and could be as short as a sentence:

“These standards (or lack of them) are stated explicitly for two reasons. First, there is a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition”.

A later elaboration of this principle by Dave Clarke of the Internet Engineering Taskforce was “We reject kings, presidents and voting. We believe in rough consensus and running code”.

The graduate students were able to take this freewheeling approach thanks to a “very, very light hand from Washington”. Both US government and military had learned the importance of investing in scientific research during World War 2. “That wisdom, in a certain sense, led to a very expansive role in terms of funding research with an adult or evolved understanding that you must not try to control exactly everything that was going on. That it’s important to cast your bread upon the waters and wait and see what happens”.

Now Crocker is concerned about the level of scientific funding in the United States. DARPA, which supported the early work on the Internet, “was given extremely broad authority to create technology and not ask permission but just go do it”. It had been founded in response to launch of Sputnik, which demonstrated Soviet mastery of rocketry and galvanised radical R&D in the US in the late 1950s. Crocker fears that the highest level of government are distracted from long term R&D. Whereas World War 2 and the Cold War focussed minds, “we do not yet have a crisis that is competing at the same level”.

As our conversation moves on to privacy Crocker says that there is no perfect solution. “Privacy is a two edged sword. If you walk into a hotel that you have frequented before, and the concierge greets you by name and had your favourite drink waiting, is that a violation of your privacy or is that high quality service?”

In social terms, Crocker says, our norms and expectations are likely to change over time. “Our world has moved from one of small towns where everybody knew pretty much everything there was to know about you, to the anonymity of big cities. Now we’re moving back”. The result of this reversion to small town intimacy is that, on the one hand “keeping secrets is going to be harder and harder”, but on the other hand we will “evolve our norms as to what is expected”. In other words, as things become impossible to hide the half-life of embarrassing disclosures may change. “If someone does something stupid as a teenager does that have to damage their career forever? Not necessarily. Would you hire somebody who didn’t behave properly but now they are ten years older? You might very well say, ‘yes I remember that, but I did that too’”.

A cameraman is filming our discussion with a supped up iPhone with a bolted on wide lens and broadcast quality microphone. Crocker has not seen such a system before, and his fascination with the technology is evident.

I pose a question that has troubled me for several years. He is one of the people who shaped and articulated the norms of a generation of engineers, values of openness, humility, and trying to get things done under adverse circumstances. Computer science has become a commercial rather than an academic vocation, and ask if he shares my fear that the architects of the coming decades will loose these values. His answer, Crocker says, will surprise me. “The Internet is not going to be The Story forever”. He sees other big changes afoot: experimental bioengineering, environmental deterioration, and the prospect that the Westphalian system is coming to an end. With such changes unfolding, the finest minds of the future may not be occupied by computer science. “The environment in which the Internet was built reflected a particular time”, Crocker says. “Everything has a time and that doesn’t last forever”.


TRANSCRIPT [contains errors] 

Johnny Ryan

There’s a wonderful story in the history of the Internet, which involves you sitting on a bathroom floor. Apparently the tiles were very cold on your ass. You were staying that night with friends who you did not want to wake up. You started working on a document, now known as RFC 1. And this is the document that defines the entire history and technology that is now shaping the world.

That’s not too much to claim. Maybe you were standing on the shoulders of some giants. Why were you on a bathroom floor?

Steve Crocker

Yes, it’s always interesting how history gets rewritten a little bit. I wasn’t sitting, I was standing. And so I didn’t have to worry about the cold floor. But I was in a bathroom. It was very late at night, around 3 in the morning, and there were people sleeping all over the house. I did not want to wake them up.

So this is tied to an amazing, interesting startup issue that is pretty different from anything today. When the network was started there were some very smart people who had a significant amount of experience behind them. The plan was a formal structure for the nuts and bolds of moving the bits around, the creation of what you would now call a “router”. The word had not been invented exactly. But the first routers were built under  a fairly good specification. A formal contract. Professional people. And they did a first class job.

What was not done in the same way was to figure out what to do with this network. Instead the network was built in a fairly rich environment. And by rich I don’t mean so much money, but I mean one where there was allot of  technological prowess. The agency that funded the network, the Defense Advanced Research Agency in the US Department of Defense, had a history of supporting really high class computer science, among other technologies.  So there were laboratories around the US that were doing work in artificial intelligence, in advanced operating systems, in advanced computer graphics, and in very interactive systems, in big databases (small by today’s standards), fancy architectures, multiple computer architectures. So in each of the different laboratories (we’re talking about MIT, Carnegie Mellon University, California Berkeley, UCLA (where I was), University of Utah, Harvard, and so forth) there were really important works going on under senior professors. And the network, ARPANET, which was called “the network” because it was the only network around, was built in that environment.

So the places that were connected were those laboratories. And in those laboratories were not only the professors who ran then, but you had a whole set of graduate students and related staff. So you had a pretty good set of people who were already involved in looking in to the future in one way or another, and who were pretty capable from a craft point of view. They knew how to programme, we knew what the structure of computers were, we what operating systems were. And so, in a sense, connecting up our computers to the network was just another task. Now, it had more interesting little details to it than it first seemed. But in some sense it was kind of an ordinary task. And my own view was, actually, that it was not nearly as challenging from a technical perspective as the really big problems that we were worried about. I was interested in artificial intelligence, I wanted to know how the mind worked. This business of connecting computers to the network seemed much less challenging, sort of mundane. I used to sneer that it only had socially redeeming value. It didn’t have any real depth. I acknowledge that I was wrong, that it was a more complicated business.

So in that environment, sort of a long answer to your question, there was a careful structure about the hardware that was necessary, and the long lines to connect the routers together. And then there was an openness  saying, well lets put it there and see what these guys do with it, and a very, very light hand from Washington suggesting what to do. The rest was self-organising.

Had we not self organized, had we not made progress I’m sure they would have stepped in and done something else. But those of us who were in the first four sites (the initial network was four sites, UCLA, Santa Barbara, SRI up in Menlo Park, and University Utah in Salt Lake City). The first thing that happened was that people from these four sites came together just to meet each other and very quickly we realized that we had much to talk about with each other. We agreed to keep talking with each other. And in fact the first decision we made was to go visit each other’s laboratories, which was exciting to use because we were young and we were excited to see what was going on. We realized instantly the irony that this network was supposed to reduce travel, and the first thing we did was to increase our travel. And eventually I wound up doing enough travel that the budget had to be increased and the contract modified with a serious impact.

Johnny Ryan

So on that, when you started this communication there were six of you in the first Network Working Group I believe? More? So you were one of three who had been to high school together, and a few of you guys were graduate students of Leonard Kleinrock. And as far as I remember, you were just one year after your undergraduate. So you were bottom of the academic pile. Firmly at the bottom. 

Steve Crocker

Yes. So first, this business of being bottom of the academic pile. These sites were all being run by professors that had their own research agenda. This network was a sideshow, an intrusion. It was OK, they were willing to have it. They couldn’t say no because the money was coming from the same place [ARPA], but it wasn’t their agenda so they passed it down.

Johnny Ryan

And it was forced on them by ARPA.

Steve Crocker

“Forced” is a strong word. In fact it’s interesting. Let me talk about the dynamics there. ARPA decided they were going to build it. The various professors around the community had different reactions. Somewhat peculiarly, in the East there was more negative reaction, and in the West there was more positive reaction. I don’t know if there is a strong reason for that, but typically the centres in the East were older, more established. They had richer facilities that they had built up over the years. At first their reaction was “why should we be sharing our facilities with these people”. Whereas the ones in the West were newer, and some of them, Kleinrock particularly at UCLA, had done allot of the theoretical work relating to networking, so he was quite excited by this. So that’s the story of how it got down to the graduate student level. I don’t know if it’s common elsewhere but in the US one of the common sayings is that graduate students are the second lowest form of life on the university campus.

So then we wound up talking to each other. Now, as it turned out, three of us had gone to the same high school. Vint Cerf and I had been friends in high scool, and there’s a whole little story about how we wound up together at UCLA. Jon Postel had gone to the same high school at the same time. We didn’t know him then. I met him as a graduate student. Exactly how far out of graduate school were we? It was more than a year. Vint and I were nominally class of ’65. I was a little slow getting out of school but Vint graduated in ’65, worked for a year and then went to UCLA. So we were a couple of years out of school.

Johnny Ryan

Let me, before we move to the present and the future, dig back to those early days of RFC1 again. The language you use in this document – for people watching I’ll explain that these documents are in the over 6,000s and specify every technology on the Internet – the language you use in those documents refer to “considerably less than authoritative ideas”. That’s from RFC 3. The idea was that one of these documents, and RFC, could be one line if you wished, or one word. And even the title “Request For Comments” was not saying “this is how it is, suck it up”. That imbued the entire project with a certain style, a certain openness. That “No Kings, Rough consensus and running code” Dave Clarke thing. And you can see that permeated through the technology. Here’s the question (that was a long non-question). There’s allot of discussion over the last 10-15 years, with Linux and Wikinomics, about opening up. When you want to run things, be as open as possible. People don’t realise, but they are actually harking back to what you did in that bathroom, what you started. But the way you worked was as part of a hierarchy actually, and you were working with people of like mind, many of whom knew each other and understood each other’s problems. So if you were running this project and it wasn’t within academia and was not funded by ARPA, do you think the RFC approach could have worked and is it a model for how things can happen in the future?

Steve Crocker

So, let me make it all more humble. It wasn’t with great insight and foresight about you know, this is the way it’s going to happen. RFC 3, which you’re referring to, which set the rules for the RFCs, was the solution to a personal problem. Which was the following: after several months of intermittently interacting with each other around these four laboratories, so we had met each other in August of ’68, and in around March of ’69 we said “you know, we have a bunch of interesting ideas here, we better start writing them down”. And that caused a bit of a problem. Because nobody had given us any authority. Each of us took an assignment. And in addition to each of the writing assignments we took I said I’ll organise them. Everybody said fine. They hated administrative work and I should have not done it myself but anyway. So I took that on, and I thought it was a trivial task. But I found myself pausing and putting it off, and really getting quite nervous about it, including loosing sleep. I had to introspect and understand what it was that was bothering me. And what was bothering me was that I was very nervous that we were going to get yelled at, that we eere going to get criticized for being presumptuous, for asserting ourselves. Nobody had told us we were in charge. I figured somebody important was going to show up. They were going to come from the East, I wasn’t sure it was going to be Boston or Washington, but I was sure somebody was going to show up and saw “who are you and what are you doing and I’m in charge”. So that was very much on my mind. And the other thing was that I have difficulty writing. So writing formal documents that meet the standards that one would want is always a big challenge. So as I was stumbling over this I hit upon two things. One was to say that these document did not have any authority. The other was to reduce the threshold for being about to write one. So that’s where the one sentence and can be incomplete, can be a question without an answer, a design without an implementation, etc. all came from. Figuring that since they don’t have any authority you could write it and replace it and replace it and replace it and that would be a process that would go on. And then the device of calling every one of them a “Request For Comments” was just one of those quirky moments that came as a matter of form, and I thought it was just temporary. I thought, here were are in the Spring of 1969. In the Fall we’ll have the network and we’d have formal documentation and these notes will disappear and we’ll have a users’ manual and an architecture manual and so forth.

The idea that they [RFCs] kept going and going and going. The six thousand figure that you walked about (is somewhat different from the numbers that we issued in the beginning because every note had its own RFC number. Then when the Internet started to work and we had E-mail led to the creation of Internet Drafts that didn’t get numbers in the same way. So every one of the RFCs that you see today corresponds to a hundred or so numbers that would have been issued had it been done the old way. So the number on the same scale is probably closer to a half a million or a million or somewhere up there. If you could have told me that that was what was going to play out I would have said “that’s just crazy. It’s not going to happen”.

Johnny Ryan

Let me ask you about something that brings us closer to the future. The history of the Internet is the story of at least two attempts where the US Government tried to gift the private sector a revolutionary technology. So back in the early 1960s [Paul] Baran working at RAND approaches AT&T and AT&T tells him to buzz off. Then in the mid ‘70s you would have been in the same position as well. There was an industry opposition to the new technology. Clearly there is a big irony there. But this conflict between the private sector and the military funded networks taps in to two things. First, people thought there was a market distortion potentially, from having massive military expenditure in this kind of R&D. And secondly, as soon as the industrial side picked up on the technology it inevitably changed the character of that technology. So you do you have thoughts on those two things? Do they jump out at you?

Steve Crocker

I can’t speak to the particular history of Baran’s interaction with anybody. He was working on a slightly different problem, and it was no a direct relationship even though there was some notions of packet switching in there. It is the case that AT&T was approached for taking over the ARPANET in the early 1970s and they walked away from it. And I suspect that were other various kinds of interactions over time. It is a very interesting phenomenon, the impact of the spending of the US Government, particularly through DARPA. So, the DARPA history is maybe special in its own right. DARPA was created in response to Sputnik. The US Government was caught off guard when the Russians put p the first satellite. The institutional reaction was “we’ve been caught flat footed, we’ve got to prevent such a thing from happening again”. And of the several things that happened in response, the creation of this agency, which was given extremely broad authority to create technology and no ask permission but just go do it. It suffered quite allot of bureaucratic in-fighting. The military wanted to kill it off. They didn’t like it. The Army, Navy, and Air Force were not happy at all about having this other agency that was not under their control. Now they love it. It totally transformed things.

So the fact that they were able to spend money, and to do it in a way that did not create enormous and immediate competition among the people doing the work – there was completion for research funding, but it wasn’t competition for the commercial dollar so much initially – it really made a big different. It was at a small school in rural Peru a couple of years ago on an impromptu visit, talking about the Internet to middle schoolers. And one of the best questions that I’ve ever heard was from one of the kids and he said “why did the military invest in the network? That seems like a strange thing for them to do”. I had to reach back and explain about the understanding from previous years, about the importance of building science and technology in World War 2, and radar and computers and so forth. That wisdom, in a certain sense, led to a very expansive role in terms of funding research with an adult or evolved understanding that you must not try to control exactly everything that was going on. That it’s important to cast your bread upon the waters and wait and see what happens. And there’s been allot of research showing that it’s hard to predict exactly. There’s a famous report tracking several technologies that were funded by the government and then funding sort of disappeared and what emerged later were multiple billion dollar industries. The work had paid off and led to commercial activity. So the Internet, though not in a planned way (transferring over to AT&T didn’t go anywhere) has now changed the world substantially.

Johnny Ryan

Dr Crocker, can I ask you about an issue that has been present since the beginning: privacy. Do you have fears about this, is it an issue that is going to change how people behave online or affect business online?

Steve Crocker

So privacy is a two edged sword. If you walk into a hotel that you have frequented before, and the concierge greets you by name and had your favourite drink waiting, is that a violation of your privacy or is that high quality service? And that tension is permeating entirely through the commercial side of things. So on the advertising side you have targeted advertising for you, as much as they know about you. Is that a violation of your privacy? Or is that a service? And even one could argue in terms of government keeping records, or other people keeping records. I think that there is no perfect solution. I don’t think that there are any absolute marks to sit on and say “this is the line and we don’t want to cross this”.

There is going to be a very complicated dance that plays out that involves understanding the technology and the shifts that it causes. We’ve moved from a world in which people lived in small towns, everybody knew each other and they knew pretty much everything there was to know about you – if you wanted to keep a secret you had to go to some lengths – to the anonymity of big cities. And now we’re moving back to where there is quite allot of information known about you. One thing is to understand the technology and its consequences. Another is to reset or to evolve our norms as to what is expected.

If you do something stupid, is that a private thing that nobody should know about? Or should you know full well that it’s going to be recorded, that it’s going to be on YouTube, that it’s going to follow you around forever? Will that do allot of damage? It already does. Maybe parents can educate their children a little better. Children will do stupid things, as they always do. In evaluating things, so if someone does something stupid as a teenager, does that have to damage their career forever? Not necessarily. Would you hire somebody who didn’t behave properly but now they are ten years older? You might very well say, “yes I remember that, but I did that too”.

I think this is going require evolution along multiple lines. Evolution in understanding technology, and evolution in deciding what’s important and what’s not so important. And I don’t know exactly. But keeping secrets is obviously going to be harder and harder.

Johnny Ryan

Let me ask you as a final question something that is a major fear, a major threat. You’re one of the people who shaped the norms of a generation of engineers, or else who reflected those norms. And they were about openness, humility, trying to get things done under adverse circumstances. Now if you are a young engineer  you are probably going to be given a  very large salary of you move into web tech, and you might be working for a very large corporation. You probably wont go into foundational science, not necessarily. And that may have an impact on the norms of engineers. Those norms may change. If they do change that means the architects of the coming decades may have a completely different mindset to what you and your colleagues had. Are you worried about this? do you think there needs to be a Hippocratic oath or something like that? I know that sounds nonsensical but do you have any thoughts on that?

Steve Crocker

I don’t think my answer is gong to go in the direction that you are expecting. Everything has a time and it doesn’t last forever. The environment in which the Internet was built reflected a particular time. The itself, although it’s an enormously important technology, it’s obviously affected my life along with everybody else’s, is not going to be the story for ever and ever and ever. There are many other big things happening in the world. You look around and we have big issues with respect to the environment, and they will become more and more important over time. There are huge changes in biology and studies that will have an impact on everything from bio-engineering of food to medicine. Those will be enormous big changes. The communications of which the Internet is a part, and other factors, are also causing changes in government structures. We’ve moved from a world in which exploration of uninhabited or ungoverned technologies was a very big deal, colonization and so forth, to a world in which most of the world is governed and you have the Westphalian model of nation states. The question is is it’s time coming to an end, or is it just maturing. So there are allot of big forces at work.

Trying to place the ethic that you’re describing in the context of commercialization versus openness of communications and publication I think is actually caught in a much bigger struggle and is not the most important question. A lot of my friends are very concerned about the level of research funding coming out of the US Government. Take that to the top level of the US Government and try to get a conversation and the answer is that they’re not involved in that at all because they have grid lock in the budget, and very big issues, terrorism, non-trivial issues that are occupying the very top levels of government. And we do not yet have a crisis that is competing at the same level at those things in terms of research funding and as a consequence you get a balance that is shifting between the openness of the universities and commercial activity.

One thought on “Interview with internet pioneer Steve Crocker

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s