Then, do you still feel justified in your own work -- on networks and Jini, which could be considered an aid to the super intelligent machines that you fear?

I'm actually not working on Jini or the network aspect of it right now. I'm trying to make machines more reliable. I'd like to have systems that worked, that were administered in a way that when you touched it, it didn't break. Every time I touch anything I break it, because change has a higher probability of causing an error.

I don't think [my work on reliability] contributes to the problem because the problem I'm concerned about is offensive uses of these things, and that doesn't require reliability. It requires overwhelming force. Defense requires more care. For example, a burglar can go down the street and look for the house where the burglar alarm is broken; I have to have one that works if I expect my house to be safe. It's the defense that needs the reliability.

So improvement in software reliability improves our ability to deal with these situations to the extent that we can abate them with technology. But in the end, I don't think there are technical fixes to a lot of these problems anyway. We're going to need to have some ethical and political and other fixes.

I'm interested in the kinds of systems that provide services to people who can't tolerate failure -- like air traffic control, hospitals and public safety. We need more and more systems that work safely and reliably in those environments, especially as medicine gets more computerized.

I don't see that it's contributing to this other problem. If I felt it was in any substantive way, then I'd really have a problem. But the rate of progress of the speed of personal computers is largely out of my hands. I sent copies of the article to Andy Grove and Gordon Moore [of Moore's law], but that's all I can do.

Wired News ran an article saying Silicon Valley didn't pay much attention to your warnings -- but personally, how would you gauge the reaction? Did anyone from Sun respond?

I sent copies to all the vice presidents and directors and other people who have equivalent categories in Sun -- that's a few hundred people -- and I would say I got 25 responses all of which were supportive. I suppose you could say that maybe some people didn't like it and didn't say anything because they were afraid. That's not really the culture though; people tend to question everything. I have one old friend who works at Intel who sent me mail saying he thought it was all way, way farther out. Then I sent him back a couple of references trying to move him gently to where I think the truth is. Now I just got mail from him a half-hour ago saying his daughter was thinking about going to school in nanotechnology, and now he's talking to her about it.

What about Hans Moravec's argument that we, as humans, are just parts of this evolutionary whole and that the advent of techno-humans represents a higher form of evolution -- one that we should be honored to bring about, even if they kill us off?

Nah. There's two things going on there. One: Science has always been fatalistic, saying we'll just pursue truth and whatever happens happens. Moravec is espousing, really, the Darwinian world. But humans haven't been subject to Darwinism for a long time. In fact you can make a strong argument that evolution stopped when we came along because evolution is no longer genetic, it's cultural. So he's arguing for bringing back something that has not been the driving force and somehow transferring it from a side effect of capitalism, creating it without anybody asking for it, and then just sort of doing surgery on ourselves. It might happen because we're not being careful, but it's an awful crazy thing.

Aristotle said, "We're clearly rational beings," and we clearly are. We didn't set up our present system without thinking. The Constitution, the whole economic system we have was designed. So I'm hopeful if only because I believe we can choose what's in our interest independent of whether it maximizes someone's notion of profit or destiny. We can choose to have a social safety net, when in fact business would be more efficient if it didn't have to pay those costs, and people had to starve in the streets. We can decide that's not acceptable and we have in most Western societies. That's not a totally Darwinian choice. The total Darwinian choice would just be to let it go. But we've decided that's not the kind of society we want to live in.

Do you think we'll actually reach the point where these technologies are considered dangerous enough to be regulated? Will Congress ever address the issue?

I think we have to be realistic that it takes a long time ... We're a very long way from any prospect of any action. And we would have to deal with the scientific community's enormous desire for lack of interference, and businesses' enormous desire for a lack of interference, and governments' desire to not do anything. Everybody's pretty happy with the status quo at this point. It's going to take some real leadership, and it's going to take time to develop.

Let's just hope no one has the breakthrough to wild self-replication. Luck may be an important aspect here. We might get lucky. Let's hope we do.

Recent Stories