Lady Christa, Triumvir 2, was pleased with her inspection of the Centre for Sentient Robotics in Tower 6. She had spent a couple of hours at the Department of Robotic Surgery, where she watched a number of operations on human patients. The good old "Master-Slave System" was working perfectly, with the Slave robot using four arms at once and the Master computer extracting as it went along the required knowledge from the vast archive of Surgical Wikipedia. Christa smiled as she recalled stories of surgery in previous centuries, where only a single doctor’s hands were at work and it was impossible to operate on tissues smaller than a millimetre. Yes, huge progress had been made. Now surgical operations were indeed under scientific control.

Christa, as a former psychology scholar, was also delighted with her visit to the Institute of Robotic Psychology, which had just been founded jointly by Towers 6 and 2. It was an offspring of Tower 2’s Department of Cognition, which she still loved to haunt in her rare moments of calm. She had been the first to realize that the advent of sentient robots would raise a whole series of new issues within the broad field of psychology. For example: are cognitive robots capable of empathy with each other? Or, more generally, are they capable of establishing social relations among themselves? And could they enter into emotional interactions with humans? Many years ago, with these questions in mind, she had created the Department of Cognition, which occupied an entire floor of Tower 2 and was endowed with three full professorships. It was something she felt rightly proud of.

But one detail of her visit to Tower 6 had left her perplexed. Nothing serious, apparently; most people would barely have given it a thought. But not Lady Christa, renowned since her student days for her almost legendary psychological insight.

It had to do with professor Wladimir Socholov, code name CDCSRT6, for Chief Director of the Centre for Sentient Robotics. He was also the major figure behind Creation Factory, the leading robot industry. Christa and he were personal friends, being the same age and having followed a similar university curriculum. They would wave and smile at each other whenever they met, although this no longer happened often. But this time... well... Wladimir (in private she used to call him Wlado) wouldn't even look at her in the eyes. He had looked at the floor, at the wall, at the others, but never at her. Nothing like this had ever happened before. So a warning bell had rung in Christa's mind. She was about to step out of Tower 6 but suddenly stopped and decided to call Wladimir on the spot.

-I need to talk to you, Wlado! - she said. It sounded like an order.

They were sitting in the small glass-walled office known as the Crystal Room, near the main entrance to Tower 6.

-All right, Wlado, what's wrong?
This time Socholov met her piercing gaze, and it was clear to him that there was no point in lying.
-Something isn’t working properly, Christa.
-Something serious?
-I’m afraid so. -Something to do with your new line of sentient robots?
-Yes, it has to do with them. But, you know, sentience is a generic term, Christa, and it comes in degrees.
-Semi-sentient then?
-You could call them that. But they’ve got far more intelligence than any others produced so far.
-Intelligence measured how?
-Usually by how long it takes them to open a safe with a 9-digit secret code. Another test we use is to see if a robot can assemble a thousand-piece jigsaw puzzle in thirty seconds. Yet another is to see if a robot can find its way out of a very complicated labyrinth. Any one of these robots, randomly chosen, can do these tests a hundred times faster than any other robot model developed so far.
-Help me get a general idea first. What actually is the situation with sentient robots? I know, of course, that there are hundreds of so-called sentient robots around now, many of them helping to care for senior citizens at home. In fact, that was why I founded the Institute of Robotic Psychology. But there’s a lot of fake news circulating about the concept of sentient robots and even up in the Triumvirate haven’t managed to get a clear picture of the situation. There was also that big scandal—you probably heard about it—the GST2 himself got caught red-handed by that detective—what was his name—Michael Mulligan?
-Yes, I remember, the usual “conflict of interest”! My colleague Dr Elizur Levi gave the detective some crucial assistance.
-Anyway, Wlado, do really sentient robots exist? Are they a reality, or just a fantasy of our AI fanatics?
-Well, Christa, first we need to exclude those robots programmed to answer as if they were sentient. Do you know what I mean?
-Yes. You ask a robot: "Can you smell mint?" and it answers “yes”, because questions and answers like this are part of its software.
-Right. But we use a different approach. First, we developed a special multi-layered neural network, to give the robot an intelligence several times higher than a conventional one...
-Does that mean that your new robots acquire awareness of their own intelligence?
-Not exactly. With the systemic type of special neural networks that we’ve given them, I believe they acquire a “feeling-of-being”. They know they exist.
-That’s a kind of consciousness.
-It’s the first gate of consciousness.
-Yes, I remember your day at the Quibble Convention, where you defended the idea that the feeling-of-being is the gateway of consciousness. Does that mean that each of those robots has a sense of individuality, that each one feels different from the other?
-We don’t have a clear answer to that question yet. But it’s quite likely in my opinion.
-Ok, Wlado. So, let’s come to the point!

Socholov was a small, completely bald man, with thick-lensed glasses. When he was nervous, he would take his glasses off and scratch his shiny pate with them. That’s what he did now.

-As you know, in Creation Factory we produce about ten thousand robots a month, i.e. over a hundred-and-twenty thousand a year, for distribution throughout the country. There are about forty different kinds of robots, for all possible uses. Most of them are for elderly people. As you well know, many people today reach the age of a hundred-and-twenty, even a hundred-and-fifty. Their minds are usually okay, but their bodies no longer respond so well; they need help, most of them. Our robots are already great at providing this.
-Yes, I know all that, Wlado. Lady Christa cut in rather impatiently.
-Right. Well, in the Creation Factory we decided to go a step further and produce sentient robots capable of showing a kind of empathy for the people they are helping. Equipping them with consciousness as you call it—though I’d never use this term—is particularly difficult. We needed a very special computer, one able to assist the robots in assembling themselves. So, we hand-built the first two robots, then fed the computer itself with all the necessary instructions, together with all the components required. The idea was that the computer would then instruct the first two how to assemble the next two robots, and these four would produce a total of eight, then sixteen, then thirty-two, sixty-four, and so on. The first phase of production was designed to stop at 512. All this under the direction of the AI super-intelligent computer.
-The computer itself was to direct a kind self-replication of robots?
-Well, let’s say self-assembly, rather, from materials supplied from outside—except for the neural network, which was to be produced by the computer each time. We decided to use a smart, dedicated computer, equipped with a degree of super-intelligence, though very limited compared with the human mind. We decided to call it Nick, in memory of one of the men who came up with the idea of super-intelligence almost two centuries ago.

Professor Socholov now gave a long, deep sigh.

-Okay, okay, so tell me what happened - snapped Lady Christa.
-Well... Nick, the AI super intelligent computer… took over.
-Took over what?
-First of all, the job itself. The computer re-targeted the self-reproduction programme, immediately deleting the limit of 512 units.
-But did it keep the same objective, to produce robots equipped with empathy—or half-sentient, as you put it?
-Yes, Nick didn’t modify the original plan. But then...
-Out with it, Wlado, what else?
-Nick took over the central computing system of the Creation Factory. To tell the truth, we don’t know exactly what he did, but we’ve lost access to the factory’s computing system.

Lady Christa jumped to her feet.

-You mean you’ve lost control of the software of the entire factory?
-Regrettably, yes - then he added - we’ve been fighting a losing software battle with Nick. Nick is super-intelligent, far ahead of us. It’s a bit like mice measuring themselves against humans in terms of intelligence. We can never win.
-This is unforgivable, Wlado! You should never have set loose a super-intelligent computer without first testing what it could do. It’s like dropping a bomb without knowing the damage it could cause!

Christa paused to get her breath.

-I’m fully aware that the field of AI is still largely unexplored. But you should have been far more cautious. You’ve been extremely superficial! I really didn't expect that from you. I’ve no choice but to report you to the Inquiry commission, which means you’ll lose your job and probably your freedom.
-I know, Christa. I haven’t slept for the last two nights.
-That’s not much help, Wlado! We need an Emergency Council. Now. I’m going to have to call a meeting of the Triumvirs immediately, along with the General Directors of all the Towers.

She strode up and down angrily for a couple of minutes, then asked:
-Has Nick got access to Internet?
-No, he hasn’t. We were able to disconnect it off in the nick of time...
-Tell me something else: can you follow what the robots are up to on video-camera?
-Yes, Christa, we’ve been filming everything they do.
-I see.

She was half-way out of the door, then stopped and asked:
-One last point: how much autonomy do they have? I mean, energy-wise?
-Unlimited. In the sense that they’re connected directly to the Seven Towers’ main power grid in Tower 5, a source of practically unlimited energy.

Lady Christa came back into the room.

-On second thoughts, before I call a meeting with the Triumvirs, let's have a preliminary session with my psychology professors and your scientists. Make enough room in the lab where you work, Wlado, to seat about a dozen people in front of the video-cameras filming the robots. Hurry! You’ve got less than an hour!

It was an interesting group that Triumvir 2 had assembled at such short notice. Sitting around the lab were Professor Socholov, the three new professors of robot psychology, and GST1, whom Lady Christa disliked but had summoned out of political necessity at this critical juncture. Also present were Professor Gregory Gimsky Korsakov, the guru of brain algorithms, and Sir William Orchid, the leading expert on consciousness. Three members of Socholov’s research team were also there, one of whom being Elizur Levi, an AI enfant prodige that Christa liked a lot.

She opened the meeting with a brief account of the situation. Socholov clarified a couple of points. Christa paused, then added:
-There’s something strange about all this. Usually, according to the literature, super intelligence is supposed to arise as a consequence of the creation of an artificial human mind. But here we’re talking about a super-intelligence without the preliminary stage of a human mind.
-Yes. It’s an aspect that’s actually been overlooked in the literature, which is why it caught us by surprise - said Socholov - the fact is that machine learning, too, can give rise to recursive self-improvement, to a step-by-step increase in software intelligence.
-A super-intelligence, then, without a mind?
-Right. A super-intelligence dedicated only to the task originally present in its software. But it could widen the scope of this task by sharpening its tools...

It was clear to everyone present how serious the problem was. But the General Secretary of Tower 1 seemed particularly uptight:
-Am I right in thinking - he spluttered - that these hyper-intelligent robots could use the whole of the Creation Factory to make copies of themselves, tens of thousands of them? With the result that all the robots made by the factory will become identical to them, instead of having different functions for many different purposes?
-I’m afraid that’s a possibility, GST1.

The General Secretary was now quite distraught, but he made an effort to remain calm and said:
-Can you explain to me, my dear fellow, just how a software programmed by experts like you could “take over”, as you put it? A machine like a computer cannot think, therefore your Nick, as you call it, cannot decide to take over anything! It sounds completely crazy to me!

Professor Socholov answered with the cool patience of someone addressing a young child:
-First of all, intelligence, or super-intelligence, as it’s called in AI, is not the same thing as the intellect of the human mind. Reasonableness or common sense are human prerogatives. These don’t exist in AI yet. A four-year-old child has a mind which is still way above the highest level of AI.
-How is that possible? - demanded GST1.
-Let’s take the example of that computer which is the all-time world chess champion.
It was young Elizur Levi speaking:
-Listen: each time the computer plays, it improves its game, it gets better, right? Now, a computer like that can play against itself a million times a day. Each time it gets a little better. Can you figure out what that means, in terms of intelligence?
-By intelligence here, you mean intelligence only for playing chess?
-Exactly, and nothing else. However, the software can still improve itself, by refining the programme and sharpening the tools required to fulfil it.
-Wait a minute!

It was OdinaTruang, one of the newly-appointed professors of robotic psychology. She was an elderly lady of few words, but when she did speak everyone stopped talking and listened.
There’s something fundamentally wrong with the analogy of the brain of a four-year-old child. The child as such may be four years old, but its brain is six or even seven million years old: it’s the product of evolution. This means that attempts by AI scientists to start from a tabula rasa, to take a completely empty vessel and fill it up with modern mechanical, artificial tools in order to manufacture a human mind, are utterly futile.

The other two professors of Robotic Psychology sitting next to her nodded their heads vigorously. But there was a roar of protest from the other side of the lab, especially from the scientists sitting around Professor Socholov. Gregory Gimsky Korsakov was most indignant and stood up shouting:
-That’s not what we do! We don't start from an empty vessel!
-We really aren’t that naive! - growled Socholov.
-Hold on, hold on! Let's not get lost in theoretical discussions! - cried lady Christa - this is a completely different kettle of fish. Recreating a human mind has always been and still is the ambition of AI, but that’s not what’s at stake here. Forget about it! We’re far away from the complexity of the human mind in our present case. What we have here isn’t a human mind. It’s a highly focused, but narrowly focused, super-intelligence without a mind.
-But can this super-intelligence turn nasty? - GST1 cut in.
-Not really, not this one - replied Socholov - but imagine what might happen if an artificial human mind were to have almost unlimited computational speed plus the full potential of our Internet—an unlimited memory archive containing all the knowledge in the Universal Wikipedia. Add to this the capacity to carry out all kinds of statistical forecasts, and what you get is an unimaginable degree of superhuman intelligence, which would reduce us humans to irrelevance and turn us into slaves.
-The literature of the past is full of this kind of scenario, where a super-intelligence with a human mind and an unlimited computing capacity takes possession of the world - pointed out Sir William Orchid - I remember two very old, prophetic books, one by R.K. and the other by N.B. They predicted that once we succeed in generating an artificial human mind, this danger will become very real. Our own intelligence will be minuscule compared to that of the machine, like an ant to an elephant. They baptized this world-dictator a "singleton".
-Do I have to keep reminding you, ladies and gentlemen? - thundered Lady Christa - that this is not the issue at hand. Forget the singleton and all this stuff about the human mind. We have no idea when AI will be able to replicate one. Let’s focus on this moderate super-intelligence that has just taken over the software of the entire Creation Factory. Who knows what it will do with it next!

The others present began to whisper among themselves in an atmosphere of fear and confusion. Lady Christa thought fit to intervene again:
-Nevertheless, we shouldn’t forget that everything Nick can do is strictly confined to the task for which it was built. There’s no reason why Nick should want to harm humans or the Seven Towers. We’re gathered here to discover a way of defeating this superintelligence by using our human common sense and reasonableness.

A long, tense silence ensued.

-You told me, Professor Socholov - continued Christa - that these robots are also partially sentient: they have a “feeling-of-being”, at an individual level. Isn’t that so? Well, tell me: is there any way we can exploit this aspect of their makeup?
-Are you thinking of appealing to their semi-consciousness to get them to take a collective stand against Nick? - said Socholov, shaking his head vigorously - the robots are completely under Nick’s control. They would never respond to any other signal. What’s more, at the moment we’ve no idea how to communicate with them.
-If that’s the case, then let’s change tack completely. I’ve already asked you about the source of their energy, but how are they actually recharged?
-They’re recharged automatically, using traditional energy. They have automatic sensors linked to the main power grid. It’s all regulated automatically by AI sensors…
-Wlado! - Lady Christa snapped - I don't care about your AI sensors! I want to know exactly where the energy comes from. Do we have a main power source? And if I dropped a bomb on it, would the energy supply be cut off?

A shocked silence followed. It was suddenly broken by Elizur Levi:
-There is a possible way out! Listen! The energy of the robots doesn’t come from the Creation Factory itself, but from an independent source, a cold fusion plant in Tower 5 that supplies the Seven Towers as a whole. Which means that Nick hasn’t yet taken possession of it!

Lady Christa looked hard at Professor Socholov:
-Can we get hold of this power source before Nick does? - asked Lady Christa - how long would it take to Nick to create a new source?
-Nick could design a cold-fusion energy source in a few seconds, but he can’t make anything without operators. He needs somebody or something to build it.
-Is there a way of cutting power off completely?
-Not by computer any more. All web connections are blocked.
-We can do it manually, then! - exclaimed Elizur, jumping to his feet - the main energy accumulator is in the entrance to the factory floor. I know exactly where. And nearby is the mains switch, a big metal lever which has never been switched off before. To get at it, you need to smash a glass window with a hammer.
-Elizur... are you capable of doing that?

Lady Christa’s words hung in the air. The others looked at each other without speaking.

-Of course I can do it! - replied Elizur.
-But it could be dangerous! - cried Professor Socholov.
-Can you see any other way of getting out of this mess, the mess you created, Wlado? - Christa sounded quite hostile this time.
-No, Lady Christa. You’re quite right. Cutting off their power supply is probably the only way to stop them. They’ll automatically go into standby when their energy falls below the critical limit.
-Can you reactivate the robots afterwards?

One of Socholov’s colleagues answered:
-Yes, I think we can. We’ll need a lot of help from Monna Lisa, but we should be able to do it.
GST1 also had a question:
-But what will Nick be doing while this is going on?
It was Elizur who replied:
-Let’s go back to our example of the computer chess champion. Suppose it had the perfect software to win any possible game, but nobody came to play. What would it do?
-It would just wait, I guess.
-Exactly. That’s just what Nick will do. Stand by and wait. Time is critical for us humans, but for a machine, an hour or a year is the same thing. Or rather, they have no meaning whatsoever. Nick will wait forever for his soldiers, the robots, to wake up again. Well, if you give me the go-ahead, I’ll switch off the power now. I know exactly where the glass cabin is. But first I’ve got a request.
-Which is? - asked Professor Socholov.
-As you know, the software programme we gave Nick has a subroutine for tracking the “biological” evolution of the robot's empathy. I’d like to do a simulation with this programme, to see where it might lead. I’d like to see what the robot’s “biological” evolution could achieve in, say, a thousand generations.
-Fine - agreed Socholov - you can start the simulation now, it won’t interfere with the rest. Then off you go… and good luck! Oh, and thank you, Elizur!

All eyes were glued on the largest screen. It was a bird’s-eye view, and they could see the robots moving quickly around the shop floor on their electric tracks. Each of them was assembling a copy of itself.

-There are now sixty-four - said Socholov - which means that they’ve reached step four. Soon there’ll be a hundred twenty-eight of them.
-Look! Elizur’s entering the shop floor.

The athletic figure of Elizur towered above the small robots. He was moving calmly, stepping around them carefully. He was carrying a heavy hammer.

-They don’t seem to notice him - said Lady Christa, visibly relieved.
-Everything will go dark in the factory if he succeeds in breaking the glass and pulling the lever - said Socholov nervously - I hope he can find his way back.

The video camera was now zooming on Elizur, who had stopped in front of a small glass cabin. He looked around, raised the hammer and hit the glass wall violently, shattering it. The robots continued to bustle about, paying no attention to the noise. Elizur edged inside the cabin and they saw him pull a lever on the wall. Almost immediately the workshop went dark.

-It’ll be a few hours before the robots notice their energy loss. Elizur has plenty of time to get back to us - said Socholov with a sigh of relief.

When Elizur came back into the room, he was greeted by a long applause. He sat down, breathing heavily.

-By tonight they’ll all be in standby - he confirmed - but now, let's have a look at the simulation. Let’s see how their genetic programme would have evolved in fifty thousand years. Of course, we know very well that this isn’t the evolution of the robots, or their behaviour, as such. It’s only the evolution of their software. Nothing more.

He stood up and turned on his computer. At first confused images appeared. Then the screen cleared.

-It looks as if the robots are clustering together in small groups - exclaimed Lady Christa.
-Right - responded Elizur - as if they want to communicate with each other.
-A kind of collective impulse? A far cry from the isolated individuality we’ve just seen today!
-It looks as if they’ve learned to work with each other!
-Do you think, Wlado, this could imply a kind of collective consciousness? Or at least some kind of cooperative spirit?
Elizur answered for him:
-It suggests that their genetic evolution may lead to the victory of empathy and mutual understanding over intelligence.

Christa and Wlado now smiled at each other like in the old days, while the rest looked on, without saying a word. At last, Lady Christa said:
-Well, if that’s the case, there’s hope for humanity after all!