Searle's Sleight-of-Hand

John Searle's Chinese Room thought experiment proposes^[0]:

Suppose that artificial intelligence research has succeeded in programming a computer to behave as if it understands Chinese. The machine accepts Chinese characters as input, carries out each instruction of the program step by step, and then produces Chinese characters as output. The machine does this so perfectly that no one can tell that they are communicating with a machine and not a hidden Chinese speaker.

Now suppose that Searle is in a room with an English version of the program, along with sufficient pencils, paper, erasers and filing cabinets. Chinese characters are slipped in under the door, he follows the program step-by-step, which eventually instructs him to slide other Chinese characters back out under the door. If the computer had passed the Turing test this way, it follows that Searle would do so as well, simply by running the program by hand.

Searle asks whether this would mean that he speaks Chinese, and concludes that it is "quite obvious" that he does not. By analogy, the original computer must also not speak Chinese in the sense of being an intelligent Chinese speaker, and the same could be said of any language, therefore it is not "intelligent" in the way a person is.

There are various responses to this argument^[1], but my fundamental issue with it is that it abuses our intuitions by distorting key aspects of the scenario. Taken literally, the scenario Searle starts to lay out does not play out at all the way he concludes. A more realistic sequence of events is:

1. I slide a Chinese message under the door
2. Searle begins to execute the program
3. Searle and I both die (hopefully of old age)

The careful reader will note that at no point did I receive a response from Searle, so there is no analogy to the machine that responds to my queries in Chinese.

Searle would likely object that I am being overly literal here, and that I should assume he is able to execute this algorithm fast enough to give me responses on some reasonable timeframe. In this case the sequence of events would be:

1. I slide a Chinese message under the door
2. Searle begins to execute the program, his hands rapidly approaching the speed of light
3. Relativistic effects impart ever-increasing amounts of mass to Searle's hands
4. Surrounding space collapses into a black hole, consuming me and Searle

Once again, at no point did I receive a response!

Perhaps Searle would object that I should assume the laws of physics do not apply to him in this thought experiment. I am supposed to imagine a Searle who can move at the speed of light, unbound by relativity, immune to gravity. One who could destroy the world with a wave of his hand but in this room chooses instead to use this power to write Chinese. What does my intuition say about this man? Firstly, it screams that he is not a man, but a god. Does this god speak Chinese? It is not intuitively clear to me that he does not.

Another possible objection is that I should assume the program is simple and fast to execute. In this case I agree with Searle's conclusion — a simple program cannot be intelligent! But this is not the claim Searle made.

My mental model of Searle responds: are these not all differences in degree, rather than differences in kind? Surely a machine cannot go from unintelligent to intelligent merely by being faster? But I would respond — is there not a difference in kind between a drop of water and an ocean? A grain of sand and a planet? Would your intuitions about a drop of water or a grain of sand really be expected to tell you about oceans and planets? A sufficiently large difference in degree is a difference in kind, and this is very much the scale of the difference between Searle's pen-and-paper calculations and today's cutting-edge AIs.

0. ^

   This version is from Wikipedia, which is more succinct but effectively the same as his original

1. ^

   I also agree with the System reply