Computing a Mind: Turing, the Chinese Room, and AI

The question “Can computers think?” turns on what it means for a machine to think. The Turing Test is a canonical attempt that provides a behavioral criterion: a machine counts as thinking if it passes as a human in conversation (Turing, 1950, p. 434). Searle’s (1980) “Chinese room counterexample” (p. 421) famously challenges this view by imagining a non-speaker of Chinese producing fluent replies solely by following symbolic instructions. Decades later, the Turing Test and the Chinese Room Argument still mark the fault line in new AI breakthroughs (Borg, 2025, p. 2). Behavioral evaluations are increasingly common (Ni et al., 2025, p. 6), with large language models (LLMs) achieving “human-level performance” on exams (OpenAI, 2023, p. 6) and “significantly above chance” win rates in Turing Test studies (Jonas & Bergen, 2025, p. 8). Consequently, LLMs can increasingly function as “an ordinary conversational partner” (Borg, 2025, p. 6), creating “the very strong impression of linguistic understanding” (p. 5). Furthermore, AI has become more brain-like, shifting from “the old style of rule-based AI systems” targeted by Searle toward artificial neural networks (ANNs) that learn distributed, data-driven representations (Jackson, 2005, p. 31). Such architectural developments thus invite skepticism about how directly the Chinese Room Argument applies to contemporary AI. This paper explores the question of artificial mentality through the Chinese Room Argument, arguing that neither human-level behavioral performance nor increasing brain-like complexity in AI decisively challenges the Chinese Room Argument. By examining the argument and its application to neural networks, it contends that such developments still do not move AI beyond syntactic symbol manipulation.

Turing (1950) claims that the question “Can machines think?” is unclear because both “machine” and “think” are both vague terms, so it should be replaced with a more precise one (p. 433). He restricts machines to digital computers (p. 436), and proposes “the imitation game” as a behavioral test for thinking (p. 434). In this game, later known as the Turing Test, a human interrogator communicates in writing with a hidden respondent and must decide whether it is a human or a computer. If a computer can reliably pass as human, it passes the Turing Test and, as judged by this behavioral proxy for mentality, counts as a thinking being.

Contra Turing, Searle’s Chinese Room thought experiment aims to show that a system with the right input-output behavior cannot indicate that it “literally has cognitive states” (Searle, 1980, p. 417). Briefly, Searle adapts the imitation game into a test of whether a system understands Chinese: he imagines himself in a closed room, where Chinese speakers outside pass written questions into the room. Inside, he has a rulebook for manipulating Chinese symbols in response to the symbols he receives, which he identifies “entirely by their shapes” (p. 418). Searle in the Chinese Room behaves like a digital computer, processing Chinese symbols step by step according to rules. A digital computer consists of a “store,” an “executive unit,” and a “control,” with instructions stored internally as a “table of instructions” (Turing, 1950, p. 437). In the Chinese Room, the store is the room’s materials, the executive unit is Searle, the control is his rule-governed operations, and the table of instructions is the rulebook. Crucially, Searle himself does not understand Chinese, yet by simply following formal rules (e.g. if you see this symbol, write down that symbol), he can produce answers indistinguishable from those of a Chinese speaker. Thus, Searle argues that rule-governed symbol manipulation alone is not sufficient for understanding, even when it produces behavior indistinguishable from genuine understanding.

A well-known objection to Searle is the “Systems Reply”: the room as a whole understands Chinese, rather than the person operating inside (Searle, 1980, p. 419). Indeed, human cognition might emerge from certain interactions within subsystems, such as neurons, that do not possess understanding by themselves. Our neurons are programmed to process signals, the phenomenon of mentality might arise on the whole. Thus, parts of a computer that do not understand the meanings of symbols might still produce thoughts at a systems’ level. However, it is unclear how thoughts could genuinely arise from a book, a room, and a person operating. Thoughts have intentionality: the capacity of mental states to be about things (Searle, 1980, p. 421). When we think about “a man entering a bar”, the thought is really about a man entering a bar, and we know what it is to understand that. Our thoughts are genuinely directed at things, whereas even highly sophisticated AI systems operate on formal representations without referring to anything beyond those representations. LLMs process conversational inputs as tokens, encode them numerically, and contextualize them through their relations to other tokens in the sequence. Likewise, convolutional neural networks process images through layered feature detection, from edges and gradients to shapes and labels.

To see how Searle’s argument applies to such systems, imagine a building with hundreds ofrooms. The ground floor receives 3x3 pixel patches of a cat image and outputs numbers based on features such as gradients and intensities. The second floor combines these numbers to detect shapes and outputs further numbers. Higher floors extract increasingly abstract features until the building outputs the label cat with 99% confidence. None of the rooms understands what a cat is, though the Systems Reply suggests that understanding would emerge at the level of the building as a whole. Yet the building is still just a building: it no more understands cats than a single room understands Chinese, it simply distributes symbol manipulation across multiple components. Thus, computational organization alone, no matter how sophisticated and brain-like, is sufficient for understanding. Recent advances in AI have made the question of artificial mentality more pressing. Yet despite the impressive behavioral success of contemporary AI systems and their increasingly brain-like architectures, such computational systems remain within the bounds of syntactic symbol manipulation. For that reason, the Chinese Room Argument still poses a serious obstacle to the claim that a computer could genuinely think.

Work Cited

1. Borg, E. (2025). LLMs, Turing tests and Chinese rooms: The prospects for meaning in large language models. Inquiry. Advance online publication. doi:10.1080/0020174X.2024.2446241

2. Jackson, P. (2005, May). Turing’s test, Searle’s Chinese Room argument, and thinking machines

(Working Paper No. 2-05). The Open Polytechnic of New Zealand.

3. Jones, C. R., & Bergen, B. K. (2025). Large language models pass the Turing test. arXiv. arXiv:2503.23674.

4. Ni, S., Chen, G., Li, S., Chen, X., Li, S., Wang, B., Wang, Q., Wang, X., Zhang, Y., Fan, L., Li,

C., Xu, R., Sun, L., & Yang, M. (2025). A survey on large language model benchmarks. arXiv. arXiv:2508.15361.

5. OpenAI. (2023). GPT-4 technical report. arXiv. arXiv:2303.08774. Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(3), 417–457.

6. Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433–460.