People keep anthropomorphizing AI. Here's why

To use a horrible but tech-friendly analogy, language seems to act as a type of "bytecode" for facts and ideas, which can be chopped up and combined according to fairly simple combinatorial rules. This always works syntactically, and can lead to correct answer prediction without actually understanding any of the underlying semantics. It is analogous to a programmer combining functions in C# purely by compatible type signature from the compiled DLLs: in many cases this will generate plausible programs, and sometimes even correct ones, especially if the DLL was written by a careful programmer. But it can also lead to critical mistakes no human dev would make if they knew what they were doing. Decompiling such a program, assuming it was written by one person, and trying to figure out what the dev was thinking would be confounding: brilliance alongside bizarre negligence and boneheaded errors. The mystery is explained once you understand how the program was written, and that most of the brilliance was inherited, the rest was accidental.