I studied transformer architecture models and have played around with them (unfortunately) enough to understand how they work. Under the surface the model produces what look like XML tags <thinking> </thinking> to designate which tokens are thinking tokens and which are "normal" output. That is literally the only hard difference between the two output modes. The reinforcement learning might tune the thinking to be more like "what a human would expect to see in a thinking block" but it's still the same RNG madlib process generating everything underneath and any attempt to ascribe intelligence to this process should be met with lethal force incredulous cynicism.

Just like any claim that "we don't know how they work" - actually yes we know exactly how they work. What we can't comprehend is the exact numbers and weights inside the massive pile of probabilistic algebra being processed to generate your slop. If I flip 5 coins in a row and the observer's belief is anything other than "you just got very lucky" most people would call them crazy rather than join the cult and worship the coin god...