I agree with most others that said that this is hardly a surprise. But there are still interesting aspects of your post worth discussing instead. First, more obvious one. What if I am delusional and go to talk to AI? Doesn't have to be this absurd, but there are already examples of this, like that guy that killed his mother because he believed she was a spy. Perhaps he was already delusional, but an LLM fed his delusion. Because LLMs aren't reasonable, they can't be made reasonable, and so any guardrails cannot be made to guard against all situation that reasoning is needed for. At the very least people need to be educated on what LLMs can and can't do and how they do it. I would even argue against guardrails, as they provide people with false sense of security.

Second, but less obvious point, related to the discussion - is the influence of training data and how training data is framed on what the LLM will say. Someone posted the response of a different LLM refusing to entertain the delusion. This is all done by manipulating the training data and framing it in a way a company making it wants. If the company wants its chatbot to be seductive, for example, that can be achieved with subtle preprocessing of training data. The point I want to get is we need to trust the companies with what narrative they want to push be good for society which is obviously a terrible idea. Just look at musks AI and what it did. This influences people whether we notice or not.

To sum up - we need heavy regulations on LLMs to make socially responsible use of them and not just how they can/can't be used, but also how they are trained, with what data and with what rhetorical goal (for the lack of better word).