Can We Understand How Large Language Models Reason?

This article is not about "reasoning" in the abstract, philosophical sense but is talking about "mechanistic interpretability" research. The title is more like, "can we understand if the 'knowledge' encoded into a neural networks actually corresponds to reasoning-like concepts" and doing that with actual experiments like tweaking weights and activations.

There's an interesting example where researchers saw a model approached clock time calculations and calendar month-day calculations using the same methodology. So then is this because an underlying concept of "cyclical measures" has emerged in the network?

I personally would not look for the way they reason in the weights, at least not directly. In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights. I have a hard time imagining how you would even tell, at the level of weights and activations, if the next token being the is the result of some proper reasoning or a hallucination. But those weights do not exist for the sake of it, they encode a lot of text the model has seen during training, and I would imagine this is what drives the reasoning. Can you evaluate the following polynomial ... will be related to To evaluate a polynomial ... seen in the training data. This is the level at which I would look for the reasoning, memorized patterns how to do specific things, maybe with some kind of placeholder variables for generalization. Ultimately such a structure would of course also be represented in the weights but I could imagine that this makes it unnecessary hard to understand. Or maybe not, maybe the learned patterns are so complex that they do not have a simple representation.

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and you can see life near the end, how the delta of all the same movies opens up into wildly different endings.

To advance further it would need the ability to abstract away the general situation shape and pattern recognize similar situations.

>“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.”

what is the basis for this optimism ?

My toaster doesn't reason, and neither do the current clankers.

there's a 2MP about the related paper: https://www.youtube.com/watch?v=l72ufA-4SzE

One plausible reason I thought of that we may not understand neural nets is that by their nature their power grows with ever-more complex connections and weights.

So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be the case, and this is analogous a complexity curve.

And so any effort to make sense of such black box computation would be like trying to reverse entropy, analogous to trying to recover information lost in waste heat. And that could be one fundamental barrier to understanding both human and artificial brains alike, relative to their internal complexity.

(Just thinking aloud my handwavy pet theory recently, I am not an expert and could be totally mistaken on this)

Do LLMs have Qualia?

They don't reason.

Clickbait article title.

The article body does not presume they reason.

Do they ?

Well, if you read the foundational paper 'All you need is Attention', review the full stack trace of any LLM system call, and have insight to the ad hoc training process to ingest additional data and knowledge, you will gain greater understanding.

If you enjoy such content, please like and subscribe to my channel: xxXNoobSmasher69Xxx

Edit: bad faith actors with no sense of humor downvote this valid starting point of discussion.