> “incompleteness theorems” established that no formal system of mathematics — no finite set of rules, or axioms, from which everything is supposed to follow — can ever be complete.'

There is usually a 'not sufficiently complex' clause in that definition. Presburger arithmetic is complete: https://en.wikipedia.org/wiki/Presburger_arithmetic

Right, you need to be able to construct numbers for Gödel's proof to apply.

Hilbert's incidence geometry, for instance, is consistent and complete. It's just rather small.

> "Even though I can motivate it in retrospect, ChatGPT’s idea to use h^2-dissociated sets to control relations of order at most h feels quite ingenious. As far as I can tell, this idea is completely original."

The question that keep bothering me is can an LLM generate an idea that is truly novel? How would/could that actually happen? But then that leads to the question - what are we actually doing when we think?

Perhaps it's as simple as the ability to just make mistakes that matters, the same things that powers evolution. As long as the LLM can make mistakes, it's capable of generating something genuinely novel. And it can make more mistakes much faster than we can.

Yes, they can.

Some people like to parrot "next token prediction", "LLMs can only interpolate", and other nonsense, but it is obviously not true for many reasons, in particular since we introduced RL.

Humans do not have the monopoly on generating novel ideas, modern AI models using post training, RL etc can come to them in the same way we do, exploration.

See also verifier's law [0]: "The ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI."

This applied to chess, go, strategy games, and we can now see it applying to mathematics, algorithmic problems, etc.

It is incredibly humbling to see AI outperform humans at creative cognitive tasks, and realise that the bitter lesson [1] applies so generally, but here we are.

[0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

I genuinely start to think that we, as humanity, severely overestimate our cognitive abilities. We act so surprised “just a few years of LLM with a few RL tweaks match our PhD levels! It must be hidden inside our knowledge base!”. Em, what if no? What if our “PhD level” is just very low level comparing to upper boundaries of measurable intelligence? What if we need to learn being humble and stop treating our minds as “sacred source of creativity and intelligence”?

RL or no RL, AI cannot escape the distribution it's trained on. It's just that the labs will put so much into the distribution that we won't be able to tell the difference that easily, nor will it matter for most tasks. The reason AI does well on ARC-AGI-2 is because the labs created synthetic training data using similar puzzles.

Yes it can! That's the whole point of RL! it generates slightly out of distribution rollouts, and rewards good rollouts to change the distribution of the output

That's not out of distributíon, that's inside the distribution of the rollout. If you don't create rollouts for the game of Chess then it doesn't know how to play Chess no matter how smart it is at tasks you've created rollouts for. It's structurally stuck in its distribution.

What if it doesn't need to escape the distribution, it can just exhaust the current distribution we have much more broadly and efficiently than humans can?

So the answers we're seeking to our bleeding edge questions are already there, we just need an AI's ability to target the answers. Then re-train on the improvements and go from there.

Just a thought.

Reinforcement learning for "reasoning" perturbs the model to generate completions in a particular chain of thought / alternative selection structure. It's three next token predictors in a trench coat.

When these things start solving many more long standing problems, and start introducing more novel problems, will people finally admit that the "next token predictor" is not the gotcha they think it is?

It's not a gotcha. It's incredible what these things can do despite being next token predictors from a weird dataset. That's at the heart of the "bitter lesson", and you don't have to believe in magic to see it.

My own take, and it's veering into the Philosophy of Mathematics, but there's a debate about whether Mathematics is "Invented" or "Discovered".

If it's "invented", then it requires ingenuity.

If it's "discovered", then it was always already there, just waiting for the right connections to be made for it to be uncovered and represented in a way we can understand.

Invention requires ingenuity, but discovery does not. So if LLMs can generate truly novel mathematics, for me that settles it that mathematics is indeed discovered, as LLMs are quite capable of discovery yet I don't consider them possible of invention.

Mathematical concepts are invented, but they live in a space of possible (conceivable) mathematical concepts, and we can only invent concepts from that selection of possible concepts. This can be reframed as a process of discovery regarding which conceptions are possible.

Furthermore, the results of theorems aren’t an invention, they are a discovery of what the base assumptions (axioms) logically entail. Finding out which theorems are true and provable is a discovery process. For example, the results of Gödel’s incompleteness theorems were a discovery. They weren’t invented, in the sense that the results couldn’t have been otherwise. We merely could have failed to discover them.

This also holds for physical inventions. You discover a working way to build some functioning mechanism. It’s a process of discovery of what is possible in the physical world.

Whether you portray somethings as a discovery or as an invention is more a matter of degree, a matter of from which angle one is looking at it.

The possible states of an LLM are finitely enumerable. The same likely holds for the possible states and configurations of a human brain, in approximation. Therefore there is only a finite set of possible ideas, thoughts, and conceptualizations an LLM or a human can have, and in principle they could be exhaustively enumerated and thus “discovered”.

I like this distinction, but it would then seem the only 'invention' would be the axioms of your mathematics. There exists numbers (natural, imaginary...), there exist shapes (a point, a line...). All the work from that point on could be 'discovered'. I agree that I don't see LLMs inventing in this way. But again, it raised the question - what are our brains doing when we 'invent' something?

Well, take any invention you like, and let's break it down.

Somebody at some point, "invented" the idea that the earth was round. Before that, the obvious "just look around you" answer would've been, duh of course the ground is flat. But we know the earth has always been round, even if humans couldn't appreciate it for hundreds of thousands of years (I don't count the pre-history before homo sapiens). So we "invented" some fields of science and the mental models / abstractions that allowed us to conceptualize what a round earth could mean and how to measure it, but we didn't invent the roundness itself -- that was always reality, and we just lacked both the thoughts and the tools to conceptualize it (until later).

Now you might say, well that is a category of "simple" physical observations. The earth is naturally round all the time and doesn't take any extra human effort to make it so (it took some effort to imagine that it could be and to find ways to measure/prove it). But what about say -- semiconductors, NVIDIA GPUs, that sort of thing? It's not like semiconductors grow on trees and we just need to find them and learn how to consume/use them... isn't that a better example of "true invention"?

Sure, I could see that. But I guess my POV would be that, the invention of the latest AI chip, or the first semiconductor, or the first vacuum tube, or whatever came before, all laddered largely incrementally on "discoveries" that were then cleverly tweaked or reapplied, so that what appears to be "true invention" is usually/more-often just another chain in a long chain of "discoveries" that led up to it. I grant you that some of what appears in hindsight to be continuous progress, really is built on small discontinuous "leaps", but I don't think that breaks the argument (strengthens it in fact, IMO). You wouldn't have semiconductors today, unless Faraday (or somebody like him) discovered that silver sulfide resistance decreases with heat, and that is more like one of those physical properties that reality has always had (much like, earth was always round, we just didn't know it at first).

So in that sense, I feel this becomes almost like an "evolution vs intelligent design" debate -- some people look at the complexity and miracle that is the human eye or the human brain, and they insist there must have been an intelligent designer, because surely no random chaotic biological process could have produced something so wonderful... And yet, I think the scientific evidence largely shows that, indeed that is what happened, just random chance + evolutionary-pressure was all you really need (plus billions of years). So if you can accept that analogical framing for a minute, then I would posit that "invention"-adherents are really making something like an intelligent design argument, vs "discovery"-adherents are saying that evolution (in an artificial sense, with the artificial selection pressures of scientific research, of capitalism, etc., and compressed into centuries or decades, not millions or billions of years) is sufficient to derive miraculous-seeming results. The little discontinuous leaps along the way, are kind of like the random mutations of genes that happen to confer an advantage -- maybe we can say that we are more intentional about seeking those leaps out, or maybe we are just right-place/right-time lucky (e.g. thinking about penicillin and the random petri dish left out).

Perhaps once (or if) there is the sort of leap that breaks us out from a Type I to a Type II+ Kardashev civilization, maybe then I would grant you something needed to be "invented" that couldn't be based on a line of "discoveries". Or maybe not, maybe it will just be another semi-random discovery.

Mathematical objects are an invention of the mind - they are abstract objects that only an entity who can process abstractions can make sense of.

There is no ‘discovery’ here nor was it waiting to be found. The human has to sacrifice and pursue the path of exploring reality and thereby is inherently inventing.

Humans built up mathematics iteratively from smaller bases extending into large ones. Is this what LLM’s do? Of course not - They are fed with vast amounts of information from the off.

Trivially the answer is yes by the infinite monkey theorem. If we allow the sampler to pick any token then any stream of arbitrary tokens can be generated. Therefore if an original idea can be represented with written words then a LLM can generate it. That is perhaps not the most satisfying answer, but if you want a better one you'll need to provide a function that determines if an idea is original.

For my paper about ME/CFS, I let an LLM integrate lots of findings of other scientific papers. Then I ask the LLM to "creatively brainstorm", given all we know of ME/CFS and the newly integrated paper, to generate new hypotheses, treatment ideas or any other kind of insight it can think of.

This works really well.

Now, it's clear that I have no idea how much of this is something we would consider new and original, and how much is a kind of systematic, but not novel, easy of thinking.

What I couldn't do so far is get an LLM to generate a truly new maths theory, with new abstract concepts and dimensions and points of view. The kind that is not just a combination of existing theories and logic.

Would you mind posting the outcome of this? A person I love dearly is struggling with Long Covid/CFS. I’ve been doing something similar to what you describe, but I’m always looking for more angles that could help.

It's about the ability to combine ideas in novel ways, without breaking the rules in relevant frameworks. Sometimes the idea may even be to contradict existing theories where they are weak.

How do you define a new idea?

To me, it's rearranging the information you had in a way that hasn't been applied or published before.

That's literally what LLMs are built for.

Theres a simple test for this.

Limit the knowledge an llm to some point in time at which a discovery was made. And check to see if the llm could produce the discovery.

If you think OAI hasn’t already tried this then think again - they have every incentive to do so and announce it to the world.

Please elaborate.

Some people have this strange idea that only "whatever humans do" counts as intelligence, despite the fact that a) we don't really have a clue what humans do, and b) "intelligence" is definitely not that strictly defined.

I think they're just trying to feel like they know some important truth that other people don't.

Agreed. I see this debate as an active discussion as to what intelligence is, not how it's currently (poorly) defined This is a philosophical discussion, and there is no correct answer, but IMO some answers will prove more useful than others. I would like to define intelligence as the ability to solve problems. Lots of other life forms have this ability, and its clear that LLMs also have this. Now, while they may not be poetic (in the literal sense of the word), or conscious, in that 'they' do not experience the world. I think there is a strong case for arguing they conform to a meaningful definition of intelligence. They solve problems.

Misprediction is?

That this was so predictable, is the hardest thing to process. A friend shared this video by Jiang Xueqin https://www.youtube.com/watch?v=7y_hbz6loEo&t=2s I find this guys hard to take seriously, his logic is erratic and often just absent. But his prediction has been frighteningly spot on regarding Iran. Towards the end he predicts American boots on the ground - and them turning into American hostages. I found that last part truly unbelievable until I heard Trump will have moved 3000 marines to the region by Friday.

I think you are missing the parent comment's point.

The point is not "this guy is a genius" but rather "this war was so predictable, even this weird guy could pinpoint with frightening accuracy how this war would happen two years before it started".

N.B. The video is from May 2004 (during the Biden administration)

I'm not sure UBI would work. But what's that got to do with communism? Communism failed (IMO), because it was incompatible with human nature. People form social hierarchies and like to own property. Would UBI prevent that?

You are answering my question - you posit that communism failed because people like to own property (and form social hierarchies) - and UBI doesn’t prevent those.

I’m not sure this is the case. Especially since people did form social hierarchies and did own propety in communist/socialist regimes of the Eastern block.

One might argue that UBI is incompatible with human nature, because most humans require a sense of worth, which paid work provides (to most). Take away the motivation, will the self-worthiness remain? (especially when thinking about _most_ people, not the struggling artists who wished they could pursue art without the burden of bills).

I’m not sure about this at all. I just think drawing the parallels and discussing communism failure modes is interesting for the discussion of UBI.

I remember thinking about this when the semantic web was first being discussed. If you think of it from the perceptive of a child, your first 'foundational' words are learned though direct experience. Then while you continue to learn words this way, we can also use those words we 'know' to define secondary or tertiary terms that we have no direct experience of. I'd like to see a graph like this with someones take on the minimum number of necessary foundational words and how that graph would look.

> If you think of it from the perceptive of a child, your first 'foundational' words are learned through direct experience.

And lest we (or AGI) forget, there's qualia in the foundations.

I think it becomes easier with age - but I went through a period where I found it very hard. This book changed the way I thought about things - you may find it helpful: https://en.wikipedia.org/wiki/Zen_Mind,_Beginner%27s_Mind

The other difficulties with older texts is not just the different spellings or the now arcane words - but that the meaning of some of those recognisable words changed over time. C.S. Lewis wrote an excellent book that describing the changing meanings of a word (he termed ramifications) and dedicated a chapter to details this for several examples including ‘Nature’, ‘Free’ and ‘Sense’. Would highly recommend a read. https://en.wikipedia.org/wiki/Studies_in_Words

I also recognise ‘thinking’ as a valuable and enjoyable activity - and have also obsessed about problems for days (even in my sleep)to reach a deeper understanding. But I think the issue here is the impact of AI on just that - ‘understanding’. It might be true that with AI we no longer need to have as deeper understanding in order to ‘ship’. And even if that is true I think there are plenty of other domains you can think deeply about without AI getting in the way e.g. mathematics or philosophy - where is object is often to understand in and of itself, not just the products it may generate.

This does not surprise me - and America is a big place, and I'm sure there are areas where Arthur would be seen in a similar light but I've worked in the US and the UK and this type of things reminds me of the phrase 'separated by a common language'. Slightly off topic perhaps but another area where I see a strong divide in sensibilities are the NewYorker cartoons - my wife (born in north America) thinks the are hilarious. I usually don't understand what's funny about them.