Some Hand Wavy Estimates on FLOPs and Computational Reducibility
This is quite facilitating…
My estimate for the ratio (Model Time/Human Time) was in the order of 10000 for medium complexity tasks.
It reduces to ~1000 as complexity goes up to competition level.
Now it’s merely ~12 for Putnam!
Some thoughts on limits of AI 🧵 https://x.com/DanHendrycks/status/1865858756040704335
I feel there is general trend that as complexity of problem increases, computational reducibility (as popularized by @stephen_wolfram) seems to be decreasing.
Problems have lower bound on amount of FLOPs you must spend with maximum computational reducibility.
It also appears that, for highly complex problems, top humans are able to get to pretty close to optimal computational reducibility.
If this remains true, the performance difference may entirely just come down to simply who can spend FLOPs faster and for how long.
The consequence: ASI might not have that amazing magic wand.
ASI can have O(1) FLOPs/sec advantage and being able to run without sleep/distractions.
We can also spin up trillion instances but I suspect there is limit on distributed bound as well.
Take for example, solving Riemann Hypothesis.
I would estimate about 10M Human-hours spent on this problem so far. Let’s assume that ~30M still needed.
If ASI is 1000X more faster, it will still need 3.5 years of continuous run to solve it!
This also points to long context short coming of our current architectures.
With our current long context schemes, we can barely manage perhaps up to 10 mins of “thinking”.
To solve something like Riemann Hypothesis we need 5-6 orders of improvement!