Rob Ricci on Nostr: I'm not convinced that they will keep growing at that rate - I think we are fairly ...

I'm not convinced that they will keep growing at that rate - I think we are fairly well along the road of diminishing returns for dollars-to-model-quality.

Yes, models are getting better. I agree.

But I don't think that it's currently true that if you spend 10x as much, you train a model that is 10x better. I'm not going to pretend that I know what the constants of this function are, but I think they are sub-linear. I think that a scenario of ever-increasing amounts of money spent for ever-decreasing marginal improvement is going to make further linear increase in models unlikely.

I'm also not convinced that number of neurons are a great proxy for intelligence - both in terms of comparison to synapses, and in terms of linear scaling. At some point, it used to be easy to convert more transistors on a chip into a commensurate amount of computing but now ... it's not so linear. I don't know if there is such a point of diminishing return for neural nets, and if there is, what it would be but - my hunch is that it likely exists. There are not a lot of things that scale linearly forever.

Thanks for this thought-provoking thread!