LLMs with access to the internet are usually about as factually correct as their search results. If it searches someone’s blog, you’re right, the results will suck. But if you tell it to use higher quality resources, it returns better information. They’re good if you know how to use them. And they aren’t good enough to be replacing as many jobs as all these companies are hoping. LLMs are just going to speed up productivity. They need babysitting and validating. But they’re still an extremely useful tool that’s only going to get better and LLMs are here to stay.
That is the thing, they are not “only going to get better” because the training has hit a wall and the compute used will have to be reduced since they are losing money with every request currently.
Technology these days works in that they always lose money at the start. Its a really stupid feature of modern startups IMO. Get people dependent and they make money later. I don’t agree with it. I don’t really think oir entire economic system is viable though and that’s another conversation.
But LLMs have been improving exponentially. I was on board with everything you’re saying just a year ago about how they suck and they’re going to hit a wall even. But the don’t need more training data or the processing power. They have those and now they’re refining the LLMs. I have a local LLM on my computer that performs better than chat GPT did a year ago and it’s only a few GB. I run it on a shitty laptop.
I experimented with quite a few local LLMs too and granted, some perform a lot better than others, but they all have the same major issues. They don’t get smarter, they just produce the same nonsense faster (or rather often it feels like they are just more verbose about the same nonsense).
I don’t know what to tell you. I have them successfully compiling tables of search outputs to compare different things for method development and generating code, saving me hours of work each week. It all needs to be checked, but the comparison comes with links and the code is proofread and benchmarked. For most of what I do it’s really just a jacked up search engine, but it’s able to scan webpages faster than me and that saves a lot of time.
As a hobby, I also have it reading old documents that are almost illegible and transcribing them pretty well.
I really don’t know what you’re doing that you’re just getting nonsense. I’m not.
One other comment pointed me at one issue that might be a major difference. Is the code you generate in one of those ultra-verbose languages like Java where we had basically IDEs generating code from much shorter descriptions already 20 years ago? I could see LLMs doing well with those.
I tend to try to generate code mostly in Rust or sometimes shell or config files or DSL for various programs and 99% of the time the code does not even come close to what I wanted it to do, mainly because it just hallucinates itself some library interfaces that do not exist.
Not super common or super niche. I use R. And it completely made up code a year ago. Sometimes I still does, but less. And when I ask it for citations it can make shit up too. I really stand by the assertion that it needs a lot of babysitting.
But, between it getting better and me getting better at asking and some patience, I get what I want. But, it does require a lot of fine tuning and patience. But its still just faster than googling. And I could see the argument that the models haven’t improved but that they just have access to search engines now and that I’m mostly using them and a search engine. And sometimes they’re so whacked out I’ll ask them to search for something but theyll tell me they don’t have access to the internet and they’re so absolutely convinced of that that I have to close that chat and start a new one.
If you feed it in documentation or ask it to search for its answers in substack (or really just whatever search constraints you want) and then tell it to give you the links it used, you might have a better time. This forces it to look up an answer instead of hallucinate one. And when it gives me code, more complicated things usually fail pretty hard at first and I have to feed it the error output for a few rounds and guide it a lot.
LLMs with access to the internet are usually about as factually correct as their search results. If it searches someone’s blog, you’re right, the results will suck. But if you tell it to use higher quality resources, it returns better information. They’re good if you know how to use them. And they aren’t good enough to be replacing as many jobs as all these companies are hoping. LLMs are just going to speed up productivity. They need babysitting and validating. But they’re still an extremely useful tool that’s only going to get better and LLMs are here to stay.
That is the thing, they are not “only going to get better” because the training has hit a wall and the compute used will have to be reduced since they are losing money with every request currently.
Technology these days works in that they always lose money at the start. Its a really stupid feature of modern startups IMO. Get people dependent and they make money later. I don’t agree with it. I don’t really think oir entire economic system is viable though and that’s another conversation.
But LLMs have been improving exponentially. I was on board with everything you’re saying just a year ago about how they suck and they’re going to hit a wall even. But the don’t need more training data or the processing power. They have those and now they’re refining the LLMs. I have a local LLM on my computer that performs better than chat GPT did a year ago and it’s only a few GB. I run it on a shitty laptop.
I experimented with quite a few local LLMs too and granted, some perform a lot better than others, but they all have the same major issues. They don’t get smarter, they just produce the same nonsense faster (or rather often it feels like they are just more verbose about the same nonsense).
I don’t know what to tell you. I have them successfully compiling tables of search outputs to compare different things for method development and generating code, saving me hours of work each week. It all needs to be checked, but the comparison comes with links and the code is proofread and benchmarked. For most of what I do it’s really just a jacked up search engine, but it’s able to scan webpages faster than me and that saves a lot of time.
As a hobby, I also have it reading old documents that are almost illegible and transcribing them pretty well.
I really don’t know what you’re doing that you’re just getting nonsense. I’m not.
One other comment pointed me at one issue that might be a major difference. Is the code you generate in one of those ultra-verbose languages like Java where we had basically IDEs generating code from much shorter descriptions already 20 years ago? I could see LLMs doing well with those.
I tend to try to generate code mostly in Rust or sometimes shell or config files or DSL for various programs and 99% of the time the code does not even come close to what I wanted it to do, mainly because it just hallucinates itself some library interfaces that do not exist.
Not super common or super niche. I use R. And it completely made up code a year ago. Sometimes I still does, but less. And when I ask it for citations it can make shit up too. I really stand by the assertion that it needs a lot of babysitting.
But, between it getting better and me getting better at asking and some patience, I get what I want. But, it does require a lot of fine tuning and patience. But its still just faster than googling. And I could see the argument that the models haven’t improved but that they just have access to search engines now and that I’m mostly using them and a search engine. And sometimes they’re so whacked out I’ll ask them to search for something but theyll tell me they don’t have access to the internet and they’re so absolutely convinced of that that I have to close that chat and start a new one.
If you feed it in documentation or ask it to search for its answers in substack (or really just whatever search constraints you want) and then tell it to give you the links it used, you might have a better time. This forces it to look up an answer instead of hallucinate one. And when it gives me code, more complicated things usually fail pretty hard at first and I have to feed it the error output for a few rounds and guide it a lot.