Exploring Llamafile: Mozilla's Attempt in the World of Open Source AI
1y 8mon ago by lemux.minnix.dev/u/minnix in linux_lugcast@lemux.minnix.dev from itsfoss.com
The issue isnt if its easy to use the issue is a matter of compute. A majority of people on the internet only have mobile access. We need a way to let the masses utilise distributed compute in a secure and private way.
You can run many 7B models on phones with 8GB RAM
I find the 7b models dont have the capabilities id like and i cant image the tokens per second would be very good.
Yeah running it and it actually being useful are two very different things.