"As a reporter covering artificial intelligence for over a decade now, I have always known that running artificial intelligence brings all kinds of computer engineering challenges. For one thing, the large language models keep getting bigger, and they keep demanding more and more DRAM memory to run their model "parameters," or "neural weights." I have known all that, but I wanted to get a feel for it firsthand. I wanted to run a large language model on my home computer."
"Now, downloading and running an AI model can involve a lot of work to set up the "environment." So, inspired by my colleague Jack Wallen's coverage of the open-source tool Ollama, I downloaded the MacOS binary of Ollama as my gateway to local AI. Ollama is relatively easy to use, and it has done nice work integrating with LangChain, Codex, and more, which means it is becoming a tool for bringing together lots of aspects of AI, which is exciting."
"Running LLMs locally, rather than just typing into ChatGPT or Perplexity online, has a lot of appeal for not just programmers, but any information worker. First, as an information worker, you will be more desirable in the job market if you can do something like download a model and run it rather than typing into the online prompt just like every free user of ChatGPT. We're talking basic professional development here."
Ollama provides an accessible path to download and run open-source large language models locally, including a macOS binary and integrations with LangChain and Codex. Local LLM use appeals to information workers for professional development and for keeping sensitive data on-device. Large language models continue to grow and demand increasing DRAM to store model parameters or neural weights, which creates significant hardware constraints. Even smaller models can perform painfully slowly on underpowered machines. Practical local LLM use therefore typically requires a modern machine with ample RAM — 32GB is recommended — to achieve acceptable performance.
Read at ZDNET
Unable to calculate read time
Collection
[
|
...
]