Mesh LLM: distributed AI computing on iroh

I have a macbook pro, figured I'd see how easy it was to contribute some vram...

And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc.

It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!

This got me thinking about experiments with models talking to each other over WebRTC: https://xt-ml.github.io/shadow-claw/

Its sort of a "P2P mesh" :-) Watch four instances of the harness running together and collaborating on checking the weather: https://www.youtube.com/watch?v=h1les1A3gcg

The first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture.

As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.

Are we talking 1 token per second for a split model? Less?

Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

I'm more interested in running distributed inference for purpose built small language models than these coding LLMs.

Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.

Nicely done OP.

Does this support Qwen 3.6 (e.g. 27B) and the myriad of llama.cpp options (batch sizes, quantization, etc.)?

I'd love to see some performance data.

I've been looking for similar distributed computing style LLM, and I found AI Horde and a few other smaller efforts like one from Aphrodite people and distributed training from Nous Research.

AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.

I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.

I spent a while trying to get mesh-llm running, but none of the installable llama.cpp builds worked with my older gpu. It looks like it should be able to be used to proxy an external llama.cpp service, but I had no luck setting that up either. Seems very cool, but definitely some rough edges.

I’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow.

What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.

the https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.

Is this truly more secure though? The host can still see your data.

I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.

All these ASICS being designed and specialized for AI but none seem to be being built for consumers. Reason?

Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?

Is there a catch? If not, this would be super useful.

I knew this was possible, i asked chatgpt about a year ago and it said no the latency would be too big of a problem. I spent the best part of a year learning libp2p and was looking for a project to do with it at the time.

distributed AI computing so your hallucinations can be geographically diverse too

It sounds like iroh enables distributed compute without having to finangle custom hardware.

The real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.

cocompute.ai is already doing this really well.

difference between this and Exo?

Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.

I just wish I had the hardware to try it out!