news.ycombinator.com• Apr 3, 2026• 1 min read
ARCHE3-7B – Sparse Moe with SmartRouter and Foundation Curriculum TrainingThis is my first post on HN — a bit nervous, but excited to share what I've been building.I’ve been working on a 7B sparse Mixture-of-Experts prototype that can actually run on consumer hardware. For example, on a Colab T4 it uses around 5 GB RAM and 5 GB VRAM during training, and roughly 3.5–5 GB for inference.A couple of things I spent a lot of time on:Routing (SmartRouter) I tried to tackle routing collapse in a practical way. Instead of letting all tokens dump into a few "favorite" experts, I combined a few things: load balancing loss, an entropy bonus to keep the distribution flat, jitter noise during training, and a learnable temperature. It works surprisingly well at keeping a good portion of experts active. I’ve open-sourced the router code (hive_router.py) if anyone wants to look at the math or grab it for their project.Foundation Curriculum Training (FCT) Before standard pretraining, I run the model through structured reasoning patterns — currently 290 of them across 14 cogni