Would anyone be willing to try my MoQ version? MoQ employs tensor scanning technology to determine the optimal precision, delivering better KLD and PPL results than the precision strategies built into llama.cpp.
Did you try to reimplement MoQ by @w-ahmad , or contacted them to get the magic sauce? Asking this because as I understood they didn't share the algorithm so far, apart privately with @bnjmnmarie Or is it a reuse of the name with your custom recipe?
Did you try to reimplement MoQ by @w-ahmad , or contacted them to get the magic sauce? Asking this because as I understood they didn't share the algorithm so far, apart privately with @bnjmnmarie Or is it a reuse of the name with your custom recipe?
My formula is indeed derived from theirs, but with some modifications; the results are similar. See details here:
While their specific algorithms are indeed not open-source, the weights themselves are; consequently, the quantization formulas can be derived directly from the weights, allowing anyone to quantize similar models using these recipes.
The key lies in how to derive these formulas—which constitutes the core of the algorithm—much like the dynamic quantization 2.0 used by Unsloth (MoQ's formula is actually better than it.), where the method for obtaining the specific formulas is not publicly disclosed.
The good news is that once a model's architecture is established, tensor sensitivity is essentially fixed; therefore, the optimal quantization formulas remain virtually identical for models sharing the same architecture.
As for "reimplementing" it—yes, I do have my own implementation of the MoQ algorithm, available here:
However, this is not a "plug-and-play" quantization algorithm; in practice, it requires a lengthy and complex process of scanning model tensor precisions, followed by statistical analysis and calculations, to determine the optimal configuration. Consequently, you cannot directly generate an MoQ model simply by using my branch.
once a model's architecture is established, tensor sensitivity is essentially fixed; therefore, the optimal quantization formulas remain virtually identical for models sharing the same architecture
Hmm I see. Well, thanks for sharing your fork, and your ggufs!
