Practical Boolean Backpropagation

To reduce the computational complexity and memory requirements of models, various quantization techniques are widely used today. Floating-po int numbers are typically reduced to 16-, 8-, or 4-bit representations. This te chnique allows preserving the traditional gradient-based optimization of a differe ntiable loss function through backpropagation. More recent research is moving toward more extreme forms of qua ntiza-tion. For instance, [1] introduces BitNet b1.58, a novel 1.58-bit Lar ge Language Model (LLM) where each parameter is represented using ternary values { -1, 0, 1 } .