TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs

However, the method of selecting -1, 0, and 1 to determine the summation and subtraction may not be optimal, as there are only a limited number of combinations of -1, 0, and 1, leading to repetitive computations for the corresponding A entries. Furthermore, when increasing computation parallelism by duplicating the selection unit of the adding and subtracting path, the resource consumption of the selection may exceed that of the TL tables themselves. This is because the multiple reading ports of the on-chip distributed RAM unit can support multiple accesses to the TL tables, requiring only additional buffers for addressing. The supportive ablation study will be presented in the next subsection. Algorithm 1: TL-based Ternary Matmul Input: A: Input activation stream (shape r M sr N s); W idx = Offline preprocess(W): Offline-preprocessed weight indices (shape r N {p T G qsrK s); Output: O: Output activation stream (shape r M sr K s); Initialize:; TL TABLEr N sr 3G s Ð 0; // Table for all signed combinations A BLOCK r T ˆ G s Ð 0; // Activation buffer O BLOCK r K s Ð 0; // Output vector accumulator for i Ð 0 to M 1 do for j Ð 0 to N 1 step T ˆ G do // Load activation block for p Ð 0 to T ˆ G 1 do A BLOCK r ps Ð A.read(); end for // Set up values of TL_TABLE for t Ð 0 to T 1 do val 1...G Ð A BLOCK r t ˆ G: p t ` 1 q ˆ G 1s; TL T ABLE set up pval 1...G q; end for // Process hidden dimension for m Ð 0 to K step Q do for n Ð 0 to Q 1 do idx vec Ð B "Y j T ˆG ]ı r m ` n s; for t Ð 0 to N 1 do TL TABLE idx Ð idx vecr t s; O BLOCK r m ` n s Ð O BLOCK rm ` n s ` TL TABLErt sr TL TABLE idx s; end for end for end for end for // Write output for p Ð 0 to K 1 do O .writep C BLOCK r p sq; O BLOCK r ps Ð 0; end for end for Function Offline preprocess(W): return Encode every G value as an index in the matrix and pack every T values as a index vector idx vec; Function TL TABLE set up(val 1...G): return return all 3G add and subtract combination; As described in Algorithm 1 and Figure 1, the TL-based matmul can be divided into two stages: (1) preprocessing the weights into groups sized G, and (2) performing the online ternary matrix multiplication computation. Figure 1: Dataflow and architecture of TL-based ternary matMul (G " 4) In the preprocessing stage, assume that every G " 3 ternary values are packed into a single index for TL table addressing, resulting in 3 G " 3 ˆ 3 ˆ 3 " 27 combinations. The index representation for this packing requires log 2 27 5 bits. Let A P N M ˆ N and W P ternary N ˆK . The preprocessing of the weights involves encoding every group of G " 3 ternary values into a 5-bit packed index.