A Lightweight and Transferable Design for Robust LEGO Manipulation