A Byte Is All We Need