Efficient Object Rearrangement via Multi-view Fusion