Fast Task-Aware Architecture Inference