Large Margin Learning of Upstream Scene Understanding Models