Convex optimization for the densest subgraph and densest submatrix problems