A deep learning framework for jointly extracting spectra and source-count distributions in astronomy