Studies of the high-redshift rest-frame ultraviolet (UV) luminosity functions (LFs) have typically treated the star-forming galaxy and active galactic nuclei (AGN) populations separately, as they have different survey depth and area requirements. However, the recent advent of wide-area deep ground-based imaging surveys now probe volumes large enough to discover AGNs, at depths sensitive enough for fainter star-forming galaxies, bridging these two populations. Using results from such surveys as observational constraints, we present a methodology to jointly empirically model the evolution of the rest-UV luminosity functions at z=3-9. We assume both populations have a LF well-described by a double power law modified to allow a flattening at the faint-end, and that all LF parameters evolve smoothly with redshift. This provides a good fit to the observations, and makes predictions to volume densities and redshifts not yet observed. We find that the volume density of bright (M_UV = -28) AGNs rises by five orders of magnitude from z=9 to z=3, while modestly bright (M_UV = -21) galaxies rise by only two orders of magnitude across the same epoch. The observed bright-end flattening of the z=9 LF is unlikely to be due to AGN, and rather is due to a shallowing of the bright-end slope, implying reduced feedback in bright galaxies at early times. Integrating our LFs we find that the intrinsic ionizing emissivity is dominated by galaxies at all z > 3, and this result holds even after applying a notional escape fraction. We compare our AGN LFs to predictions based on different black-hole seeding models, finding decent agreement on average, but that all models are unable to predict the observed abundance of bright AGNs. We make predictions for the upcoming Euclid and Roman observatories, showing that their respective wide-area surveys should be capable of discovering AGNs to z ~ 8.