# Proper losses for discrete generative models

Rafael Frongillo, Dhamma Kimpara, Bo Waggoner

Submitted on 7 November 2022

## Abstract

We initiate the study of proper losses for evaluating generative models in
the discrete setting. Unlike traditional proper losses, we treat both the
generative model and the target distribution as black-boxes, only assuming
ability to draw i.i.d. samples. We define a loss to be black-box proper if the
generative distribution that minimizes expected loss is equal to the target
distribution. Using techniques from statistical estimation theory, we give a
general construction and characterization of black-box proper losses: they must
take a polynomial form, and the number of draws from the model and target
distribution must exceed the degree of the polynomial. The characterization
rules out a loss whose expectation is the cross-entropy between the target
distribution and the model. By extending the construction to arbitrary sampling
schemes such as Poisson sampling, however, we show that one can construct such
a loss.