Density functional theory can be generalized to mixtures of ground and excited states, for the purpose of determining energies of excitations using low-cost density functional approximations. Adapting approximations originally developed for ground states to work in the new setting would fast-forward progress enormously. But, previous attempts have stumbled on daunting fundamental issues. Here we show that these issues can be prevented from the outset, by working from a fluctuation dissipation theorem (FDT). We thereby show that existing exchange energy approximations are readily adapted to excited states, when combined with a rigorous exact Hartree term that is different in form from its ground state counterpart, and counterparts based on ensemble ansatze. Applying the FDT to correlation energies also provides insights into ground state-like and ensemble-only correlations. We thus provide a comprehensive and versatile framework for ensemble density functional approximations.