We introduce a new agent-based framework for materials discovery that combines multi-fidelity modeling and sequential learning to lower the number of expensive data acquisitions while maximizing discovery. We demonstrate the framework's capability by simulating a materials discovery campaign using experimental and DFT band gap data. Using these simulations, we determine how different machine learning models and acquisition strategies influence the overall rate of discovery of materials per experiment. The framework demonstrates that including lower fidelity (DFT) data, whether as a-priori knowledge or using in-tandem acquisition, increases the discovery rate of materials suitable for solar photoabsorption. We also show that the performance of a given agent depends on data size, model selection, and acquisition strategy. As such, our framework provides a tool that enables materials scientists to test various acquisition and model hyperparameters to maximize the discovery rate of their own multi-fidelity sequential learning campaigns for materials discovery.