NoiseSearch
Metaheuristic search over diffusion model noise achieves better performance than best-of-N sampling (early work on test-time scaling).
Links
Paper: PDF (much more information can be found in the paper)
Code: aklein4/NoiseSearch @ github
Attribution Note
This work was publicly released to GitHub on April 20th 2024, and also possibly through the Stanford CS236 project directory. Later research, including this 2025 paper from Google, explores similar methods and perform comparable experiments (without citing this work).
Overview
NoiseSearch uses inference-time search algorithms to optimize the 'noise' in diffusion-based text-to-image models for arbitrary objective optimization. Search algorithms that we tested include Stochastic Hill Climbing (SHC), Simulated Annealing (SA), and random sampling.
We show that deliberate metaheuristics like SHC and SA achieve higher scores than the naive best-of-N random sampling commonly used in image generation evaluations.
Concept
Below, we see the optimization landscape of various objective functions (ImageReward, CLIPScore, JPEG compression) with respect to 2 directions in noise space. We find that these landscapes tend to be locally smooth. This means that local optimization algorithms like HC and SA should achieve better scores than random independent sampling.
Results
Below is the average score across different objective functions as a function of the number of search steps. We see than SHC and SA outperform random sampling at almost every point.
We also see that these purely inference-time algorithms can boost the performance of weak models past stronger contemporaries.
