GuidanceNFT: NFT-Based Reward Fine-Tuning of Guided Diffusion Models

Jiayin Liu, Zeyu Michael Li, Bingshuo Qian, Xiang Cheng

Under reviewDuke University, with Prof. Xiang Cheng

Overview

Guidance strengthens prompt alignment and reward fine-tuning with DiffusionNFT (NFT) raises a task reward, but combining them is not straightforward. An OCR-specialized NFT model can render the requested words while omitting the surrounding scene; applying classifier-free guidance (CFG) to the fine-tuned velocity field reduces the optimized reward for all three SD3.5-M training objectives; and applying standard NFT directly to guided samples causes reward degradation or training collapse. Why does NFT fail on guided samples, and how can it be fixed?

Toy example on a tree-shaped data distribution with five panels: pretrained samples (hit 0.25, on-tree 0.85), pretrained samples with CFG (hit 0.35, on-tree 0.95), samples after NFT (hit 0.63, on-tree 0.81), after NFT with CFG added (hit 0.85, on-tree 0.71), and after updating the guided field (hit 0.97, on-tree 0.91) — Updating the guided field improves reward while retaining data structure. (a, b) Pretrained samples without and with CFG. (c) NFT moves samples toward the rewarded branches, but (d) adding CFG afterward pushes many samples off the tree. (e) Updating the guided field better combines reward targeting with tree fidelity. Panels (c) to (e) zoom into the dashed region; blue branches mark the reward target and orange points mark off-tree samples.

Key idea

NFT improves a model by contrast. It re-noises generated samples and compares the velocity field of the higher-reward samples, \(v^+\), with that of the lower-reward samples, \(v^-\). Their difference \(\Delta \propto v^+ - v^-\) is the update NFT wants to learn. Fitting \(v^+\) and \(v^-\) separately would be costly, so NFT measures both against a shared reference velocity \(v_{\mathrm{ref}}\), which lets it train directly on rewarded samples.

The NFT objective with a reference is

\[\alpha\,\big\| (v_\theta - v^{\mathrm{old}}) - (v^+ - v_{\mathrm{ref}}) \big\|^2 + (1-\alpha)\,\big\| (v_\theta - v^{\mathrm{old}}) - (v_{\mathrm{ref}} - v^-) \big\|^2,\]

where \(\alpha \in [0, 1]\) is the average reward weight at that state. It recovers \(\Delta\) only if the reference is the mean velocity \(\bar v\) of the re-noised samples. Under guidance, the model's own prediction no longer equals this mean, so the reference has a gap \(e = v_{\mathrm{ref}} - \bar v\), and the objective becomes

\[\left\| (v_\theta - v^{\mathrm{old}}) - \Delta - (1 - 2\alpha)\,e \right\|^2 + \mathrm{const}.\]

The model learns \(\Delta + (1 - 2\alpha)\,e\) instead of \(\Delta\). The extra term can shrink, rotate or reverse the update, and leads to the collapse shown below.

Five panels on the toy tree: samples with a matched reference cover the rewarded branches (hit 0.98, 73 cells); the intended update field; samples with the conditional prediction as reference concentrate on a small part (hit 0.97, 8 cells); the reference error field; and samples moved by the error term alone leave the rewarded branches (hit 0.00) — Reference error distorts the reward-driven update. (a) A matched reference spreads samples across the rewarded branches through the intended update in (b). (c) An incorrect reference, here the conditional prediction, concentrates samples on fewer branches. (d) The reference error adds a displacement that (e), applied alone, moves samples away from the reward target. In this toy model the error term's root-mean-square magnitude in the first round is 2.4 times that of Δ. Hit is the fraction reaching the reward target; cells measures spatial diversity.

GuidanceNFT removes the gap by learning the reference online, with flow matching on the re-noised guided samples. It works with explicit CFG and with guidance built into the model, and adds no extra inference passes.

Results

We first test whether reference correction sustains reward improvement under explicit CFG. The comparisons below support estimating the reference from guided rollouts: restoring the pretrained empty-condition branch at inference or using the analytic CFG field as the reference is not enough.

Three plots: on SiT at evaluation CFG 1.5, NFT loses PickScore after about 200 updates, with or without the pretrained empty branch, while GuidanceNFT keeps improving; on SD3.5 trained at CFG 4, GuidanceNFT's online GenEval reward rises while NFT and the analytic correction fall; the reference gap grows for the collapsing runs and shrinks for GuidanceNFT — **Online reference estimation sustains reward improvement under CFG.** (a) On SiT at evaluation CFG 1.5, NFT's reward declines after an initial gain; restoring the pretrained empty-condition branch yields only a modest recovery. (b) On SD3.5-M with CFG-4 rollouts, standard NFT collapses, and using the analytic CFG field as the reference accelerates the decline. GuidanceNFT continues to improve. (c) Disagreement with independently fitted rollout references grows for both alternatives but remains small for GuidanceNFT, consistent with the reference-error mechanism. Reference gaps are measured at noise level 0.5; horizontal axes show training updates.

On SD3.5-M, GuidanceNFT achieves higher optimized rewards than NFT with and without CFG for all three training objectives: GenEval, OCR, and Learned3, which combines learned preference and alignment scores. Adding CFG after standard NFT reduces the optimized reward for each objective. The table below reports GenEval and OCR fine-tuning, where GuidanceNFT also leads both NFT variants on all seven held-out evaluation metrics.

Method	Target reward	Held-out evaluation
Method	Target reward	GenEval	OCR	PickScore	CLIPScore	HPSv2.1	Aesthetic	ImageReward
GenEval objective
NFT, no CFG	0.900	0.855	0.434	21.70	0.265	0.234	4.886	0.271
NFT, CFG 4	0.657	0.565	0.624	21.19	0.271	0.216	4.773	0.148
GuidanceNFT	0.914	0.877	0.674	21.83	0.277	0.250	4.947	0.682
OCR objective
NFT, no CFG	0.971	0.035	0.971	19.61	0.208	0.137	4.208	−1.679
NFT, CFG 4	0.192	0.265	0.235	20.21	0.242	0.188	4.596	−0.919
GuidanceNFT	0.975	0.488	0.974	21.83	0.274	0.256	5.120	0.425

GuidanceNFT combines task specialization with stronger performance beyond the optimized reward. For OCR fine-tuning, it achieves an on-domain reward of 0.975 versus 0.971 for NFT without CFG, while raising held-out GenEval from 0.035 to 0.488. Target reward reports the fine-tuning objective evaluated on its test set. All runs use SD3.5-M; CFG scale is 4 where used. Shaded cells mark the metric optimized during training; bold marks the best score within each objective. Rows average two evaluation seeds.

The selected images below illustrate how task specialization can retain broader prompt content, including object relationships under Learned3 and scene context under OCR fine-tuning.

Eight prompts by three methods on SD3.5-M. NFT without CFG gives plain or blurry images, NFT with CFG gives distorted images with illegible text, GuidanceNFT gives images that match the prompt with legible text — **GuidanceNFT retains prompt content during reward specialization.** After Learned3 fine-tuning (first four columns), selected examples preserve requested objects and spatial relations, including a portrait within a triangular frame and an umbrella resting on a spoon. After OCR fine-tuning (last four), GuidanceNFT places readable text within the requested scenes, while the NFT examples omit scene content or distort the text when CFG is added. Rows show NFT at CFG 1, NFT at CFG 4, and GuidanceNFT at CFG 4; initial noise is paired within each column.

Guidance built into the model

Model Guidance and ELF incorporate guidance into their native predictions. We test whether fitting the reference to each model's own rollouts also stabilizes reward fine-tuning in these settings, spanning ImageNet generation, OpenWebText language modeling, and GSM8K reasoning.

Three training curves comparing NFT and GuidanceNFT: on Model Guidance NFT's PickScore peaks near 200 updates and falls while GuidanceNFT keeps rising; on ELF-B NFT's entropy collapses while GuidanceNFT's stays high; on GSM8K NFT's accuracy drops sharply after 250 updates while GuidanceNFT's stays level — **Reference correction also stabilizes fine-tuning with built-in guidance.** GuidanceNFT sustains reward gains on Model Guidance for ImageNet generation (a), maintains text entropy on ELF-B for OpenWebText (b), and retains reasoning accuracy on the Qwen-based ELF model for GSM8K (c). Standard NFT degrades in all three settings. The language experiments use self-conditioning CFG scales of 3 and 2, respectively. Horizontal axes show training updates. The GSM8K curve uses 32-step synchronous evaluation; the table below uses 64-step asynchronous evaluation.

In a separate ten-step evaluation, GuidanceNFT raises PickScore from 20.16 to 21.58 on SiT (NFT at CFG 1 versus GuidanceNFT at CFG 1.5), and from 20.01 to 21.58 on Model Guidance. Both comparisons use NFT's best retained checkpoint. The images below illustrate differences in visual detail using longer sampling runs.

ImageNet samples for four classes each from NFT and GuidanceNFT, on SiT and on Model Guidance; the GuidanceNFT samples are sharper and more recognizable — **GuidanceNFT produces more detailed class-conditional images under explicit and built-in guidance.** Selected examples show clearer facial features and fur on SiT (a), and more distinct object structure, including the snowmobile body and bighorn horns, on Model Guidance (b). NFT appears above GuidanceNFT, with initial noise paired within each column. SiT uses CFG 1 for NFT and CFG 1.5 for GuidanceNFT; Model Guidance uses external CFG 1 for both. Images use 250 and 1,000 sampling steps, respectively; the PickScores reported in the text come from a separate ten-step evaluation.

On GSM8K with the Qwen-based ELF model, GuidanceNFT raises mean Pass@1 from 41.68% to 43.03% and Pass@8 from 67.02% to 68.61% relative to NFT's best retained checkpoint. The table also compares the released CEDR checkpoint, which adapts NFT to self-conditioned models through velocity reconstruction.

	Pass@1	Pass@2	Pass@4	Pass@8
Pretrained	39.17	49.82	59.20	67.32
NFT	41.68	51.53	59.78	67.02
CEDR	42.16	51.36	59.40	66.26
GuidanceNFT	43.03	52.66	61.24	68.61

GuidanceNFT improves reasoning performance beyond NFT's best checkpoint before collapse. It achieves the highest mean Pass@\(k\) among the compared checkpoints for all reported \(k\), including the released CEDR checkpoint. Pass@\(k\) is the percentage of problems with at least one correct answer in \(k\) attempts. Evaluation uses 64 steps with asynchronous time sampling and self-conditioning CFG scale 2. Bootstrap standard deviations over problems are 1.0 to 1.3 percentage points for every entry.

On OpenWebText, low perplexity alone can conceal repetitive output. We therefore evaluate perplexity alongside token entropy and compare GuidanceNFT with the pretrained model and our reimplementation of CEDR's velocity reconstruction, denoted CEDR-style reconstruction.

Left: an NFT text sample that repeats 'health care for children' and a GuidanceNFT sample of varied text. Right: perplexity against entropy for the pretrained model, CEDR-style reconstruction and GuidanceNFT; the GuidanceNFT curve has lower perplexity at the same entropy — **GuidanceNFT improves the perplexity–entropy tradeoff while avoiding repetitive collapse.** Left: standard NFT repeats a short phrase despite its low perplexity, whereas GuidanceNFT retains more varied output; both train and sample at self-conditioning CFG scale 3. Right: across evaluation guidance scales, GuidanceNFT achieves lower perplexity than the pretrained model at matched entropy and higher entropy than CEDR-style reconstruction at comparable perplexity. GuidanceNFT and reconstruction train at guidance scales 3 and 1, respectively. Markers vary the evaluation guidance scale \(w\); the dotted line marks entropy 5.15 nats as a reference for comparing perplexity at matched diversity.

Scope

The analysis concerns the consistency condition that this NFT formulation needs from its reference velocity. It explains one source of instability under guidance; it does not show that every other construction must fail.