Skip to content

fix(dpo): honor objective configuration in training and validation - #508

Open
JiahaoTanXX wants to merge 1 commit into
alibaba:mainfrom
JiahaoTanXX:fix/dpo-objective-config
Open

JiahaoTanXX wants to merge 1 commit into
alibaba:mainfrom
JiahaoTanXX:fix/dpo-objective-config

Conversation

@JiahaoTanXX

Copy link
Copy Markdown

DPOConfig.ipo, beta, and label_smoothing were omitted from the batches created by both DPOPipeline.run() and val(). The worker and validation loss therefore silently used the defaults (False, 0.1, 0.0), even when the user selected another objective or hyperparameters. For example, a fixed preference pair configured with beta=0.7 produced loss 0.644397 instead of 0.403186.

Pass all three settings through batch metadata in training and validation. Add five CPU regression cases covering default DPO, non-default beta, label smoothing, IPO, and IPO with non-default beta. The tests execute the real driver run()/val(), DataProto operations, and worker loss with local cluster/model stand-ins. They check training and validation loss, reward metrics, and the training gradient against independently computed scalar expectations.

Validation:

  • Before the fix: the default case passed and all four non-default cases failed.
  • python -m pytest -q tests/pipeline/dpo/test_objective_config.py tests/utils/test_functionals.py --torch_ver 2.8.0: 9 passed.
  • pre-commit run --files roll/pipeline/dpo/dpo_pipeline.py tests/pipeline/dpo/test_objective_config.py: all four hooks passed.
  • git diff --check: passed.

Tested on macOS CPU with Python 3.12.14, PyTorch 2.8.0, Transformers 4.57.0, and Ray 2.48.0. No model weights, Ray cluster, or full GPU/distributed training run was used.

@CLAassistant

CLAassistant commented Oct 16, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants