Significant difference in results with different GPU counts on WMDP bio dataset using GradDiff

[ZeguanXiao]

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task
• My own task or dataset (give details below)

Reproduction

I’ve encountered a very strange issue when running the WMDP bio dataset with the GradDiff method. Specifically, I observed that the number of GPUs used (I tested with 4 vs 8 GPUs) leads to significant differences in the results, even though the effective batch size was kept constant (32).

Has anyone else experienced this? Could there be any known reason why the GPU count would have such a strong impact on the outcomes?

Expected behavior

The results should be identical or similar.

[Dornavineeth]

Hi, sorry for the late reply. It is something we observed too. We initially thought it was a transformer issue (See #36 ). There have been a few recent issues/fixes for the multi-node training(see #139 #145). Could you please give it a try after these fixes?