Skip to content

Allow explicit input files in the all-in-one gym eval run flow #3501

Description

@rystewart-nvidia

Use cases, pain points, and background

The all-in-one gym eval run flow currently rejects --input and requires a configured dataset split. To run an already-prepared JSONL file, users have to start servers separately with gym env start, then collect rollouts with gym eval run --no-serve --input.

Supporting explicit inputs in the all-in-one flow would simplify running subsets or retrying selected tasks without adding a separate dataset configuration.

Description:

Allow gym eval run --input <prepared-tasks.jsonl> to handle server startup, rollout collection, and shutdown in one command. Any required dataset preparation still happens beforehand.

Design:

  • Update nemo_gym/rollout_collection.py to accept an explicit input path without requiring a split.
  • Update nemo_gym/cli/eval.py to validate the file before server startup and use it directly, bypassing preparation of configured dataset splits.
  • Give explicit input files precedence over configured splits, including when using a custom collection driver.
  • Preserve existing split preparation and cache reuse when no input file is supplied.
  • Update CLI help in nemo_gym/cli/main.py, the CLI reference under fern/, and the relevant unit tests.

Out of scope:

  • Automatically downloading or preparing the dataset supplied through --input.
  • Changing dataset formats, agent behavior, or verification logic.
  • Changing the existing --no-serve workflow.

Acceptance Criteria:

  • An already-prepared input file can be run without --split or --no-serve.
  • Missing or invalid file paths fail before servers start.
  • Explicit input takes precedence over a configured split.
  • Standard collection and custom collection drivers receive the supplied file.
  • Existing split preparation and cache reuse still work.
  • CLI help and documentation explain that input files must be prepared beforehand.
  • Regression tests and a live end-to-end smoke test pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions