kd.evals.StandaloneEveryCheckpoint

class kauldron.evals.StandaloneEveryCheckpoint(**kwargs)

Bases: kauldron.evals.run_strategies.Standalone

Run eval continuously everytime a new checkpoint is found.

If the eval takes too long, intermediate checkpoints might be skipped.

Run as a separate XM job. All kxm.Job parameters are optionally supported.

Example:

kd.evals.Evaluator(
    run=kd.evals.StandaloneEveryCheckpoint(platform='a100=1'),
)

If job_group=’group_name’, all the evaluators sharing the same job_group will share the same XManager job (to save resources).