kd.evals.EvaluatorBase

class kauldron.evals.EvaluatorBase(**kwargs)

Bases: kauldron.utils.config_util.BaseConfig, kauldron.utils.config_util.UpdateFromRootCfg

Base class for inline evaluators.

Evaluators should inherit from this class and implement the evaluate method.

name

Eval name (collection name for TensorBoard and Datatables)

Type:

str

run

How/when to run this eval (e.g. kd.evals.EveryNSteps(100) or kd.evals.StandaloneEveryCheckpoint())

Type:

kauldron.evals.run_strategies.RunStrategy

writer

Metric writer (set automatically)

Type:

kauldron.train.metric_writer.WriterBase

base_cfg

reference to the experiment configuration (set automatically).

Type:

kauldron.train.trainer_lib.Trainer

discard_opt

Whether to discard the optimizer state for the evaluator. This is useful to save memory in case the evaluator does not need access to the optimizer state.

Type:

bool

name: str = 'eval'

run: run_strategies.RunStrategy

writer: metric_writer.WriterBase = _FakeRootCfg('cfg.writer')

base_cfg: trainer_lib.Trainer = _FakeRootCfg('cfg')

discard_opt: bool = False

maybe_eval(

*,

step: int,

state: kauldron.train.train_step.TrainState,

) → Any

Run or skip the evaluator for the given train-step.

evaluate(

state: kauldron.train.train_step.TrainState,

step: int,

) → Any

Run this evaluator then write and optionally return the results.