Python API reference
⚠️ Pre-1.0 - Python API surface is unstable. LLenergyMeasure is currently pre-1.0. The Python library API documented here is not yet a stable public surface - class signatures, function names, and module paths may change between minor versions without notice.
The supported user-facing interfaces are the CLI (
llem run,llem doctor) and the YAML study config. See CLI reference and Study config for stable contracts.The library API will stabilise at v1.0.0.
Auto-generated by
scripts/generate_api_docs.pyfrom docstrings insrc/llenergymeasure/. Do not edit manually - edits are overwritten on the next build.
__version__
Current version: 0.7.0
class ExperimentConfig
v2.0 experiment configuration.
Central configuration object controlling all aspects of a single LLM inference efficiency measurement. Organised into semantic groups:
- task: What to measure (model, dataset, token limits, seed)
- measurement: How to measure (warmup, baseline, energy sampler)
- Engine sections (transformers:, vllm:, tensorrt:): How to execute
The engine section must match the engine field. Providing a transformers: section when engine=vllm is a configuration error.
Fields
| Field | Type | Default | Description |
|---|---|---|---|
task | llenergymeasure.config.models.TaskConfig | (required) | Task configuration: model, dataset, workload shape |
engine | Engine | <Engine.TRANSFORMERS: 'transformers'> | Inference engine |
serving_mode | Literal['offline', 'server'] | (required) | Serving mode discriminator (required, no default). 'offline' measures batch inference over a fixed prompt set; 'server' measures online serving from a traffic spec (requires a server: section). Both modes have a measurement execution path. A conditioning identity axis - it enters the declared, resolved, and observed config hashes, so an offline config and a server config never deduplicate together. The matching mode namespace (server:) is legal only under its own mode; a mismatch fails loudly. |
measurement | llenergymeasure.config.models.MeasurementConfig | (required) | Measurement methodology: baseline, energy sampling (mode-invariant) |
sampling_preset | Optional[Literal['deterministic', 'standard', 'creative', 'factual']] | None | Sampling preset. When set, preset values are merged into the active engine's sampling section at parse time; explicit YAML values take precedence over preset values. |
transformers | `llenergymeasure.config.llem_execution.TransformersSection | None` | None |
vllm | `llenergymeasure.config.generated.vllm.Config | None` | None |
tensorrt | `llenergymeasure.config.generated.tensorrt.Config | None` | None |
server | `llenergymeasure.config.models.ServerSection | None` | None |
offline | `llenergymeasure.config.models.OfflineSection | None` | None |
passthrough_kwargs | `dict[str, Any] | None` | None |
class ExperimentResult
Experiment result - the user-visible output of a measurement run.
Produced once per single-process measurement run by the harness. Holds the final metrics (energy, throughput, FLOPs, latency) directly; there is no per-process breakdown.
Fields
| Field | Type | Default | Description |
|---|---|---|---|
bundle_version | str | '2.0' | Results-bundle version (layout + artefact set + per-artefact schema, as one contract). Replaces the retired per-artefact result schema_version. |
experiment_id | str | (required) | Unique experiment identifier |
declared_config_hash | str | (required) | SHA-256[:16] of the whole declared ExperimentConfig (compute_declared_config_hash). Environment fields are not part of ExperimentConfig, so they are naturally excluded. Same term as the declared_config block in the config.json sidecar. |
llenergymeasure_version | `str | None` | None |
serving_mode | str | 'offline' | Serving mode that produced this result: the offline/server discriminator, mirroring the config-side ExperimentConfig.serving_mode. "offline" for batch measurement, "server" for online serving measurement. A plain string, not a closed vocabulary, so the mode set can grow without a schema break. Stamped per result from its measurement source (the offline assembler, see harness.result_assembly.SourceMetrics; the server per-window mapper for server mode). |
engine | str | 'transformers' | Inference engine used. Convenience copy; authoritative home is the config.json sidecar. |
model_name | str | 'unknown' | Model name/path used. Convenience copy; authoritative home is the config.json sidecar. |
input_tokens | `int | None` | (required) |
output_tokens | int | (required) | Actual output (decode) tokens as observed by the engine. total_tokens = input_tokens + output_tokens. In server mode this is the client-side canonical count (span-received streamed deltas), and is always real. |
total_tokens | `int | None` | (required) |
total_energy_j | float | (required) | Total energy (sum across processes) |
total_inference_time_sec | float | (required) | Total inference time |
avg_tokens_per_second | float | (required) | Average throughput |
avg_energy_per_token_j | float | (required) | Average energy per token |
energy_per_token_mj_adjusted | `float | None` | None |
energy_per_token_mj_total | `float | None` | None |
total_flops | float | (required) | Total FLOPs (reference metadata) |
flops_per_output_token | `float | None` | None |
flops_per_input_token | `float | None` | None |
flops_per_second | `float | None` | None |
energy_adjusted_j | `float | None` | None |
energy_per_device_j | `list[float] | None` | None |
energy_breakdown | `llenergymeasure.domain.metrics.EnergyBreakdown | None` | None |
multi_gpu | `llenergymeasure.domain.metrics.MultiGPUMetrics | None` | None |
measurement_warnings | list[str] | (required) | Measurement quality warnings (e.g., short duration, thermal drift) |
warmup_excluded_samples | `int | None` | None |
model_load_time_sec | `float | None` | None |
engine_build_cache_hit | `bool | None` | None |
reproducibility_notes | str | 'Energy measured via NVML polling. Accuracy +/-5%. Results may vary with thermal state and system load.' | Fixed disclaimer about measurement accuracy |
timeseries | `str | None` | None |
runner_provenance | `llenergymeasure.domain.provenance.RunnerProvenance | None` | None |
session | `llenergymeasure.domain.session.SessionBlock | None` | None |
server | `llenergymeasure.domain.experiment.ServerWindowProvenance | None` | None |
server_metrics | `llenergymeasure.domain.experiment.ServerWindowMetrics | None` | None |
environment | `llenergymeasure.domain.environment.EnvironmentSnapshot | None` | None |
start_time | datetime.datetime | (required) | Earliest process start time |
end_time | datetime.datetime | (required) | Latest process end time |
aggregation | `llenergymeasure.domain.experiment.AggregationMetadata | None` | None |
throttle | `llenergymeasure.domain.metrics.ThrottleInfo | None` | None |
warmup_result | `llenergymeasure.domain.metrics.WarmupResult | None` | None |
latency_stats | `llenergymeasure.domain.metrics.LatencyStatistics | None` | None |
extended_metrics | `llenergymeasure.domain.metrics.ExtendedEfficiencyMetrics | None` | None |
class StudyConfig
Thin resolved container for a study (list of experiments + execution config).
Populated by :func:llenergymeasure.study.loading.resolve_study, the single
entry point every study passes through. The experiments list contains
fully-validated ExperimentConfig objects ready for execution.
skipped_configs records any grid points that failed Pydantic validation so
they can be displayed to the researcher in pre-flight output.
Constructing one directly is supported (for programmatic and pipeline use):
run_study and run_experiment resolve it before running it. The
resolution outputs - study_design_hash, dedup_mode,
pre_run_equivalence_groups, declared_resolved_config_hashes - are
resolution's to write, not a caller's.
Fields
| Field | Type | Default | Description |
|---|---|---|---|
experiments | list[llenergymeasure.config.models.ExperimentConfig] | (required) | Resolved list of experiments to run |
study_name | `str | None` | None |
output | llenergymeasure.config.models.OutputConfig | (required) | Study-level output configuration (results_dir, save_timeseries) |
study_execution | llenergymeasure.config.models.ExecutionConfig | (required) | Cycle repetition and ordering controls |
runners | `dict[str, str] | None` | None |
images | `dict[str, str] | None` | None |
study_design_hash | `str | None` | None |
provenance_logs | dict[str, dict[str, Any]] | (required) | Per-experiment provenance for the config.json sidecars, keyed by declared-config hash. Each entry maps dotted field paths to {'effective', 'source', 'default'}, with sources from the merges that resolved the experiment ('call_site' / 'sweep' / 'yaml'). |
settings_provenance | dict[str, str] | (required) | Which precedence layer supplied each study-wide setting, keyed by study-file path ('output.results_dir', 'study_execution.n_cycles', 'runners.vllm', 'images.vllm'). Values are the source vocabulary 'call_site' / 'env' / 'yaml' / 'user_config' / 'default'. Emitted by the resolution merge itself, so it records which layer actually won. |
skipped_configs | list[dict[str, Any]] | (required) | Grid points that failed Pydantic validation during expansion. Persisted for post-hoc review and pre-flight display. |
dedup_mode | Literal['resolved', 'off'] | 'resolved' | Effective-config resolution dedup mode. 'resolved' applies dormant-rule effective-config resolution at expansion and collapses resolved-config-hash-equivalent configs to a single run. 'off' runs every declared config regardless of equivalence. Set via ExecutionConfig.deduplicate_equivalent / --no-dedup. |
pre_run_equivalence_groups | list[dict[str, Any]] | (required) | Pre-run equivalence groups computed at sweep-expansion time. Each group records the resolved_config_hash, canonical excerpt, and member declared-indices. Written to 'equivalence_groups.json' alongside the results bundle. |
declared_resolved_config_hashes | list[str] | (required) | Per-declared-config resolved_config_hashes (parallel to the pre-resolved sweep input). Harness consults this to tag each experiment with its equivalence group at sidecar-write time. |
dormant_observations | list[dict[str, Any]] | (required) | Distinct auto-normalised settings applied during effective-config resolution (keys: engine, rule_id, field_path, normalisation). These are fields the engine silently rewrites in the executed config, so they are surfaced in 'llem study plan' and preflight output. Empty when nothing was normalised. |
class StudyResult
Final return value of a study run.
Distinct from StudyManifest (the in-progress checkpoint). StudyResult is assembled once after all experiments complete (or after interrupt) and returned to the caller.
Fields
| Field | Type | Default | Description |
|---|---|---|---|
experiments | list[llenergymeasure.domain.experiment.ExperimentResult] | (required) | Results for each experiment in the study |
study_name | `str | None` | None |
study_design_hash | `str | None` | None |
measurement_protocol | dict[str, Any] | (required) | Flat dict from ExecutionConfig: n_cycles, experiment_order, experiment_gap_seconds, cycle_gap_seconds, shuffle_seed, experiment_timeout_seconds |
result_files | list[str] | (required) | Paths to per-experiment result.json files (paths, not embedded) |
summary | llenergymeasure.domain.experiment.StudySummary | (required) | Computed aggregate statistics (counts, totals, warnings) |
skipped_experiments | list[dict[str, Any]] | (required) | Grid points skipped due to validation errors (raw_config + reason + errors) |
run_experiment
run_experiment(config: 'str | Path | ExperimentConfig | None' = None, *, model: 'str | None' = None, engine: 'str | None' = None, n_prompts: 'int' = 100, dataset: 'str' = 'aienergyscore', skip_preflight: 'bool' = False, progress: 'ProgressCallback | None' = None, output_dir: 'str | Path | None' = None, **kwargs: 'Any') -> 'ExperimentResult'
Run a single LLM inference efficiency experiment.
Three call forms: run_experiment("config.yaml") # YAML path run_experiment(ExperimentConfig(...)) # config object run_experiment(model="gpt2", engine="Y") # kwargs convenience
Args:
config: YAML file path, ExperimentConfig object, or None (use kwargs).
model: Model name/path (kwargs form only).
engine: Inference engine (kwargs form only, defaults to ExperimentConfig default).
n_prompts: Number of prompts (kwargs form only, default 100).
dataset: Dataset source name (kwargs form only, default "aienergyscore").
skip_preflight: Skip Docker pre-flight checks (GPU visibility, CUDA/driver compat).
progress: Optional callback for step-by-step progress reporting.
output_dir: Base directory for results. When provided, overrides the
default ./results directory. A timestamped study subdirectory
is created within this path.
**kwargs: Additional ExperimentConfig fields (kwargs form only).
Returns: ExperimentResult: Experiment measurements and metadata.
Raises: ConfigError: Invalid config path, missing model in kwargs form. pydantic.ValidationError: Invalid field values (passes through unchanged).
run_study
run_study(config: 'str | Path | StudyConfig', *, skip_preflight: 'bool' = False, progress: 'ProgressCallback | None' = None, resume_dir: 'Path | None' = None, resume: 'bool' = False, output_dir: 'Path | None' = None, skip_set: 'set[tuple[str, int]] | None' = None, no_lock: 'bool' = False, config_path: 'Path | None' = None, cli_overrides: 'dict[str, Any] | None' = None, preresolved: 'tuple[dict[str, RunnerSpec], dict[str, dict[str, str]]] | None' = None) -> 'StudyResult'
Run a multi-experiment study.
Always writes manifest.json to disk (documented side-effect).
A StudyConfig built in memory is resolved here exactly as a YAML study is:
equivalent configs are deduplicated, the study_design_hash is computed,
n_cycles is expanded into the execution sequence, and the equivalence
groups are recorded. An already-resolved StudyConfig (from load_study)
passes through unchanged.
Resolving a caller-built StudyConfig is not free of side effects on its argument: for a server-mode experiment, the resolved warmup protocol is attached to the caller's OWN ExperimentConfig objects (as side-channel state, not a declared field), so those objects carry it after the call. The declared fields are never rewritten - the configs that actually run are copies.
Args:
config: YAML file path or a StudyConfig.
skip_preflight: Skip Docker pre-flight checks (GPU visibility, CUDA/driver compat).
CLI --skip-preflight flag and YAML execution.skip_preflight: true also bypass.
progress: Optional StudyProgressCallback for live per-experiment display.
When provided, the study runner emits begin/end experiment events and
forwards per-step progress from worker subprocesses.
resume_dir: Explicit study directory to resume. Overrides resume.
resume: When True and resume_dir is None, auto-detect the most recent
resumable study in output_dir (default results/).
output_dir: Dual role by run mode. For a fresh run it is the results-dir
override (precedence: output_dir > YAML output.results_dir >
user config > ./results). Applies to an already-resolved
StudyConfig input too - redirecting where results land never touches
the resolved identity. For an auto-detect resume it is the base
directory searched for the most recent resumable study. Ignored when
resume_dir is given explicitly.
skip_set: Set of (config_hash, cycle) pairs to skip (already completed in a
previous run). Populated automatically when resuming; callers rarely
need to set this directly.
no_lock: Skip GPU advisory lock acquisition. Use with --no-lock CLI flag.
config_path: Original YAML config file path for copying to study artefacts.
When config is a StudyConfig object, callers should pass the original
path separately so the YAML is preserved for reproducibility.
cli_overrides: Overrides applied on top of the study file when config
is a path (forwarded to :func:load_study as its call-site layer, so
they win over what the file declares and are recorded as call_site
in the provenance). Study-file-shaped and nested only, e.g.
{"task": {"model": "gpt2"}} or {"study_execution": {"n_cycles": 5}} - flat or dotted keys ({"model": ...},
{"task.model": ...}) are not study-file keys and fail loudly at
load. An n_cycles override really multiplies the dispatched
experiment list, like any resolved cycle count. Ignored for a
StudyConfig input, which the caller has already built.
preresolved: Optional (runner_specs, system_overrides) already
computed by a prior run_study_preflight call (e.g. the CLI runs
preflight to render the panel). When supplied, the orchestrator reuses
it instead of re-running preflight. Must be paired with
skip_preflight=True so the precomputed result is trusted.
Returns: StudyResult with experiments, result_files, measurement_protocol, and inline summary fields.
Raises: ConfigError: Invalid config path or parse error. PreFlightError: Multi-engine study without Docker. StudyError: No resumable study found (when resume=True). StudyError: Config drift detected (study_design_hash changed). pydantic.ValidationError: Invalid field values (passes through unchanged).