Driver File v2¶

The external interface implementation provides a series of individual processing steps that can be selected and chained by means of the JSON driver file. Within this file, each processing step is represented as a JSON object with the following properties:

kind (required, String): This is a unique identifier for every different kind of steps and therefore determines what this step will do when exectuted.
id (String): Unique identifier for this particular step. This can be used to refer to (the output of) other steps.
options (Object): Set of options to tune the behavior of the step as needed. Some steps don’t have options, whereas others require you to specify them. For most steps, options are optional. The properties of the options object depend on the step’s kind - see Available Processing Step Kinds.
inherit_options_from (String): The id of another step of the same kind (anywhere in the steps array) whose options shall be used as the base for this step’s options. The local options (if any) are applied on top as a JSON merge patch: nested objects are merged recursively, all other values replace the inherited ones and a value of null removes the inherited option. The referenced step may itself inherit its options from yet another step.
outputs (Object): This can be used to give human-readable names to individual outputs of the current step in form of name-output pairs. See Referencing Outputs - the step ID in this case is implicitly the current step’s ID and must not be included explicitly. These names are automatically propagated through the processing chain (for most kinds of steps).
inputs (String or Array of Strings): Specifies the inputs of this step, which must be outputs of other steps. See Referencing Outputs. Every step requires at least one input, except for steps that produce data from scratch (such as read_input); omitting inputs on a step that needs it is an error.

The different steps are listed objects in the top-level steps array. Steps are processed in order according to their order in this array.

Referencing Outputs¶

Outputs can be referenced by means of their IDs. An ID has the general format <step_id>.<output_id>. <step_id> must refer to the id of one of the steps that have been executed before. <output_id> can be the human-readable names specified via a step’s outputs property or an integer. In the latter case, the integer refers to the index of the output. Outputs are indexed starting from zero in the order they are produced.

<output_id> can also be an expression enclosed in square brackets in order to refer to multiple outputs at once. This can be a comma-delimited (no spaces!) list of output names or indices, or a range of indices which are of the form <from>-<to> like 0-5.

Examples of output IDs:

my_step.0
my_step.some_name
my_step.[0-2]
my_step.[0-2,some_name,5]

Finally, to refer to all outputs of a given step, just use the step’s ID without anything appended to it. That is, my_step would automatically refer to all outputs of the respective step.

How the Pipeline Executes¶

Steps are executed strictly in the order in which they appear in the steps array - there is no separate dependency resolution based on inputs. Given IDs must be unique across the whole driver file.

Two kinds of data can flow between steps: Expression data (one or more named result expressions) and ExportTree data (the export-ready tree form produced by the to_export_tree step). Every step kind’s documentation below states which of these it consumes and which it produces. Feeding a step data of the wrong kind (e.g. passing plain expressions into a step that expects an ExportTree) is an error.

Not every step produces output data - they exist purely for their side effects (e.g. aborting on invalid input, printing to the console or writing something to a file) - and therefore cannot be referenced by another step’s inputs.

Errors encountered anywhere in the pipeline (be it while parsing a step’s options or while executing it) abort the entire run; there is no partial/best-effort processing.

Templates¶

A template is a reusable, parameterized sequence of steps. It is defined once under the top-level templates object and then stamped out wherever it is needed, so a pipeline tail that would otherwise be written out several times (differing only in a few values) is written once. Templates are expanded into ordinary steps before the pipeline runs, so everything described above - inherit_options_from, output aliasing, input resolution - applies unchanged to the result.

Templates can be shared between drivers by placing them in a separate file and pulling it in with $include (see Including Other JSON Files).

Each entry of templates maps a template name to a definition with two properties:

parameters (Object): Declares the knobs the template exposes. Each entry maps a parameter name to a declaration object that must specify either required: true (the value must be supplied on every use) or a default (any JSON value, used when the parameter is not supplied). A declaration that specifies neither is an error.
steps (Array): The steps the template expands to, written just like top-level steps but allowed to contain ${parameter} placeholders (see below). May itself contain template uses.

A template is used by writing, in place of a regular step, an object with a template property (naming the template) instead of a kind. Such a step accepts only:

template (required, String): Name of the template to expand here.
id (required, String): Prefix under which this use’s steps are placed (see namespacing below).
parameters (Object): Binds the template’s parameters. Every key must be a declared parameter and every required parameter must be given; omitted optional parameters fall back to their default.

Placeholder substitution¶

Within a template’s steps, ${name} placeholders are recognized in string values at any nesting depth (object keys are never substituted). There are two forms:

A string that is exactly "${name}" is replaced by the parameter’s full JSON value, with its type preserved - so a boolean stays a boolean, a number stays a number, and an array or object is spliced in as-is.
A placeholder embedded in a longer string (e.g. "file_${name}.itfaa") is substituted textually; the parameter must be a scalar (a boolean renders as true/false).
A whole-value "${keys(name)}" or "${values(name)}" is replaced by an array holding the keys or the values, respectively, of the object parameter name. The i-th key and the i-th value belong together; otherwise their order is unspecified. These forms cannot be embedded in a longer string.

If a whole-value placeholder resolves to null, its enclosing object member is dropped (and an options object left empty this way is removed entirely). This lets a single template serve both the “option present” and “option absent” cases: give the parameter a default of null and pass a value only when the option is wanted. To write a literal ${, double the dollar sign: $${.

Namespacing¶

When a template is used with id P, every step inside it that has an id is renamed to P_<id>, and references to those steps (in inputs and inherit_options_from) are updated to match. References that do not point at one of the template’s own steps - in particular, values that came in through parameters, or an inherit_options_from pointing at a top-level step - are left untouched. The expanded steps are therefore addressed from the outside as P_<id> (and their outputs as P_<id>.<output>), and two uses of the same template never collide because their id prefixes differ. The prefix is joined with an underscore rather than a dot (and a use’s id may not contain a dot) because a step ID must stay a single dot-free token for output names to propagate correctly.

Note

Only inputs and inherit_options_from are namespaced. If a template’s step refers to one of the template’s own steps from within an option value (for example the deps of a dependencies step or the grouping of an export step), that reference is not rewritten - keep such internal wiring in inputs, or pass the referenced name in as a parameter.

For example, the optimize + to_export_tree sequence below is factored into one template and used twice - once plainly and once with replay enabled - without repeating the sequence or its inputs:

{
  "templates": {
    "optimized_tree": {
      "parameters": {
        "source": { "required": true },
        "replay": { "default": null }
      },
      "steps": [
        {
          "id": "opt",
          "kind": "optimize",
          "inputs": "${source}",
          "options": { "replay": "${replay}" }
        },
        {
          "id": "tree",
          "kind": "to_export_tree",
          "inputs": "opt"
        }
      ]
    }
  },
  "steps": [
    {
      "template": "optimized_tree",
      "id": "base",
      "parameters": { "source": ["traced.en", "biorth"] }
    },
    {
      "template": "optimized_tree",
      "id": "replay",
      "parameters": {
        "source": ["traced.en", "biorth"],
        "replay": { "volatile": "T\\d", "weight": 10 }
      }
    }
  ]
}

The base use produces steps base_opt and base_tree (its replay parameter defaults to null, so the replay option vanishes and the now-empty options is dropped); the replay use likewise produces replay_opt and replay_tree, with the replay option present. Subsequent steps refer to the resulting trees as base_tree and replay_tree.

Conditional steps and optional references¶

A step inside a template (including a nested template use) may carry a when property. The step is only part of the expansion if its value is truthy. Usually when is a whole-value placeholder such as "${replay_input}"; any other value is taken literally. Everything except null, false and empty strings, arrays and objects is truthy. This is decided while expanding the template, so a skipped step never exists at run time.

Steps that process the output of a skipped step would then refer to something that isn’t there. A reference (an entry of inputs, the value of inherit_options_from, or an ID inside the grouping, merge_inputs or relative_order options) can be marked as optional by a trailing ?. After all templates have been expanded:

An optional reference to an existing step (the part before the first .) loses its ? and is otherwise unchanged. Whether the output the reference names exists is not checked.
An optional reference to a step that doesn’t exist is removed from the array or object it is part of. If nothing is left of inputs, the whole step is removed, which in turn makes optional references to it unresolvable. If nothing is left of inherit_options_from, grouping, merge_inputs or relative_order, that property is omitted.
References without a ? are never touched, so a misspelled ID is still an error.

Because output aliases only propagate along steps that exist, there is no implicit pass-through for a skipped step: either a downstream reference is optional, or the step is skipped for good.

Together with null defaults, this lets one template cover variants that differ in additional steps. In the following, replay_input selects inputs that are optimized with replay and exported with the remaining ones, if there are any:

{
  "templates": {
    "variant": {
      "parameters": {
        "source": { "required": true },
        "replay_input": { "default": [] },
        "replay": { "default": null }
      },
      "steps": [
        { "id": "opt", "kind": "optimize", "inputs": "${source}" },
        { "id": "opt_replay", "kind": "optimize", "inputs": "${replay_input}",
          "options": { "replay": "${replay}" }, "when": "${replay_input}" },
        { "id": "tree", "kind": "to_export_tree", "inputs": "opt" },
        { "id": "tree_replay", "kind": "to_export_tree", "inputs": "opt_replay", "when": "${replay_input}" },
        { "id": "export", "kind": "export", "inputs": ["tree", "tree_replay?"],
          "options": { "language": "itf", "output": "out.itfaa" } }
      ]
    }
  }
}

Without replay_input only opt, tree and export remain, and export has the single input tree.

As IDs inside option values are not namespaced (see the note above), optional references there are meant to be passed in through parameters, e.g. a grouping parameter containing "Init": "my_variant_opt_replay.persistent?", where my_variant is the id of the use.

Available Processing Step Kinds¶

batch_indices¶

Configures the computation of results to happen in batches over certain result indices. For every result, its indices are ranked by the size of the index space they belong to. The smallest min_unbatched indices are always left un-batched; among the remaining indices, up to max_batched become batch indices, chosen from the end of the size-ranked list selected by selection_strategy. This step only annotates which indices shall be batched - whether batching is actually realized (and how) is up to whichever code-generation backend is used by the subsequent export step, and not every backend supports it.

Input	Output
ExportTree	ExportTree

Options¶
Option	Description	Default	Required
`min_unbatched`	Minimum number of (smallest-space) indices of a given result that are always left un-batched.	2	No
`max_batched`	Upper limit on the number of indices that may be selected for batching.	unbounded	No
`selection_strategy`	Which end of the size-ranked index list batching candidates are taken from. `largest` selects the largest-space indices (after excluding the `min_unbatched` smallest ones); `smallest` selects the smallest-space indices remaining after excluding the `min_unbatched` largest ones.	`largest`	No

canonicalize¶

Canonicalizes the input expressions.

Input	Output
Expression	Expression

This step does not accept any options.

cse¶

Performs common-subexpression elimination (CSE). Dependencies already recorded on the inputs (e.g. by a preceding dependencies step) are carried over onto the corresponding results. The introduced intermediates are named only uniquely within the entries of a single result, so a dependencies step with autodetect should run before this step rather than after it (see the note under dependencies).

Input	Output
ExportTree	ExportTree

Options¶
Option	Description	Default	Required
`min_usage`	Minimum number of times a given subexpression has to be used in order to be eligible for subexpression elimination.	2	No
`merge_inputs`	Whether multiple inputs shall be merged together in order to undergo combined rather than individual CSE. Instead of a boolean, an array of input IDs/aliases can be given, in which case merging happens only within the inputs matching each listed ID (matched like the groups of the `grouping` option of the export step). Each listed ID yields one output, aliased to `<step>.<ID>`. An entry of the array may itself be an array of IDs, in which case all inputs matching any of them are merged into one output, aliased to `<step>.<ID>` for each of these IDs. Inputs matching none of the entries undergo individual CSE; inputs matching more than one entry are an error.	false	No

density_fitting¶

Inserts the density-fitting decomposition of the two-electron integrals. Every occurrence of the tensor named by integral_label is rewritten into a contraction of two three-index tensors (named by df_tensor_label) carrying an additional auxiliary index from auxiliary_space. Expressions that don’t contain the targeted tensor are passed through unmodified.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`auxiliary_space`	Label of the index space (as declared under the top-level `index_spaces`) used for the auxiliary index introduced by the decomposition.		Yes
`integral_label`	Label of the tensor representing the two-electron integrals that shall be decomposed.	`g`	No
`df_tensor_label`	Label given to the two three-index tensors produced by the decomposition.	`DF`	No

dependencies¶

Establishes hard dependencies between result trees: whenever one tree is made to depend on another, the latter is guaranteed to be computed first. The dependencies are recorded on the inputs and consumed by the export step when it orders the results. Like export, this step is not restricted to a single input - all inputs are considered jointly so that dependencies can be detected and declared across them. The inputs are annotated in place and passed through unchanged, so they remain referenceable (under this step’s id) by subsequent steps. At least one of autodetect or deps must be given.

Input	Output
ExportTree	ExportTree

Options¶
Option	Description	Default	Required
`autodetect`	Whether to automatically add a dependency whenever the result of one tree is used as an input in another tree. Matching is done on the tensor block (label and index spaces) and hence is insensitive to the concrete index labels.	false	No (but see above)
`deps`	Object declaring dependencies explicitly. Each key is the id/alias of the dependent tree(s); its value is a single id/alias, or an array of them, naming the tree(s) that must be computed first. Group ids are allowed on both sides. These dependencies are added on top of any that `autodetect` discovers.	`{}`	No (but see above)

A dependency cycle - whether autodetected or declared - is an error and aborts the run when the export step tries to order the results.

Note

Run this step with autodetect before any cse step, not after. Common-subexpression elimination introduces intermediate results whose names are only unique within the entries of a single result; across results the same name (e.g. CSE1) is reused for unrelated intermediates. Since autodetection matches producers and consumers on the tensor block (see autodetect above), running it on CSE output conflates these same-named intermediates and can wire up spurious dependencies that form a cycle, aborting the subsequent export. Autodetecting dependencies first is safe because cse preserves the dependencies already recorded on its inputs.

export¶

Exports the given expressions as code. Unlike most other steps, export is not restricted to a single input - if several inputs are given, they are merged and jointly ordered before being exported together.

Input	Output
ExportTree	None

Options¶
Option	Description	Default	Required
`language`	Target language/format to export to. Currently, the only supported value is `itf`.		Yes
`output`	Path of the file the generated code is written to (relative to the driver file’s location, unless given as an absolute path).		Yes
`optimize`	Whether to run a further optimization pass (eliminating redundant operations) over the intermediate representation before generating code.	true	No
`check_stack_order`	Whether to verify that the generated memory operations (create, load, unload, destroy, persist) obey a stack-like order, as required by the target language (and throw an error if they do not). The check is applied to the final operations, after `optimize`.	false	No
`grouping`	Object mapping a group name to an input id/alias, or to an array of them. All entries belonging to any of these ids/aliases are exported as part of the named group. An id/alias must not be associated with more than one group. Any entry not matched by any of these rules is put into an implicit group named `Default`.	`{}`	No
`relative_order`	Array of id/alias names giving a hint for the relative ordering of results within the generated code. Used together with the hard dependencies among results (see the `dependencies` step) when determining the final order; the dependencies take precedence and `relative_order` only breaks ties.	`[]`	No
`imports`	Object mapping a serialized tensor expression to an import name. Tensor blocks matching the given expression are treated as externally supplied (imported) rather than being computed by the generated code.	`{}`	No
`irreps`	Point-group symmetry (IRREP) information. If given, the IRREPs of all exported tensors are determined from the IRREPs of the input tensors and emitted into the generated code. See below for the sub-schema.		No
`recycle_intermediates`	Whether to reuse the names (and thereby the storage) of intermediates introduced by the `cse` step once they are no longer needed. A later intermediate takes over the name of an earlier one if the earlier one’s last use precedes its definition, both are batched over the same indices, both span the same IRREPs (if `irreps` is given) and their shapes are compatible. Either a boolean or an object with the property `shape`, which is `multiset` (default; same index spaces up to permutation, the indices of the later intermediate are rearranged accordingly) or `block` (same index spaces in the same order). Independent of this option, if `irreps` is given, an intermediate whose name is used for different IRREPs in different groups is renamed such that every name spans a single set of IRREPs.	false	No
`meta`	Additional, language-specific metadata. See below for the sub-schema accepted when `language` is `itf`.	`{}`	No

The irreps option accepts the following properties:

Option	Description	Default	Required
`group`	The symmetry group. Either the name of one of the abelian point groups `C1`, `Ci`, `C2`, `Cs`, `C2v`, `C2h`, `D2` and `D2h` (with IRREPs named and ordered as in Molpro), `symbolic` for a group consisting of symbolic IRREPs only, or an object describing an explicit (possibly non-abelian) multiplication table with the properties `totally_symmetric` (name of the totally symmetric IRREP), `products` (object mapping `"X*Y"` to the IRREP(s) of the product; one ordering per pair suffices and products with the totally symmetric IRREP are implicit) and `conjugates` (object mapping complex IRREPs to their complex conjugate; one direction per pair suffices).		Yes
`symbols`	Array of names of symbolic IRREPs, i.e. IRREPs whose value is only known at runtime (e.g. the symmetry of a perturbation). Symbols remain unevaluated in the generated code (e.g. `B1uGp`), where every IRREP is its own inverse (`GpGp` is totally symmetric). For the named point groups, every name that is not an IRREP of the group must be declared here (this catches typos). For `symbolic`, any name is accepted unless this list is given. Not supported for explicit multiplication tables.	`[]`	No
`relations`	Array of relations between IRREPs of the form `"My1*My2 = My3"`, which are used to simplify products of symbolic IRREPs. Not supported for explicit multiplication tables.	`[]`	No
`tensors`	Object mapping tensors to their IRREP(s), given as a single IRREP name or an array of names (for tensors spanning several IRREPs). `"*"` stands for any IRREP. Keys are either a serialized tensor (matching the corresponding tensor block, as for `imports`) or otherwise a full-match regular expression for the tensor label (applying to all blocks of matching tensors, as for the `label` rules of the `filter` step). Tensor block keys take precedence over label keys and a label key spelling out a tensor’s label literally takes precedence over other matching label keys. Beyond that, all keys matching a given tensor must specify the same IRREPs. Tensors are referred to by their names in the expressions (e.g. `g` rather than `K`/`J`). Reserved tensors (e.g. Kronecker deltas) are totally symmetric and tensors computed during the export (results, intermediates) need not be listed.	`{}`	No
`variables`	Object mapping scalar variables to their IRREP(s), in the same format as `tensors`, except that keys are always full-match regular expressions for the variable label (an exactly spelled label takes precedence). Entries of `tensors` never apply to variables and vice versa. Variables not listed are totally symmetric and are not subject to `default`. A variable with an IRREP other than the totally symmetric one contributes that IRREP to the products it appears in (symbolic bookkeeping only). Powers of such variables are not supported.	`{}`	No
`default`	IRREP(s) of tensors not listed in `tensors`. The special value `error` turns unlisted tensors into an error, which is recommended to catch missing specifications.	`"*"`	No

Whitespace around IRREP names, * and = is ignored ("A * B = C" equals "A*B=C"). IRREP names themselves must not contain whitespace.

For language: itf, the IRREPs of created tensors are emitted as an irrep: entry of their !Create{...} statement, whereas the IRREPs of imported tensors are given as a trailing comment of their declaration (for informational purposes only).

For language: itf, meta accepts the following properties:

Option	Description	Default	Required
`index_spaces`	Object mapping an index space’s label to an object with `name` and `tag` properties, overriding the display name and tag used for that space in the generated ITF code. Spaces not listed here (or fields left unset for a listed space) fall back to built-in defaults (which likely results in an error).	`{}`	No
`min_index_id`	Non-negative integer offset that index IDs emitted into the generated ITF code start counting from.	0	No
`irrep_names`	Object mapping IRREP names (as used in the `irreps` option, e.g. `Ag` or `B1uGp`) to the names used in the generated ITF code. The key `""` refers to tensors spanning any IRREP. IRREPs not listed here keep their name.	`{}`	No

filter¶

Filters the input expressions and assigns them into different named groups based on provided filter rules. For a summed expression, each summand is tested against every group’s rules independently; for a non-summed expression, the expression as a whole is tested. Matching summands/expressions are accumulated into their group’s output, so this step can (and typically does) produce more than one output per input - one for each configured group.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`groups`	Object mapping a group name to a filter specification (see below). At least one group must be given.		Yes
`keep_empty`	Whether a group that ends up empty for a given input is still emitted as an (empty) output, rather than being omitted entirely.	true	No

Each entry of groups is a filter specification with the following properties:

Option	Description	Default	Required
`mode`	Whether a term has to satisfy `all` of the group’s rules (logical AND) or just `any` of them (logical OR) in order to be considered a match.	`all`	No
`rules`	Array of filter rules (see below) that define what this group matches.		Yes

Each entry of rules is an object with the following properties:

Option	Description	Default	Required
`type`	Kind of rule. `contains` matches if the term (or any of its subexpressions) contains a match for `expr` or `label`. `index_spaces` matches if the term contains a tensor whose number of indices in each of the spaces listed in `spaces` meets the given count/bounds. `index_count` matches if the term contains a tensor with exactly `count` indices in total.		Yes
`negate`	If true, inverts the result of this rule.	false	No
`expr`	Serialized expression that is searched for (recursively, i.e. also within subexpressions) within the term being tested. Mutually exclusive with `label`.		One of `expr`/`label`
`tensor_equality_mode`	Only used together with `expr`. Controls how tensors are compared while matching: `identity` requires exact index labels to match, whereas `block`/`shape` (synonyms) only compare index spaces (and basis instances, if any), ignoring concrete index labels.	`identity`	No
`label`	One (String) or several (Array of Strings) full-match regular expressions that are matched (recursively) against the labels of subexpressions within the term being tested. A match against any of the given patterns counts as a match for this rule. Mutually exclusive with `expr`. Be aware that in order to enter a backslash into the regular expression, you have to escape it in a JSON string as `\`. Together with `index_spaces`, the patterns instead restrict which tensors are considered, i.e. the same tensor has to satisfy both the label and the index counts.		One of `expr`/`label` for `contains`
`spaces`	Only used by `index_spaces`. Object mapping index space labels to the number of indices a tensor must have in that space, e.g. `{"u": 2, "a": 1}` (exact counts) or `{"u": {"min": 3}}` (`min` and/or `max` bounds). Spaces are matched exactly (indices in other spaces, including related ones, are not counted), and indices in spaces that are not listed are not constrained.		For `index_spaces`
`count`	Only used by `index_count`. Total number of (non-null) indices (bra, ket and auxiliary) a tensor must have.		For `index_count`

optimize¶

Symbolically rewrites the expressions for an improved numerical evaluation. Most prominently, this factors tensor contractions into a series of binary tensor contractions, chosen so as to minimize the configured objective.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`objective`	Cost metric to minimize when factorizing contractions: `DenseFLOPs` (the floating-point operation count).	`DenseFLOPs`	No
`reorder_sums`	Whether summands of a sum may be reordered so that terms sharing common intermediates end up next to each other.	true	No
`cse`	Whether to perform common-subexpression elimination while searching for a single term’s evaluation order. `subnet` recognizes equivalent subnetworks (more search effort, potentially fewer operations); `none` disables this. Not to be confused with the standalone `cse` step, which operates across a whole processing tree rather than within a single term.	`none`	No
`intermediate_size_penalty`	Per-intermediate memory-footprint penalty added to the cost of a contraction (only consulted when `objective` is `DenseFLOPs`).	0.0	No
`prune_outer_products`	Whether to prune disconnected (outer-product) subsets from the search space while looking for the best evaluation order.	true	No
`replay`	Configures the treatment of expressions that are evaluated repeatedly (e.g. once per iteration of an iterative solver), in which only some tensors (the volatile ones, e.g. amplitudes) change between evaluations. See below.		No

replay is an object with the following properties:

Option	Description	Default	Required
`volatile`	Identifies the volatile tensors. Either a single entry or an array of entries, where each entry is a String (a full-match regular expression for the tensor label), an object with a `label` property or an object with an `expr` property (a serialized tensor) and an optional `tensor_equality_mode` (`identity`, `block` or `shape`). The objects accept the same properties as the `contains` rules of the `filter` step, except that `tensor_equality_mode` defaults to `block`. A tensor is volatile if it matches any of the entries.		Yes
`weight`	Weight of the cost of contractions involving volatile tensors, relative to the ones that don’t (conceptually the number of evaluations).	1.0	No
`intermediate_label`	Label prefix of the persistent intermediates (see below). Has to be unique among all `optimize` steps.	`Persistent`	No
`persistent_id`	Name of the sub-ID referring to the persistent part of an output.	`persistent`	No
`volatile_id`	Name of the sub-ID referring to the volatile part of an output.	`volatile`	No

With replay, every largest subexpression that contains at least two tensors but no volatile one is replaced by a persistent intermediate, which only needs to be computed once. Equal subexpressions share an intermediate across all inputs of the step. Every input then produces two outputs: the definitions of the persistent intermediates (the persistent part) and the input’s optimized expressions, which reference these intermediates (the volatile part). For every ID <id> under which the outputs can be referenced (including the step ID itself), the two parts are available as <id>.persistent and <id>.volatile. <id> still refers to both parts.

The main use case is to export the persistent parts into their own code section, which is run once, before the sections that are run in every iteration:

{
    "steps": [
        {
            "id": "opt",
            "kind": "optimize",
            "inputs": "biorth",
            "options": {
                "replay": { "volatile": "T\\d\\w*", "weight": 10 }
            }
        },
        {
            "id": "treeify",
            "kind": "to_export_tree",
            "inputs": "opt"
        },
        {
            "kind": "export",
            "inputs": "treeify",
            "options": {
                "language": "itf",
                "output": "code.itfaa",
                "grouping": {
                    "Init": "persistent",
                    "Residual": "res.volatile"
                }
            }
        }
    ]
}

Only the steps relevant for this feature are shown.

Keep the following in mind:

A persistent intermediate that is used by several outputs is only part of the persistent part of the first of them. Use the step-wide <step>.persistent to get all of them.
There are no dependencies between the parts of an output, so exporting both into the same group gives no guarantee that the intermediates are computed before they are used. Put them into different groups.
Don’t reference both an ID and one of its sub-IDs in the grouping of the export step, as it is unspecified which of them takes effect.
Steps that merge their inputs (e.g. cse with merge_inputs) merge the parts back together if given both of them.

output¶

Outputs expressions in the chosen markup style. Mainly intended for debugging purposes.

Input	Output
Expression	None

Options¶
Option	Description	Default	Required
`format`	Markup style to print in. `latex` prints a LaTeX representation; `serialize` prints SeQuant’s own textual serialization format (see Serialization).	`serialize`	No
`annotate_symmetry`	Only relevant for `format: serialize`. Whether to include symmetry annotations in the printed representation.	true	No

project¶

Performs the chosen projection with the inputs.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`method`	Projection method to apply. Currently, the only supported value is `biorthogonal`, which transforms the result into a biorthogonal basis.		Yes

Note

Unlike in the v1 driver, there is no no-op (primitive) value for method - simply omit this step if no projection is needed.

read_input¶

Reads and parses expressions from files.

Input	Output
None	Expression

This step does not accept any options.

simplify¶

Simplifies the given expressions.

Input	Output
Expression	Expression

This step does not accept any options.

sort¶

Sorts the input expressions into lexicographic order: the summands of every sum are sorted, as are the factors of every product insofar as they commute. Unlike the order produced by canonicalize, which is based on hash values, this order depends only on the expressions themselves (tensors with the cardinal tensor labels of the context come first) and is hence stable across changes of the hashing algorithm. The order of the expressions within an input is not changed.

Input	Output
Expression	Expression

This step does not accept any options.

spintracing¶

Spintraces the given expressions. That is, it performs spin-integration and potentially also spin-summation.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`algorithm`	Spintracing algorithm to use. `rigorous` applies an algorithm that works for all cases; `closed_shell` applies a cheaper algorithm that assumes a closed-shell reference.	`rigorous`	No

Note

There is no none value for algorithm - simply omit this step if spintracing isn’t needed.

substitute¶

Makes substitutions in the given expressions. Substitution targets and their replacements are given directly as serialized expressions in the options (there is no mechanism to source them from another step’s output); they are applied in the order in which they are declared, with each rule operating on the result of the previous one.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`substitutions`	Object mapping a serialized expression identifying what to match (the key) to a serialized expression it shall be replaced with (the value). Multiple entries are applied in order.		Yes
`tensor_equality_mode`	Controls how tensors are compared while matching substitution targets: `identity` requires exact index labels to match, whereas `block`/`shape` (synonyms) only compare index spaces (and basis instances, if any), ignoring concrete index labels.	`block`	No
`result_relabeling`	Object mapping an old result label to a new one. After substitutions have been applied to an expression, if its result label matches a key in this map, it is renamed to the associated value.	`{}`	No
`keep_unchanged`	Whether an input that none of the substitutions changed is still emitted (as is), so that there is exactly one output per input, in input order.	false	No

Note

Unlike most other steps, substitute does not propagate the named aliases of its inputs onto its outputs - downstream steps need to reference its outputs via the auto-generated <step_id>.<index> scheme (or via this step’s own outputs property). Additionally, if none of the expressions belonging to a given input end up changed by any substitution, that input is dropped from the output entirely rather than being passed through unchanged (unless keep_unchanged is set).

to_export_tree¶

Converts the given expressions into a tree data structure suitable for exports.

Input	Output
Expression	ExportTree

This step does not accept any options.

validate¶

Validates the given expressions, e.g. checking that indices occur at most twice within a product, that all summands of a sum share the same external indices, and that a result’s declared indices are consistent with those actually occurring in its expression. Processing is aborted as soon as the first invalid expression is encountered.

Input	Output
Expression	None

This step does not accept any options.

Note

Since this step never produces any output, it is typically used without an explicit id.

Examples¶

{
  "driver_format_version": 2,
  "index_spaces": [
    {
      "label": "a",
      "size": 1000,
      "real_valued": true
    },
    {
      "label": "u",
      "size": 5,
      "real_valued": true
    },
    {
      "label": "i",
      "size": 80,
      "real_valued": true
    },
    {
      "label": "F",
      "size": 1500,
      "real_valued": true
    }
  ],
  "steps": [
    {
      "id": "input",
      "kind": "read_input",
      "options": {
        "file_path": [
          "nevpt2/nevpt2_en0.inp",
          "nevpt2/nevpt2_en.inp",

          "nevpt2/nevpt2_res1_i1.inp",
          "nevpt2/nevpt2_res1_s0.inp",
          "nevpt2/nevpt2_res2_s1_singles.inp",
          "nevpt2/nevpt2_res1_s1.inp",

          "nevpt2/nevpt2_res2_p0.inp",
          "nevpt2/nevpt2_res2_p2.inp",
          "nevpt2/nevpt2_res2_i2.inp",
          "nevpt2/nevpt2_res2_p1.inp",
          "nevpt2/nevpt2_res2_s1.inp",
          "nevpt2/nevpt2_res2_s2.inp"
        ],
        "default_symmetry": "antisymmetric"
      },
      "outputs": {
        "ecc0": "0",
        "ecc": "1",
        "en": "0-1",
        "res1": "2-5",
        "res1_i1": "2",
        "res1_s0": "3",
        "res2_s1_singles": "4",
        "res1_s1": "5",
        "res2": "6-11",
        "res2_p0": "6",
        "res2_p2": "7",
        "res2_i2": "8",
        "res2_p1": "9",
        "res2_s1": "10",
        "res2_s2": "11",
        "res": "2-11"
      }
    },
    {
      "kind": "validate",
      "inputs": "input"
    },
    {
      "id": "DF",
      "kind": "density_fitting",
      "inputs": "input",
      "options": {
        "auxiliary_space": "F"
      }
    },
    {
      "id": "traced",
      "kind": "spintracing",
      "inputs": "DF",
      "options": {
        "algorithm": "closed_shell"
      }
    },
    {
      "id": "biorth",
      "kind": "project",
      "inputs": "traced.res",
      "options": {
        "method": "biorthogonal"
      }
    },
    {
      "id": "opt",
      "kind": "optimize",
      "inputs": [
        "traced.en",
        "biorth"
      ]
    },
    {
      "id": "treeify",
      "kind": "to_export_tree",
      "inputs": "opt"
    },
    {
      "kind": "export",
      "inputs": "treeify",
      "options": {
        "language": "itf",
        "optimize": true,
        "output": "nevpt2_v2.itfaa",
        "grouping": {
          "Energy0": "ecc0",
          "Energy": "ecc",
          "Residual": "res"
        },
        "relative_order": [
          "ecc0",
          "ecc",
          "res1_i1",
          "res1_s0",
          "res2_s1_singles",
          "res1_s1",
          "res2_p0",
          "res2_p2",
          "res2_i2",
          "res2_p1",
          "res2_s1",
          "res2_s2"
        ],
        "imports": {
          "R2{a1;i1}": "R2:ec"
        },
        "meta": {
          "index_spaces": {
            "a": {
              "name": "External",
              "tag": "e"
            },
            "u": {
              "name": "Active",
              "tag": "a"
            },
            "i": {
              "name": "Closed",
              "tag": "c"
            },
            "F": {
              "name": "BasisMp2Fit",
              "tag": "F"
            }
          },
          "min_index_id": 1
        }
      }
    }
  ]
}

This is an abridged excerpt. The full, runnable driver file can be found at utilities/external-interface/examples/nevpt2_v2.json; it additionally demonstrates the batch_indices step and merging several inputs into a single export step.