Driver File v2¶

The external interface implementation provides a series of individual processing steps that can be selected and chained by means of the JSON driver file. Within this file, each processing step is represented as a JSON object with the following properties:

kind (required, String): This is a unique identifier for every different kind of steps and therefore determines what this step will do when exectuted.
id (String): Unique identifier for this particular step. This can be used to refer to (the output of) other steps.
options (Object): Set of options to tune the behavior of the step as needed. Some steps don’t have options, whereas others require you to specify them. For most steps, options are optional. The properties of the options object depend on the step’s kind - see Available Processing Step Kinds.
outputs (Object): This can be used to give human-readable names to individual outputs of the current step in form of name-output pairs. See Referencing Outputs - the step ID in this case is implicitly the current step’s ID and must not be included explicitly. These names are automatically propagated through the processing chain (for most kinds of steps).
inputs (String or Array of Strings): Specifies the inputs of this step, which must be outputs of other steps. See Referencing Outputs.

The different steps are listed objects in the top-level steps array. Steps are processed in order according to their order in this array.

Referencing Outputs¶

Outputs can be referenced by means of their IDs. An ID has the general format <step_id>.<output_id>. <step_id> must refer to the id of one of the steps that have been executed before. <output_id> can be the human-readable names specified via a step’s outputs property or an integer. In the latter case, the integer refers to the index of the output. Outputs are indexed starting from zero in the order they are produced.

<output_id> can also be an expression enclosed in square brackets in order to refer to multiple outputs at once. This can be a comma-delimited (no spaces!) list of output names or indices, or a range of indices which are of the form <from>-<to> like 0-5.

Examples of output IDs:

my_step.0
my_step.some_name
my_step.[0-2]
my_step.[0-2,some_name,5]

Finally, to refer to all outputs of a given step, just use the step’s ID without anything appended to it. That is, my_step would automatically refer to all outputs of the respective step.

How the Pipeline Executes¶

Steps are executed strictly in the order in which they appear in the steps array - there is no separate dependency resolution based on inputs. Given IDs must be unique across the whole driver file.

Two kinds of data can flow between steps: Expression data (one or more named result expressions) and ExportTree data (the export-ready tree form produced by the to_export_tree step). Every step kind’s documentation below states which of these it consumes and which it produces. Feeding a step data of the wrong kind (e.g. passing plain expressions into a step that expects an ExportTree) is an error.

Not every step produces output data - they exist purely for their side effects (e.g. aborting on invalid input, printing to the console or writing something to a file) - and therefore cannot be referenced by another step’s inputs.

Errors encountered anywhere in the pipeline (be it while parsing a step’s options or while executing it) abort the entire run; there is no partial/best-effort processing.

Available Processing Step Kinds¶

batch_indices¶

Configures the computation of results to happen in batches over certain result indices. For every result, its indices are ranked by the size of the index space they belong to. The smallest min_unbatched indices are always left un-batched; among the remaining indices, up to max_batched become batch indices, chosen from the end of the size-ranked list selected by selection_strategy. This step only annotates which indices shall be batched - whether batching is actually realized (and how) is up to whichever code-generation backend is used by the subsequent export step, and not every backend supports it.

Input	Output
ExportTree	ExportTree

Options¶
Option	Description	Default	Required
`min_unbatched`	Minimum number of (smallest-space) indices of a given result that are always left un-batched.	2	No
`max_batched`	Upper limit on the number of indices that may be selected for batching.	unbounded	No
`selection_strategy`	Which end of the size-ranked index list batching candidates are taken from. `largest` selects the largest-space indices (after excluding the `min_unbatched` smallest ones); `smallest` selects the smallest-space indices remaining after excluding the `min_unbatched` largest ones.	`largest`	No

canonicalize¶

Canonicalizes the input expressions.

Input	Output
Expression	Expression

This step does not accept any options.

cse¶

Performs common-subexpression elimination (CSE).

Input	Output
ExportTree	ExportTree

Options¶
Option	Description	Default	Required
`min_usage`	Minimum number of times a given subexpression has to be used in order to be eligible for subexpression elimination.	2	No
`merge_inputs`	Whether multiple inputs shall be merged together in order to undergo combined rather than individual CSE.	false	No

density_fitting¶

Inserts the density-fitting decomposition of the two-electron integrals. Every occurrence of the tensor named by integral_label is rewritten into a contraction of two three-index tensors (named by df_tensor_label) carrying an additional auxiliary index from auxiliary_space. Expressions that don’t contain the targeted tensor are passed through unmodified.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`auxiliary_space`	Label of the index space (as declared under the top-level `index_spaces`) used for the auxiliary index introduced by the decomposition.		Yes
`integral_label`	Label of the tensor representing the two-electron integrals that shall be decomposed.	`g`	No
`df_tensor_label`	Label given to the two three-index tensors produced by the decomposition.	`DF`	No

export¶

Exports the given expressions as code. Unlike most other steps, export is not restricted to a single input - if several inputs are given, they are merged and jointly ordered before being exported together.

Input	Output
ExportTree	None

Options¶
Option	Description	Default	Required
`language`	Target language/format to export to. Currently, the only supported value is `itf`.		Yes
`output`	Path of the file the generated code is written to (relative to the driver file’s location, unless given as an absolute path).		Yes
`optimize`	Whether to run a further optimization pass (eliminating redundant operations) over the intermediate representation before generating code.	true	No
`grouping`	Object mapping a group name to an input id/alias. All entries belonging to that id/alias are exported as part of the named group. Any entry not matched by any of these rules is put into an implicit group named `Default`.	`{}`	No
`relative_order`	Array of id/alias names giving a hint for the relative ordering of results within the generated code. Used together with the hard dependencies among results when determining the final order.	`[]`	No
`imports`	Object mapping a serialized tensor expression to an import name. Tensor blocks matching the given expression are treated as externally supplied (imported) rather than being computed by the generated code.	`{}`	No
`meta`	Additional, language-specific metadata. See below for the sub-schema accepted when `language` is `itf`.	`{}`	No

For language: itf, meta accepts the following properties:

Option	Description	Default	Required
`index_spaces`	Object mapping an index space’s label to an object with `name` and `tag` properties, overriding the display name and tag used for that space in the generated ITF code. Spaces not listed here (or fields left unset for a listed space) fall back to built-in defaults (which likely results in an error).	`{}`	No
`min_index_id`	Non-negative integer offset that index IDs emitted into the generated ITF code start counting from.	0	No

filter¶

Filters the input expressions and assigns them into different named groups based on provided filter rules. For a summed expression, each summand is tested against every group’s rules independently; for a non-summed expression, the expression as a whole is tested. Matching summands/expressions are accumulated into their group’s output, so this step can (and typically does) produce more than one output per input - one for each configured group.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`groups`	Object mapping a group name to a filter specification (see below). At least one group must be given.		Yes
`keep_empty`	Whether a group that ends up empty for a given input is still emitted as an (empty) output, rather than being omitted entirely.	true	No

Each entry of groups is a filter specification with the following properties:

Option	Description	Default	Required
`mode`	Whether a term has to satisfy `all` of the group’s rules (logical AND) or just `any` of them (logical OR) in order to be considered a match.	`all`	No
`rules`	Array of filter rules (see below) that define what this group matches.		Yes

Each entry of rules is an object with the following properties:

Option	Description	Default	Required
`type`	Kind of rule. Currently, the only supported value is `contains`, which matches if the term (or any of its subexpressions) contains a match for `expr` or `label`.		Yes
`negate`	If true, inverts the result of this rule.	false	No
`expr`	Serialized expression that is searched for (recursively, i.e. also within subexpressions) within the term being tested. Mutually exclusive with `label`.		One of `expr`/`label`
`tensor_equality_mode`	Only used together with `expr`. Controls how tensors are compared while matching: `identity` requires exact index labels to match, whereas `block`/`shape` (synonyms) only compare index spaces, ignoring concrete index labels.	`identity`	No
`label`	One (String) or several (Array of Strings) full-match regular expressions that are matched (recursively) against the labels of subexpressions within the term being tested. A match against any of the given patterns counts as a match for this rule. Mutually exclusive with `expr`. Be aware that in order to enter a backslash into the regular expression, you have to escape it in a JSON string as `\`.		One of `expr`/`label`

optimize¶

Symbolically rewrites the expressions for an improved numerical evaluation. Most prominently, this factors tensor contractions into a series of binary tensor contractions, chosen so as to minimize the configured objective.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`objective`	Cost metric to minimize when factorizing contractions. One of `DenseFLOPs`, `DenseSize`, `DensePeakSize` or `DensePeakSizeBatched`.	`DenseFLOPs`	No
`reorder_sums`	Whether summands of a sum may be reordered so that terms sharing common intermediates end up next to each other.	true	No
`cse`	Whether to perform common-subexpression elimination while searching for a single term’s evaluation order. `subnet` recognizes equivalent subnetworks (more search effort, potentially fewer operations); `none` disables this. Not to be confused with the standalone `cse` step, which operates across a whole processing tree rather than within a single term.	`none`	No
`intermediate_size_penalty`	Per-intermediate memory-footprint penalty added to the cost of a contraction (only consulted when `objective` is `DenseFLOPs`).	0.0	No
`prune_outer_products`	Whether to prune disconnected (outer-product) subsets from the search space while looking for the best evaluation order.	true	No

output¶

Outputs expressions in the chosen markup style. Mainly intended for debugging purposes.

Input	Output
Expression	None

Options¶
Option	Description	Default	Required
`format`	Markup style to print in. `latex` prints a LaTeX representation; `serialize` prints SeQuant’s own textual serialization format (see Serialization).	`serialize`	No
`annotate_symmetry`	Only relevant for `format: serialize`. Whether to include symmetry annotations in the printed representation.	true	No

project¶

Performs the chosen projection with the inputs.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`method`	Projection method to apply. Currently, the only supported value is `biorthogonal`, which transforms the result into a biorthogonal basis.		Yes

Note

Unlike in the v1 driver, there is no no-op (primitive) value for method - simply omit this step if no projection is needed.

read_input¶

Reads and parses expressions from files.

Input	Output
None	Expression

This step does not accept any options.

simplify¶

Simplifies the given expressions.

Input	Output
Expression	Expression

This step does not accept any options.

spintracing¶

Spintraces the given expressions. That is, it performs spin-integration and potentially also spin-summation.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`algorithm`	Spintracing algorithm to use. `rigorous` applies an algorithm that works for all cases; `closed_shell` applies a cheaper algorithm that assumes a closed-shell reference.	`rigorous`	No

Note

There is no none value for algorithm - simply omit this step if spintracing isn’t needed.

substitute¶

Makes substitutions in the given expressions. Substitution targets and their replacements are given directly as serialized expressions in the options (there is no mechanism to source them from another step’s output); they are applied in the order in which they are declared, with each rule operating on the result of the previous one.

Input	Output
Expression	Expression

Options¶
Option	Description	Default	Required
`substitutions`	Object mapping a serialized expression identifying what to match (the key) to a serialized expression it shall be replaced with (the value). Multiple entries are applied in order.		Yes
`tensor_equality_mode`	Controls how tensors are compared while matching substitution targets: `identity` requires exact index labels to match, whereas `block`/`shape` (synonyms) only compare index spaces, ignoring concrete index labels.	`block`	No
`result_relabeling`	Object mapping an old result label to a new one. After substitutions have been applied to an expression, if its result label matches a key in this map, it is renamed to the associated value.	`{}`	No

Note

Unlike most other steps, substitute does not propagate the named aliases of its inputs onto its outputs - downstream steps need to reference its outputs via the auto-generated <step_id>.<index> scheme (or via this step’s own outputs property). Additionally, if none of the expressions belonging to a given input end up changed by any substitution, that input is dropped from the output entirely rather than being passed through unchanged.

to_export_tree¶

Converts the given expressions into a tree data structure suitable for exports.

Input	Output
Expression	ExportTree

This step does not accept any options.

validate¶

Validates the given expressions, e.g. checking that indices occur at most twice within a product, that all summands of a sum share the same external indices, and that a result’s declared indices are consistent with those actually occurring in its expression. Processing is aborted as soon as the first invalid expression is encountered.

Input	Output
Expression	None

This step does not accept any options.

Note

Since this step never produces any output, it is typically used without an explicit id.

Examples¶

{
  "driver_format_version": 2,
  "index_spaces": [
    {
      "label": "a",
      "size": 1000,
      "real_valued": true
    },
    {
      "label": "u",
      "size": 5,
      "real_valued": true
    },
    {
      "label": "i",
      "size": 80,
      "real_valued": true
    },
    {
      "label": "F",
      "size": 1500,
      "real_valued": true
    }
  ],
  "steps": [
    {
      "id": "input",
      "kind": "read_input",
      "options": {
        "file_path": [
          "nevpt2/nevpt2_en0.inp",
          "nevpt2/nevpt2_en.inp",

          "nevpt2/nevpt2_res1_i1.inp",
          "nevpt2/nevpt2_res1_s0.inp",
          "nevpt2/nevpt2_res2_s1_singles.inp",
          "nevpt2/nevpt2_res1_s1.inp",

          "nevpt2/nevpt2_res2_p0.inp",
          "nevpt2/nevpt2_res2_p2.inp",
          "nevpt2/nevpt2_res2_i2.inp",
          "nevpt2/nevpt2_res2_p1.inp",
          "nevpt2/nevpt2_res2_s1.inp",
          "nevpt2/nevpt2_res2_s2.inp"
        ],
        "default_symmetry": "antisymmetric"
      },
      "outputs": {
        "ecc0": "0",
        "ecc": "1",
        "en": "0-1",
        "res1": "2-5",
        "res1_i1": "2",
        "res1_s0": "3",
        "res2_s1_singles": "4",
        "res1_s1": "5",
        "res2": "6-11",
        "res2_p0": "6",
        "res2_p2": "7",
        "res2_i2": "8",
        "res2_p1": "9",
        "res2_s1": "10",
        "res2_s2": "11",
        "res": "2-11"
      }
    },
    {
      "kind": "validate",
      "inputs": "input"
    },
    {
      "id": "DF",
      "kind": "density_fitting",
      "inputs": "input",
      "options": {
        "auxiliary_space": "F"
      }
    },
    {
      "id": "traced",
      "kind": "spintracing",
      "inputs": "DF",
      "options": {
        "algorithm": "closed_shell"
      }
    },
    {
      "id": "biorth",
      "kind": "project",
      "inputs": "traced.res",
      "options": {
        "method": "biorthogonal"
      }
    },
    {
      "id": "opt",
      "kind": "optimize",
      "inputs": [
        "traced.en",
        "biorth"
      ]
    },
    {
      "id": "treeify",
      "kind": "to_export_tree",
      "inputs": "opt"
    },
    {
      "kind": "export",
      "inputs": "treeify",
      "options": {
        "language": "itf",
        "optimize": true,
        "output": "nevpt2_v2.itfaa",
        "grouping": {
          "Energy0": "ecc0",
          "Energy": "ecc",
          "Residual": "res"
        },
        "relative_order": [
          "ecc0",
          "ecc",
          "res1_i1",
          "res1_s0",
          "res2_s1_singles",
          "res1_s1",
          "res2_p0",
          "res2_p2",
          "res2_i2",
          "res2_p1",
          "res2_s1",
          "res2_s2"
        ],
        "imports": {
          "R2{a1;i1}": "R2:ec"
        },
        "meta": {
          "index_spaces": {
            "a": {
              "name": "External",
              "tag": "e"
            },
            "u": {
              "name": "Active",
              "tag": "a"
            },
            "i": {
              "name": "Closed",
              "tag": "c"
            },
            "F": {
              "name": "BasisMp2Fit",
              "tag": "F"
            }
          },
          "min_index_id": 1
        }
      }
    }
  ]
}

This is an abridged excerpt. The full, runnable driver file can be found at utilities/external-interface/examples/nevpt2_v2.json; it additionally demonstrates the batch_indices step and merging several inputs into a single export step.