InferenceReport

Summary

InferenceReport is a Julia package to automatically create reports based on MCMC inference methods.

Installing InferenceReport

  1. If you have not done so, install Julia. Julia 1.9 and higher are supported.
  2. Install InferenceReport using
using Pkg; Pkg.add("InferenceReport")

Usage

InferenceReport's main function is report. It takes one mandatory argument, which at the moment can be either Pigeons' Parallel Tempering (PT) output, or MCMCChains' Chains struct.

We provide an example of each in the next two sections.

From Pigeons

First, run Parallel Tempering (PT) via Pigeons.

Then call report() on the resulting PT struct:

using InferenceReport
using Pigeons

pt = pigeons(
        target = toy_mvn_target(2),
        n_rounds = 4,
        record = [traces; round_trip; record_default()])

report(pt)
───────────────────────────────────────────────────────────────────────────────────────
  scans     restarts      Λ        time(s)    allc(B)  log(Z₁/Z₀)   min(α)     mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ────────── ──────────
        2          0      0.609   4.53e-05   1.16e+04      -1.68      0.705      0.932
        4          0       1.28    5.5e-05   1.84e+04      -1.86      0.527      0.858
        8          0       1.39   6.15e-05   3.34e+04      -2.29      0.458      0.845
       16          1       1.55   0.000101   5.82e+04      -2.27      0.646      0.828
───────────────────────────────────────────────────────────────────────────────────────
target_description...[skipped: no description provided]
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress... ✓
swaps_plot... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-02-55-bFzIKdoX

This will generate an HTML report with various useful diagnostic plots and open it in your default browser.

See Examples in the left side bar to see examples of such reports.

See report for more information on the options available.

You may also generate a TeX/PDF version using Documenter.jl's TeX support as follows:

report(pt; writer=InferenceReport.Documenter.LaTeX(platform = "docker"))
target_description...[skipped: no description provided]
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress... ✓
swaps_plot... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-03-01-1L7BkYVX

From MCMCChains

MCMCChains.jl is used to store MCMC samples in packages such as Turing.jl.

Simply pass it into report to generate and open an HTML report:

using InferenceReport
using Turing
using MCMCChains

# example from Turing.jl's webpage
@model function coinflip(; N::Int)
    p ~ Beta(1, 1)
    y ~ filldist(Bernoulli(p), N)
    return y
end;

y = Bool[1, 1, 1, 0, 0, 1]
model = coinflip(; N=length(y)) | (; y)
chain = sample(model, NUTS(), 100; progress=false)

report(chain)
┌ Info: Found initial step size
└   ϵ = 3.2
target_description...[skipped: no description provided]
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: only applies to Pigeons]
lcb...[skipped: only applies to Pigeons]
gcb_progress...[skipped: only applies to Pigeons]
logz_progress...[skipped: only applies to Pigeons]
round_trip_progress...[skipped: only applies to Pigeons]
swaps_plot...[skipped: only applies to Pigeons]
pigeons_summary...[skipped: only applies to Pigeons]
pigeons_inputs...[skipped: only applies to Pigeons]
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-07-52-E2xtzfEw

Target descriptions

Two methods are available to specify descriptions for target distributions.

First, a TargetDescription can be passed in as argument:

using InferenceReport
using Pigeons

target = toy_mvn_target(2)
pt = pigeons(; target, n_rounds = 2)

target_description = TargetDescription(
    text = """
    The model description can use math: ``x^2``.
    """)
report(pt; target_description)
┌ Info: Neither traces, disk, nor online recorders included.
│    You may not have access to your samples (unless you are using a custom recorder, or maybe you just want log(Z)).
└    To add recorders, use e.g. pigeons(target = ..., record = [traces; record_default()])
────────────────────────────────────────────────────────────────────────────
  scans        Λ        time(s)    allc(B)  log(Z₁/Z₀)   min(α)     mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ──────────
        2      0.609   4.23e-05   1.14e+04      -1.68      0.705      0.932
        4       1.28   5.23e-05    1.8e+04      -1.86      0.527      0.858
────────────────────────────────────────────────────────────────────────────
Could not build traces: ErrorException("type NamedTuple has no field traces")
target_description... ✓
pair_plot...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
trace_plot...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
moments...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
trace_plot_cumulative...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress...[skipped: number restarts not recorded]
swaps_plot... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-11-39-lHedHob4

Sometimes we may have a family of targets (e.g., in the above example, normals indexed by their dimensionality) and would like the documentation to be automatically generated based on the parameters.

To do so, implement your own pigeons_target_description as done in the following example:

using InferenceReport
using Pigeons

target = toy_mvn_target(2)
pt = pigeons(; target, n_rounds = 2)

const MyTargetType = typeof(target)
InferenceReport.pigeons_target_description(target::MyTargetType) = TargetDescription(
    text = """
    Some description.

    It can use information in the target, e.g. here
    to report that its dimension is: $(target.dim)
    """
)
┌ Info: Neither traces, disk, nor online recorders included.
│    You may not have access to your samples (unless you are using a custom recorder, or maybe you just want log(Z)).
└    To add recorders, use e.g. pigeons(target = ..., record = [traces; record_default()])
────────────────────────────────────────────────────────────────────────────
  scans        Λ        time(s)    allc(B)  log(Z₁/Z₀)   min(α)     mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ──────────
        2      0.609    2.1e-05   1.14e+04      -1.68      0.705      0.932
        4       1.28   2.27e-05    1.8e+04      -1.86      0.527      0.858
────────────────────────────────────────────────────────────────────────────

Adding references to a bibliography

To create a bibliography, we provide automatic integration with DocumenterCitations.

In the target description's text, use a syntax like 

[bib_key](@citet)

to include a citation (see DocumenterCitations for more citation styles). Then pass include the contents of bibtex file to pigeons_target_description's bibliography argument.

using InferenceReport
using Pigeons

target = toy_mvn_target(2)
pt = pigeons(; target, n_rounds = 2)

report(pt;
    target_description = TargetDescription(
        text = """
        The model description can use math: ``x^2``.
        Citation: [neal_slice_2003](@citet)
        """,
        bibliography = InferenceReport.example_bib()
    ))
┌ Info: Neither traces, disk, nor online recorders included.
│    You may not have access to your samples (unless you are using a custom recorder, or maybe you just want log(Z)).
└    To add recorders, use e.g. pigeons(target = ..., record = [traces; record_default()])
────────────────────────────────────────────────────────────────────────────
  scans        Λ        time(s)    allc(B)  log(Z₁/Z₀)   min(α)     mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ──────────
        2      0.609   1.51e-05   1.14e+04      -1.68      0.705      0.932
        4       1.28   2.02e-05    1.8e+04      -1.86      0.527      0.858
────────────────────────────────────────────────────────────────────────────
Could not build traces: ErrorException("type NamedTuple has no field traces")
target_description... ✓
pair_plot...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
trace_plot...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
moments...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
trace_plot_cumulative...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress...[skipped: number restarts not recorded]
swaps_plot... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-11-43-u3lKE7fZ

See funnel_model and Bibliography in the navigation bar to see examples of linked bibliographic items.

Reproducibility instructions

Simple Pigeons reproducibility instructions can be added to the generated page. This is done via the reproducibility_command argument to report. This should be a string showing how to create an Inputs struct. When reproducibility_command is provided, InferenceReport will combine it to information queried in the current git repository to attempt to string together a mini-script that can be used to reproduce the result.

To help filling reproducibility_command, we provide a macro, @reproducible, which, given an expression, produces a pair containing the verbatim expression as well as its string value. Here his an example:

using InferenceReport
using Pigeons

inputs, reproducibility_command =
        @reproducible Inputs(
            target = toy_mvn_target(1), n_rounds = 4, record = [traces])
pt = pigeons(inputs)

report(pt; reproducibility_command)
──────────────────────────────────────────────────────
  scans        Λ      log(Z₁/Z₀)   min(α)     mean(α)
────────── ────────── ────────── ────────── ──────────
        2      0.417      -1.56      0.782      0.954
        4      0.564     -0.653       0.61      0.937
        8      0.597     -0.833      0.869      0.934
       16      0.686      -1.35      0.729      0.924
──────────────────────────────────────────────────────
target_description... ✓
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress...[skipped: number restarts not recorded]
swaps_plot... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info... ✓
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-11-46-hFomjP6f

Adding postprocessors

Calling report() triggers a list of postprocessors, each creating a section in the generated report.

To add a custom section in the report, first, create a function taking as input the PostprocessContext which contains all required information such as the MCMC traces.

Here is an example of postprocessor outputting the number of dimensions:

using InferenceReport
using MCMCChains
import InferenceReport: get_chains, default_postprocessors, add_table, add_plot, add_markdown

function report_dim(context)
    chns = get_chains(context)
    params = names(chns, :parameters)
    n_params = length(params)
    add_markdown(context;
        title = "Dimension",
        contents = "The target has $n_params parameters."
    )
end
report_dim (generic function with 1 method)

Various utilities are available to generate contents, see for example add_table, add_plot, add_markdown.

Then, to use the custom postprocessor, add it to the postprocessors argument:

using Pigeons

pt = pigeons(
        target = toy_mvn_target(2),
        n_rounds = 4,
        record = [traces; round_trip; record_default()])

report(pt; postprocessors = [report_dim; default_postprocessors()])
───────────────────────────────────────────────────────────────────────────────────────
  scans     restarts      Λ        time(s)    allc(B)  log(Z₁/Z₀)   min(α)     mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ────────── ──────────
        2          0      0.609   4.11e-05   1.16e+04      -1.68      0.705      0.932
        4          0       1.28   5.85e-05   1.84e+04      -1.86      0.527      0.858
        8          0       1.39   6.22e-05   3.34e+04      -2.29      0.458      0.845
       16          1       1.55    7.1e-05   5.82e+04      -2.27      0.646      0.828
───────────────────────────────────────────────────────────────────────────────────────
report_dim... ✓
target_description... ✓
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress... ✓
swaps_plot... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-09-20-19-11-49-ITIFgSZQ

Pull requests for additional postprocessors are welcome.

Documenter.jl integration

Note: at the moment, this feature is supported on Julia 1.10+ only.

See our Documenter.jl make file to see how to integrate InferenceReport into a broader documentation page. The key functions used are headless, report_to_docs and as_doc_page.