InferenceReport
Summary
InferenceReport is a Julia package to automatically create reports based on MCMC inference methods.
Installing InferenceReport
- If you have not done so, install Julia. Julia 1.9 and higher are supported.
- Install
InferenceReportusing
using Pkg; Pkg.add("InferenceReport")Usage
InferenceReport's main function is report. It takes one mandatory argument, which at the moment can be either Pigeons' Parallel Tempering (PT) output, or MCMCChains' Chains struct.
We provide an example of each in the next two sections.
From Pigeons
First, run Parallel Tempering (PT) via Pigeons.
Then call report() on the resulting PT struct:
using InferenceReport
using Pigeons
pt = pigeons(
target = toy_mvn_target(2),
n_rounds = 4,
record = [traces; round_trip; record_default()])
report(pt)───────────────────────────────────────────────────────────────────────────────────────
scans restarts Λ time(s) allc(B) log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ────────── ──────────
2 0 0.609 3.98e-05 1.16e+04 -1.68 0.705 0.932
4 0 1.28 5.17e-05 1.84e+04 -1.86 0.527 0.858
8 0 1.39 5.44e-05 3.34e+04 -2.29 0.458 0.845
16 1 1.55 6.08e-05 5.82e+04 -2.27 0.646 0.828
───────────────────────────────────────────────────────────────────────────────────────
target_description...[skipped: no description provided]
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-02-26-23-36-15-mGJZTe4TThis will generate an HTML report with various useful diagnostic plots and open it in your default browser.
See Examples in the left side bar to see examples of such reports.
See report for more information on the options available.
From MCMCChains
MCMCChains.jl is used to store MCMC samples in packages such as Turing.jl.
Simply pass it into report to generate and open an HTML report:
using InferenceReport
using Turing
using MCMCChains
# example from Turing.jl's webpage
@model function coinflip(; N::Int)
p ~ Beta(1, 1)
y ~ filldist(Bernoulli(p), N)
return y
end;
y = Bool[1, 1, 1, 0, 0, 1]
model = coinflip(; N=length(y)) | (; y)
chain = sample(model, NUTS(), 100; progress=false)
report(chain)┌ Info: Found initial step size
└ ϵ = 3.2
target_description...[skipped: no description provided]
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: only applies to Pigeons]
lcb...[skipped: only applies to Pigeons]
gcb_progress...[skipped: only applies to Pigeons]
logz_progress...[skipped: only applies to Pigeons]
round_trip_progress...[skipped: only applies to Pigeons]
pigeons_summary...[skipped: only applies to Pigeons]
pigeons_inputs...[skipped: only applies to Pigeons]
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-02-26-23-38-05-18TKbV9nTarget descriptions
Two methods are available to specify descriptions for target distributions.
First, a markdown description can be passed in as argument:
using InferenceReport
using Pigeons
target = toy_mvn_target(2)
pt = pigeons(; target, n_rounds = 2)
report(pt;
target_description =
"""
The model description can use math: ``x^2``.
""")┌ Info: Neither traces, disk, nor online recorders included.
│ You may not have access to your samples (unless you are using a custom recorder, or maybe you just want log(Z)).
└ To add recorders, use e.g. pigeons(target = ..., record = [traces; record_default()])
────────────────────────────────────────────────────────────────────────────
scans Λ time(s) allc(B) log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ──────────
2 0.609 3.01e-05 1.14e+04 -1.68 0.705 0.932
4 1.28 4.5e-05 1.8e+04 -1.86 0.527 0.858
────────────────────────────────────────────────────────────────────────────
Could not build traces: ErrorException("type NamedTuple has no field traces")
target_description... ✓
pair_plot...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
trace_plot...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
moments...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
trace_plot_cumulative...[skipped: no traces provided (in Pigeons, use 'record = [traces])]
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress...[skipped: number restarts not recorded]
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-02-26-23-41-44-xHVOcAEEBut often, we may have a family of targets (e.g., in the above example, normals indexed by their dimensionality) and would like the documentation to be automatically generated based on the parameters.
To do so, implement your own pigeons_target_description as done in the following example:
using InferenceReport
using Pigeons
target = toy_mvn_target(2)
pt = pigeons(; target, n_rounds = 2)
const MyTargetType = typeof(target)
InferenceReport.pigeons_target_description(target::MyTargetType) =
"""
Some description.
It can use information in the target, e.g. here
to report that its dimension is: $(target.dim)
"""┌ Info: Neither traces, disk, nor online recorders included.
│ You may not have access to your samples (unless you are using a custom recorder, or maybe you just want log(Z)).
└ To add recorders, use e.g. pigeons(target = ..., record = [traces; record_default()])
────────────────────────────────────────────────────────────────────────────
scans Λ time(s) allc(B) log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ──────────
2 0.609 2.42e-05 1.14e+04 -1.68 0.705 0.932
4 1.28 2.51e-05 1.8e+04 -1.86 0.527 0.858
────────────────────────────────────────────────────────────────────────────Reproducibility instructions
Simple Pigeons reproducibility instructions can be added to the generated page. This is done via the reproducibility_command argument to report. This should be a string showing how to create an Inputs struct. When reproducibility_command is provided, InferenceReport will combine it to information queried in the current git repository to attempt to string together a mini-script that can be used to reproduce the result.
When the model is very simple, this can be done manually:
using InferenceReport
using Pigeons
inputs = Inputs(target = toy_mvn_target(1), n_rounds = 4, record = [traces])
pt = pigeons(inputs)
report(pt;
reproducibility_command = "Inputs(target = toy_mvn_target(1), n_rounds = 4, record = [traces])")──────────────────────────────────────────────────────
scans Λ log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ──────────
2 0.417 -1.56 0.782 0.954
4 0.564 -0.653 0.61 0.937
8 0.597 -0.833 0.869 0.934
16 0.686 -1.35 0.729 0.924
──────────────────────────────────────────────────────
target_description... ✓
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress...[skipped: number restarts not recorded]
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info... ✓
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-02-26-23-41-48-vkV8xSJkHowever, this approach is not recommended as the duplicated code leads to high risk that the reproducibility instruction drifts apart from the actual command used.
To avoid this, we provide a macro, @reproducible, which, given an expression, produces a pair containing the verbatim expression as well as its string value. Using this we can rewrite the above as:
using InferenceReport
using Pigeons
inputs, reproducibility_command =
@reproducible Inputs(
target = toy_mvn_target(1), n_rounds = 4, record = [traces])
pt = pigeons(inputs)
report(pt; reproducibility_command)──────────────────────────────────────────────────────
scans Λ log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ──────────
2 0.417 -1.56 0.782 0.954
4 0.564 -0.653 0.61 0.937
8 0.597 -0.833 0.869 0.934
16 0.686 -1.35 0.729 0.924
──────────────────────────────────────────────────────
target_description... ✓
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress...[skipped: number restarts not recorded]
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info... ✓
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-02-26-23-41-51-e1638OIpAdding postprocessors
Calling report() triggers a list of postprocessors, each creating a section in the generated report.
To add a custom section in the report, first, create a function taking as input the PostprocessContext which contains all required information such as the MCMC traces.
Here is an example of postprocessor outputting the number of dimensions:
using InferenceReport
using MCMCChains
import InferenceReport: get_chains, default_postprocessors, add_table, add_plot, add_markdown
function report_dim(context)
chns = get_chains(context)
params = names(chns, :parameters)
n_params = length(params)
add_markdown(context;
title = "Dimension",
contents = "The target has $n_params parameters."
)
endreport_dim (generic function with 1 method)Various utilities are available to generate contents, see for example add_table, add_plot, add_markdown.
Then, to use the custom postprocessor, add it to the postprocessors argument:
using Pigeons
pt = pigeons(
target = toy_mvn_target(2),
n_rounds = 4,
record = [traces; round_trip; record_default()])
report(pt; postprocessors = [report_dim; default_postprocessors()])───────────────────────────────────────────────────────────────────────────────────────
scans restarts Λ time(s) allc(B) log(Z₁/Z₀) min(α) mean(α)
────────── ────────── ────────── ────────── ────────── ────────── ────────── ──────────
2 0 0.609 4.04e-05 1.16e+04 -1.68 0.705 0.932
4 0 1.28 5.62e-05 1.84e+04 -1.86 0.527 0.858
8 0 1.39 4.31e-05 3.34e+04 -2.29 0.458 0.845
16 1 1.55 6.71e-05 5.82e+04 -2.27 0.646 0.828
───────────────────────────────────────────────────────────────────────────────────────
report_dim... ✓
target_description... ✓
pair_plot... ✓
trace_plot... ✓
moments... ✓
trace_plot_cumulative... ✓
mpi_standard_out...[skipped: no MPI std out files found]
lcb... ✓
gcb_progress... ✓
logz_progress... ✓
round_trip_progress... ✓
pigeons_summary... ✓
pigeons_inputs... ✓
reproducibility_info...[skipped: missing reproducibility_command]
Report generated at: /home/runner/work/InferenceReport.jl/InferenceReport.jl/docs/build/results/all/2024-02-26-23-41-52-wNioGwSCPull requests for additional postprocessors are welcome.
Documenter.jl integration
See our Documenter.jl make file to see how to integrate InferenceReport into a broader documentation page. The key functions used are headless, report_to_docs and as_doc_page.