MissingPatterns.jl

MissingPatterns

Terminal-based text visualizations for missing data patterns in any Tables.jl-compatible source (DataFrames, CSV.File, NamedTuples of vectors, XLSX tables, ...), with zero plotting-library dependencies.

Exported API:

• plotmissing — where/how much is missing (heatmap; optional temporal grouping via by/period).
• missingpatterns — which columns are missing together (unique row patterns, à la mice::md.pattern()).
• missingcooccurrence — pairwise ϕ/Jaccard association of missingness masks.
• missingsummary — per-column table with counts, % and a sparkline of where along the rows the missing values concentrate.
• plotmissingdiff — before/after comparison (e.g. auditing an imputation step).
• missinghtml — the heatmap as a standalone HTML fragment.

ColorRamp

Monochromatic truecolor ramp for cell coloring. base is the dark neutral "no ink" tone, target the full color, and emphasis decides which side of the data gets the ink:

• :present (default) — present data is painted in target; missing data fades toward base. Holes read as dark gaps in a colored field.
• :missing — the inverse: fully-present blocks stay dark, missing data is painted in target.

MissingGridStats

Immutable, purely numeric/string result of scanning a DataFrame for missing values. Contains everything the renderer needs and nothing about how it will be drawn (no IO, no colors, no character choices). This separation is what makes the calculation independently unit-testable — no ANSI-stripping regexes required.

Fields:

• nrows, ncols: original DataFrame dimensions.
• dr, dc: displayed grid dimensions (== nrows/ncols when uncompressed).
• rows_per_cell, cols_per_cell: how many original rows/cols each block spans.
• needs_compression: whether any grouping occurred.
• proportions: dr × dc matrix, missing-fraction of each displayed block.
• col_header_pct: length-dc vector, % missing across each column group (full row range), used for the header row.
• colnames: length-dc display names (already range-joined when compressed).
• row_labels: length-dr row-range (or period-range) labels.
• row_lo, row_hi: the two endpoints of each row label, kept separate so the half-block renderer can splice pair labels ("lo of top – hi of bottom").
• group_desc: human-readable grouping description (e.g. "by DATA (year)"), empty when rows are grouped positionally.
• missing_count, total_cells: whole-table totals (not display-bounded).

PatternStats

Pure result of compute_pattern_stats: the set of unique row-wise missingness signatures found in a DataFrame, sorted by descending frequency (ties broken by first appearance in the data, so results are deterministic across runs regardless of hashing/iteration order).

Fields:

• nrows, ncols: original DataFrame dimensions.
• pattern_missing::BitMatrix: npatterns × ncols; true = missing in that pattern.
• counts::Vector{Int}: row count matching each pattern (same order as pattern_missing).
• colnames::Vector{String}.

RenderStyle

Everything the renderer needs about how to draw, precomputed once up front (border strings, cell width, ANSI codes) so the per-cell hot loop only ever reads plain fields — no recomputation, no closures capturing mutable state.

_accumulate_column!(block_counts, col, rows_per_cell) -> Int

Single pass over one DataFrame column, tallying missing values both into the per-row-block block_counts accumulator and into a running column total (returned). col arrives as a concrete-eltype AbstractVector because compute_missing_stats calls this through a per-column dynamic dispatch — this is the classic Julia "function barrier": the outer loop over heterogeneous DataFrame columns pays dynamic dispatch once per column, while everything inside this function specializes and compiles for that column's concrete type, giving fully type-stable, @simd-friendly scalar code with no Union{T,Missing} boxing in the hot inner loop.

_accumulate_column_grouped!(group_counts, col, gids) -> Int

Grouped sibling of _accumulate_column!: tallies missing values into per-group buckets (group membership given by gids, one id per row) instead of positional row blocks. Same function-barrier design — the inner loop specializes on the column's concrete eltype.

_bar_cell!(buf, ratio, cw, prefix, suffix)

Left-aligned horizontal frequency bar, filling ratio of the available interior width (cw - 2) with '█'. The rest is spaces. Padding and ANSI prefix/suffix follow the same convention as _cell! and _data_cell!.

_cell_glyph(prop, char_missing, char_present) -> Char

Single source of truth for "which glyph represents this block's missing fraction", used uniformly whether or not the display was compressed. For an uncompressed cell prop is always exactly 0.0 or 1.0, so this naturally degrades to char_present/char_missing with no special-casing needed.

_colored_cell!(buf, content, cw, prefix, suffix)

Center-padded text cell like _cell!, but with an ANSI prefix/suffix wrapping the content (padding stays uncolored so backgrounds don't bleed).

_compact_header_text(name, pct, cw, name_width) -> String

Compose the single compact header cell, e.g. "PREC 6%", guaranteed to fit in a cell of interior width cw - 2. The percentage is never sacrificed; the name is truncated (with …) as needed.

_compact_max_rows(target_lines, halfblock) -> Int

How many grid rows fit in target_lines total output lines under the compact layout. With half-blocks each output line carries two grid rows.

_data_cell!(buf, glyph, cell_chars, cw, prefix, suffix)

Writes one heatmap data cell directly to buf: padding, optional ANSI prefix/suffix, and the glyph repeated cell_chars times — with zero intermediate String allocations (compare to the original repeat(string(char), cell_chars) + double repeat(' ', pad) per cell). Since cw = max(cell_chars + 2, 9) by construction, cell_chars <= cw - 2 always holds, so (unlike _cell!) no truncation branch is needed here.

_diff_counts(before_col, after_col) -> (resolved, introduced)

Row-aligned pass over one column of both tables: resolved counts cells missing before but present after (e.g. imputed), introduced the reverse. Same function-barrier pattern as every other hot loop in the package.

_diff_rgb(delta, worse, better) -> NTuple{3,Int}

Signed variant of _ramp_rgb: zero delta is neutral dark gray; positive deltas (more missing after) blend toward worse, negative (holes filled) toward better — with the same ~30% + √ visibility floor, so even a single changed cell inside a huge block produces a visible tint.

_glyph_prefix(style, prop) -> String

ANSI foreground prefix for a colored glyph cell (classic layout and pattern table). Under :missing emphasis, fully-present cells keep the terminal's default color (historic behavior); under :present emphasis every cell is colored, since present data is exactly what carries the ink.

_halfblock_cell!(buf, fg, bg, cell_chars, cw, rst)

One compact data cell: cell_chars copies of '▀' whose ANSI foreground is the RGB tuple fg (top grid row) and background bg (bottom grid row). Color semantics live entirely with the caller, so the same primitive serves plotmissing (missingness ramp) and plotmissingdiff (signed delta colors). bg may be nothing when the grid has an odd number of rows and this is the final, unpaired line — the bottom half then keeps the terminal background.

_make_render_style(io; cell_chars, char_missing, char_present, name_width,
                    color_cells, show_row_range=false, row_labels=String[])

Builds a RenderStyle up front (border strings, cell width, ANSI codes). Deliberately decoupled from MissingGridStats — it only needs row_labels to size the optional row-range column — so it can be reused by any renderer in this package (currently the heatmap grid and the pattern table), not just plotmissing.

_pair_row_label(stats, top, bot) -> String

Row-range (or period-range) label spanning the two grid rows folded into one half-block line: the low endpoint of top joined to the high endpoint of bot. Works uniformly for positional row indices and temporal group labels.

_parse_hex(s) -> NTuple{3,Int}

Parse "#rrggbb" (leading # optional) into an RGB tuple.

_ramp_rgb(ramp, prop) -> NTuple{3,Int}

Map a block's missing fraction prop to an RGB color.

Downsampling-fidelity guarantee, in both emphasis modes: any prop > 0 gets a minimum blend of ~30% away from the fully-present color (with a square-root scale below that), so a single missing value averaged over thousands of rows still produces a visibly different shade. Small holes never vanish under compression.

• emphasis == :present: prop == 0 → full target color; increasing missingness darkens toward base (holes = dark gaps in a colored field).
• emphasis == :missing: prop == 0 → base; increasing missingness brightens toward target.

_table_info(tbl) -> (cols, colnames::Vector{String}, nrows, ncols)

Resolve any Tables.jl-compatible source to a column-accessible object plus its dimensions. This is the single entry point through which every public function consumes data, so the package works identically for DataFrames, CSV.File, NamedTuples of vectors, XLSX.gettable results, etc.

_use_color(io::IO) -> Bool

Canonical ecosystem convention for color-aware output: defer to the :color property of io via get(io, :color, false). On Julia >= 1.11 this already resolves correctly for a raw Base.TTY (terminfo-based detection is wired into Base.get for TTY). For older Julia versions (down to the package's 1.6 floor) a raw, unwrapped TTY doesn't carry that information yet, so we conservatively fall back to io isa Base.TTY. Callers who want to force (or suppress) color regardless of io's concrete type should wrap it explicitly, e.g. IOContext(io, :color => true) — exactly as any other Base/ecosystem show-like function expects.

compute_cooccurrence(tbl; method=:phi) -> (M, colnames, n_missing_per_col, nrows)

Pairwise association between the missingness masks of every pair of columns — the correlation-style answer to the question missingpatterns answers by enumeration: which columns go missing together?

Built on top of compute_pattern_stats: since each unique pattern already carries its row count, the pairwise tallies cost O(npatterns × k²) (k = missing columns per pattern) instead of O(nrows × ncols²) — for real data with few distinct patterns this is essentially free.

Methods:

• :phi — Pearson's ϕ coefficient of the two binary masks, in [-1, 1]. Positive: the columns tend to be missing together (rarely MCAR!); negative: their missingness repels.
• :jaccard — |A ∩ B| / |A ∪ B| of the missing-row sets, in [0, 1].

Degenerate pairs (a column with zero or all-missing rows) yield NaN.

compute_missing_stats(tbl; max_rows, max_cols) -> MissingGridStats

Compute all display-independent statistics for a Tables.jl-compatible tbl in a single pass per column, without ever materializing an nrows × ncols missing-value matrix. Memory footprint is O(dr*dc + dc + dr) — bounded by the display size (max_rows × max_cols), not by the data itself. Row-range labels are always built (they are at most max_rows tiny strings).

compute_missing_stats_grouped(tbl, by, period; max_rows, max_cols) -> MissingGridStats

Like compute_missing_stats, but rows are grouped by the calendar period of the by column's values instead of by position. Groups are sorted chronologically; rows with a missing date form a trailing ∅ group. When there are more periods than max_rows, consecutive periods are merged into one block and labeled as a range (e.g. 2004-2005), with proportions weighted by each period's true row count — so unequal-sized periods never distort the picture.

compute_pattern_stats(tbl) -> PatternStats

Compute the unique row-wise missingness patterns of a Tables.jl-compatible tbl and their frequencies, sorted most-common first.

missingcooccurrence([io::IO=stdout], tbl; method=:phi, cell_chars=5,
                     name_width=4, color=:auto, missing_color="#f3a9a9",
                     max_cols=20)

Display the pairwise co-occurrence matrix of missingness between columns — ϕ coefficient (default) or Jaccard index of the missing masks. High positive values mean "these columns go missing together", which is strong evidence against MCAR and directly informs imputation strategy.

If the table has more than max_cols columns, the max_cols columns with the most missing values are shown (they carry the information) and a note reports how many were omitted. Diagonal cells are —; degenerate pairs (columns with no missing values) are ·.

Cell text is the coefficient (-1.00…1.00); with color enabled, cell intensity scales with |value| using missing_color.

missinghtml(tbl; max_rows=200, max_cols=60, missing_color="#f3a9a9",
            emphasis=:present, title="Missing data") -> String
missinghtml(path::AbstractString, tbl; kwargs...) -> path

Render the missing-data heatmap as a standalone HTML fragment (dark-themed <div>, no external CSS/JS) — suitable for pasting into a blog post, notebook export, or report. Column headers are rotated for readability; every cell carries a tooltip with its row range and exact missing percentage. The same compression engine and color ramp as plotmissing are used, so the two outputs always agree — HTML just affords a much larger grid (defaults: 200×60 blocks).

The one-argument form returns the HTML String; the two-argument form writes it to path and returns the path.

missingpatterns([io::IO=stdout], tbl; max_patterns=20, cell_chars=5,
                 char_missing='█', char_present='░', name_width=4,
                 color_cells=false, missing_color="#f3a9a9",
                 emphasis=:present, show_bar=true, min_pct=0.0)

Display the unique row-wise missingness patterns found in a Tables.jl-compatible tbl, sorted by descending frequency — i.e. which columns tend to be missing together.

This complements plotmissing, which shows where/how much is missing: missingpatterns shows which combinations of missing columns actually occur, the same diagnostic produced by R's mice::md.pattern(). Useful for reasoning about the missingness mechanism (e.g. if two columns are always missing together, that's rarely MCAR) and for choosing an imputation strategy. For a correlation-style view of the same question, see missingcooccurrence.

Arguments

• io::IO: output stream (default: stdout).
• tbl: any Tables.jl-compatible table.
• max_patterns::Int: maximum number of patterns to display, most-frequent first (default: 20). Patterns beyond this are summarized in a trailing count.
• cell_chars::Int: number of repeated characters per cell (default: 5, max: 80).
• char_missing::Char: character for a missing column in a pattern (default: '█').
• char_present::Char: character for a present column in a pattern (default: '░').
• name_width::Int: max characters shown for column names before truncating with … (default: 4; set to 0 to show full names, bounded by cell width).
• color_cells::Bool: apply the same color ramp as plotmissing (default: false).
• missing_color::String: hex color of the ramp, as in plotmissing (default: "#f3a9a9").
• emphasis::Symbol: :present (present columns colored, missing dark) or :missing (inverted), as in plotmissing (default: :present).
• show_bar::Bool: append an UpSet-style horizontal frequency bar per pattern, scaled to the most common displayed pattern (default: true).
• min_pct::Float64: hide patterns matching fewer than this percentage of rows (default: 0.0 — show all). Hidden patterns are reported in the trailing summary line.

Returns

• nothing. The table is written to io.

missingsummary([io::IO=stdout], tbl; bins=20, sortby=:missing,
                color=:auto, missing_color="#f3a9a9")

Per-column missing-data overview: name, element type, missing count, %, and a sparkline showing where along the rows the missing values concentrate — the row axis is split into bins equal blocks and each block maps to a bar height proportional to its missing fraction. A block with even one missing value renders at least the smallest bar (same visibility guarantee as plotmissing); a block with none renders blank.

Arguments

• bins::Int: sparkline resolution (default: 20).
• sortby::Symbol: :missing (descending missing count, default), :name, or :none (table order).
• color::Symbol / missing_color::String: as in plotmissing; the sparkline colors bars by their missing fraction.

plotmissing([io::IO=stdout], tbl; cell_chars=5, char_missing='█', char_present='░',
            name_width=4, color_cells=false, show_row_range=false,
            max_rows=50, max_cols=20,
            layout=:auto, target_lines=28, color=:auto,
            missing_color="#f3a9a9", emphasis=:present,
            by=nothing, period=:year)

Display a text-based heatmap of missing value patterns in any Tables.jl-compatible source (DataFrame, CSV.File, NamedTuple of vectors, ...). When the data exceeds the display limits, multiple rows/columns are grouped into a single cell using a Unicode block-character gradient (classic layout) or an ANSI-colored half-block encoding (compact layout).

Layouts

• :classic — the original layout: one grid row per line, 3-line header, 6-line summary. Best in a full terminal with room to scroll.
• :compact — fits the entire plot (grid + header + summary) in at most target_lines lines, so IDE/Jupyter output cells never truncate it. With color available, each output line encodes two grid rows via '▀' (foreground = top row, background = bottom row), doubling vertical resolution; without color it falls back to the glyph gradient at one row per line. Any block containing even a single missing value is rendered in a shade distinct from "fully present", so fine holes survive compression.
• :auto (default) — uses :classic when it fits within target_lines, :compact otherwise.

Temporal grouping

• by::Union{Nothing,Symbol,String}: name of a Date/DateTime column. When set, rows are grouped by the values of that column (not by position), so the vertical axis becomes honest calendar time and row labels show periods (e.g. 2004, 2013-Q2). Rows whose by value is missing form a trailing ∅ group. Row labels are always shown in this mode. If there are more periods than fit the budget, consecutive periods are merged and labeled as ranges.
• period::Symbol: :year (default), :quarter, :month, :week, :day.

Arguments

• io::IO: output stream (default: stdout).
• tbl: any Tables.jl-compatible table.
• cell_chars::Int: number of repeated characters per heatmap cell (default: 5, max: 80).
• char_missing::Char: character for fully-missing cells (default: '█').
• char_present::Char: character for fully-present cells (default: '░').
• name_width::Int: max characters shown for column names before truncating with … (default: 4; set to 0 to show full names, bounded by cell width).
• color_cells::Bool: apply the color ramp to classic-layout glyphs. The compact half-block layout always colors its cells (default: false).
• show_row_range::Bool: display row-range (or period) labels in a left-hand column (default: false; forced true when by is set).
• max_rows::Int: maximum display rows before compression in the classic layout (default: 50). Ignored by :compact, which derives its own limit from target_lines.
• max_cols::Int: maximum display columns before compression (default: 20).
• layout::Symbol: :auto, :classic, or :compact (default: :auto).
• target_lines::Int: total line budget for the compact layout, including borders, header and summary (default: 28 — safely under typical IDE output-cell limits of ~30 lines).
• color::Symbol: :auto (respect io's :color property / TTY detection), :always (force ANSI codes — use this in VS Code/Jupyter notebooks, whose output cells render ANSI but whose stdout is not a TTY), or :never (plain text — use when redirecting to a file).
• missing_color::String: hex color ("#rrggbb") of the ramp (default: "#f3a9a9").
• emphasis::Symbol: which side of the data carries the ink (default: :present). With :present, present data is painted in missing_color and missing data fades to dark gray — holes read as dark gaps in a colored field. With :missing, the ramp is inverted. In both modes, any block containing even one missing value renders in a shade visibly different from a fully-present block.

Returns

• nothing. The plot is written to io.

plotmissingdiff([io::IO=stdout], before, after; cell_chars=5, name_width=4,
                 max_cols=20, target_lines=28, color=:auto,
                 missing_color="#f3a9a9", filled_color="#a9f3c1")

Compare the missingness of two same-shaped tables — typically the same dataset before and after an imputation step, or two releases of a periodic microdata file. Rendered in the compact layout (fits target_lines):

• neutral dark gray — block unchanged;
• tint of missing_color — block got more missing (introduced holes);
• tint of filled_color — block got less missing (holes resolved).

With color, half-blocks encode two row blocks per line (as in plotmissing); without color, cells fall back to + (more missing), - (fewer) and · (unchanged) glyphs. The summary line reports exact cell-level counts of resolved and introduced missing values, computed by a row-aligned pass (not from block averages).

Both tables must have identical dimensions and column names, in order.

render_grid_compact!(buf, stats, style; halfblock)

Compact grid: condensed chrome always; half-block vertical doubling when halfblock (requires style.use_color), classic glyphs otherwise.

render_pattern_table!(buf, stats, style, max_patterns;
                      show_bar=true, min_pct=0.0) -> (shown, nkept)

Draws the pattern table for stats, reusing the exact same border/cell primitives as render_grid! — one row per unique missingness pattern (already sorted most-common first), one column per variable plus trailing n/% columns and, when show_bar, an UpSet-style horizontal frequency bar scaled to the most common displayed pattern. Patterns whose relative frequency is below min_pct (percent of rows) are filtered out before the max_patterns cap is applied. Returns (shown, nkept): how many patterns were rendered and how many survived the min_pct filter, so the caller can report both kinds of omission.

render_summary_compact!(buf, stats, style)

Single-line summary carrying the same information as render_summary! (dimensions, compression ratio, missing/present counts and percentages).