Internals

ColorRamp

Monochromatic truecolor ramp for cell coloring. base is the dark neutral "no ink" tone, target the full color, and emphasis decides which side of the data gets the ink:

• :present (default) — present data is painted in target; missing data fades toward base. Holes read as dark gaps in a colored field.
• :missing — the inverse: fully-present blocks stay dark, missing data is painted in target.

MissingGridStats

Immutable, purely numeric/string result of scanning a DataFrame for missing values. Contains everything the renderer needs and nothing about how it will be drawn (no IO, no colors, no character choices). This separation is what makes the calculation independently unit-testable — no ANSI-stripping regexes required.

Fields:

• nrows, ncols: original DataFrame dimensions.
• dr, dc: displayed grid dimensions (== nrows/ncols when uncompressed).
• rows_per_cell, cols_per_cell: how many original rows/cols each block spans.
• needs_compression: whether any grouping occurred.
• proportions: dr × dc matrix, missing-fraction of each displayed block.
• col_header_pct: length-dc vector, % missing across each column group (full row range), used for the header row.
• colnames: length-dc display names (already range-joined when compressed).
• row_labels: length-dr row-range (or period-range) labels.
• row_lo, row_hi: the two endpoints of each row label, kept separate so the half-block renderer can splice pair labels ("lo of top – hi of bottom").
• group_desc: human-readable grouping description (e.g. "by DATA (year)"), empty when rows are grouped positionally.
• missing_count, total_cells: whole-table totals (not display-bounded).

MissingReport

A table plus display options, rendered on demand in whatever medium asks for it. Construct it with missingreport rather than directly.

PatternStats

Pure result of compute_pattern_stats: the set of unique row-wise missingness signatures found in a DataFrame, sorted by descending frequency (ties broken by first appearance in the data, so results are deterministic across runs regardless of hashing/iteration order).

Fields:

• nrows, ncols: original DataFrame dimensions.
• pattern_missing::BitMatrix: npatterns × ncols; true = missing in that pattern.
• counts::Vector{Int}: row count matching each pattern (same order as pattern_missing).
• colnames::Vector{String}.

RenderStyle

Everything the renderer needs about how to draw, precomputed once up front (border strings, cell width, ANSI codes) so the per-cell hot loop only ever reads plain fields — no recomputation, no closures capturing mutable state.

_accumulate_column!(block_counts, col, rows_per_cell, isna) -> Int

Single pass over one DataFrame column, tallying missing values both into the per-row-block block_counts accumulator and into a running column total (returned). col arrives as a concrete-eltype AbstractVector because compute_missing_stats calls this through a per-column dynamic dispatch — this is the classic Julia "function barrier": the outer loop over heterogeneous DataFrame columns pays dynamic dispatch once per column, while everything inside this function specializes and compiles for that column's concrete type, giving fully type-stable, @simd-friendly scalar code with no Union{T,Missing} boxing in the hot inner loop.

isna is taken as a type parameter so the predicate is baked into that specialization too: with the ismissing default the branch compiles away entirely for a column whose eltype admits no Missing.

_accumulate_column_grouped!(group_counts, col, gids) -> Int

Grouped sibling of _accumulate_column!: tallies missing values into per-group buckets (group membership given by gids, one id per row) instead of positional row blocks. Same function-barrier design — the inner loop specializes on the column's concrete eltype.

_accumulate_row!(per_row, col) -> per_row

Add one column's missingness into the per-row tally. Function barrier again: specialized on the column's concrete eltype, so the loop stays scalar.

_bar_cell!(buf, ratio, cw, prefix, suffix)

Left-aligned horizontal frequency bar, filling ratio of the available interior width (cw - 2) with '█'. The rest is spaces. Padding and ANSI prefix/suffix follow the same convention as _cell! and _data_cell!.

_cell!(buf, content, cw, prefix="", suffix="")

Center-padded text cell. With prefix/suffix given, they wrap the content as an ANSI escape pair while the padding itself stays uncolored (so backgrounds don't bleed into neighboring cells).

_cell_glyph(prop, char_missing, char_present) -> Char

Single source of truth for "which glyph represents this block's missing fraction", used uniformly whether or not the display was compressed. For an uncompressed cell prop is always exactly 0.0 or 1.0, so this naturally degrades to char_present/char_missing with no special-casing needed.

_cell_prefix_suffix(style, prop, color_on) -> (prefix, suffix)

ANSI prefix/suffix pair for a heatmap glyph cell, ready to pass straight into _cell!/_data_cell!. Both are "" when color_on is false, or when _glyph_prefix itself opts out (e.g. a fully-present cell under :missing emphasis) — the single place this "no coloring → no prefix/suffix" idiom lives, instead of being repeated at every call site.

_check_isna(isna, colnames)

Reject a per-column isna naming a column the table does not have. Silently ignoring the entry would leave the user staring at a plot that shows no missing data with no hint that their key was a typo.

_column_order(tbl, cols, colnames, ncols, order, isna) -> Union{Nothing,Vector{Int}}

Resolve an order keyword into a permutation of source-column indices, or nothing for :table (the identity, which the kernels take as a fast path).

• :table — table order (nothing).
• :missing — most missing values first; ties keep table order.
• :name — alphabetical by column name.
• :cluster — _seriate_columns over the ϕ matrix, so columns that go missing together sit next to each other. Costs one extra pass over the data to build the pattern table.

_compact_header_text(name, pct, cw, name_width) -> String

Compose the single compact header cell, e.g. "PREC 6%", guaranteed to fit in a cell of interior width cw - 2. The percentage is never sacrificed; the name is truncated (with …) as needed.

_compact_max_rows(target_lines, halfblock) -> Int

How many grid rows fit in target_lines total output lines under the compact layout. With half-blocks each output line carries two grid rows.

_cooccurrence_counts(ps::PatternStats) -> (n1, n11)

Marginal (n1[a]: rows where column a is missing) and joint (n11[a, b]: rows where a and b are both missing) missingness counts, accumulated from the already-deduplicated pattern table — so the cost is O(npatterns * k^2) in the number of distinct patterns and missing columns per pattern, not O(nrows * ncols^2).

n11 is symmetric by construction (co-occurrence of a with b equals b with a), so the loop only visits the upper triangle (idxs comes out of findall sorted ascending) and mirrors into the lower one — half the work for the same result.

_count_missing(col) -> Int

Missing-value tally for one column. Same function-barrier pattern as _accumulate_column!: the caller pays dynamic dispatch once per column, and this body specializes on the column's concrete eltype, so the inner loop is scalar and allocation-free with no Union{T,Missing} boxing.

_data_cell!(buf, glyph, cell_chars, cw, prefix, suffix)

Writes one heatmap data cell directly to buf: padding, optional ANSI prefix/suffix, and the glyph repeated cell_chars times — with zero intermediate String allocations (compare to the original repeat(string(char), cell_chars) + double repeat(' ', pad) per cell). Since cw = max(cell_chars + 2, 9) by construction, cell_chars <= cw - 2 always holds, so (unlike _cell!) no truncation branch is needed here.

_diff_counts(before_col, after_col) -> (resolved, introduced)

Row-aligned pass over one column of both tables: resolved counts cells missing before but present after (e.g. imputed), introduced the reverse. Same function-barrier pattern as every other hot loop in the package.

_diff_rgb(delta, worse, better) -> NTuple{3,Int}

Signed variant of _ramp_rgb: zero delta is neutral dark gray; positive deltas (more missing after) blend toward worse, negative (holes filled) toward better — with the same ~30% + √ visibility floor, so even a single changed cell inside a huge block produces a visible tint.

_drop_path(ps::PatternStats) -> (dropped::Vector{Int}, complete::Vector{Int})

Greedy column-dropping path. complete[k] is the number of complete rows after the first k - 1 drops (so complete[1] is the untouched complete-case count), and dropped[k] is the column removed at step k.

At each step the column removed is the one that turns the most rows complete — i.e. the one that is the sole remaining missing column of the heaviest set of patterns. When no single column completes another row, the tie falls to the column with the most missing values (the one most likely to pay off once a later drop joins it), then to the lowest index, so the path is deterministic.

The walk stops as soon as every row is complete, or when one column is left: past either point dropping more columns only discards information.

_glyph_prefix(style, prop) -> String

ANSI foreground prefix for a colored glyph cell (classic layout and pattern table). Under :missing emphasis, fully-present cells keep the terminal's default color (historic behavior); under :present emphasis every cell is colored, since present data is exactly what carries the ink.

_halfblock_cell!(buf, fg, bg, cell_chars, cw, rst)

One compact data cell: cell_chars copies of '▀' whose ANSI foreground is the RGB tuple fg (top grid row) and background bg (bottom grid row). Color semantics live entirely with the caller, so the same primitive serves plotmissing (missingness ramp) and plotmissingdiff (signed delta colors). bg may be nothing when the grid has an odd number of rows and this is the final, unpaired line — the bottom half then keeps the terminal background.

_isna_for(isna, colname::Symbol)

Resolve the isna argument down to the predicate for one column.

A bare predicate applies to every column. A NamedTuple or AbstractDict maps column names to predicates, with ismissing for any column it omits — necessary because a sentinel is a property of a variable, not of a table: 9 means "ignored" in a coded field but is a perfectly good age.

Resolution happens once per column, on the dispatch-heavy side of the function barrier, so the predicate reaching the inner loop is still a concrete type the loop specializes on.

_jaccard(nab, na, nb) -> Float64

Jaccard index of the two missingness masks: |A ∩ B| / |A ∪ B|. NaN when neither column has any missing value (empty union).

_make_render_style(io; cell_chars, char_missing, char_present, name_width,
                    color_cells, show_row_range=false, row_labels=String[])

Builds a RenderStyle up front (border strings, cell width, ANSI codes). Deliberately decoupled from MissingGridStats — it only needs row_labels to size the optional row-range column — so it can be reused by any renderer in this package (currently the heatmap grid and the pattern table), not just plotmissing.

_missing_per_row_hist(cols, nrows, ncols) -> Vector{Int}

Histogram of "missing values in this row", as a ncols + 1 element vector indexed by count + 1 (so hist[1] is the number of fully complete rows). Costs one O(nrows) Int buffer and a single pass per column.

_pair_row_label(stats, top, bot) -> String

Row-range (or period-range) label spanning the two grid rows folded into one half-block line: the low endpoint of top joined to the high endpoint of bot. Works uniformly for positional row indices and temporal group labels.

_parse_hex(s) -> NTuple{3,Int}

Parse "#rrggbb" (leading # optional) into an RGB tuple.

_phi(nab, na, nb, n) -> Float64

ϕ (mean square contingency) coefficient of the two binary missingness indicators, from the 2x2 table implied by nab/na/nb/n. NaN when either column is entirely missing or entirely present (the denominator vanishes — the coefficient is genuinely undefined, not zero).

The products are taken in Float64: na*(n-na)*nb*(n-nb) overflows Int64 around n ~ 10^5, which is an ordinary table size here.

_ramp_rgb(ramp, prop) -> NTuple{3,Int}

Map a block's missing fraction prop to an RGB color.

Downsampling-fidelity guarantee, in both emphasis modes: any prop > 0 gets a minimum blend of ~30% away from the fully-present color (with a square-root scale below that), so a single missing value averaged over thousands of rows still produces a visibly different shade. Small holes never vanish under compression.

• emphasis == :present: prop == 0 → full target color; increasing missingness darkens toward base (holes = dark gaps in a colored field).
• emphasis == :missing: prop == 0 → base; increasing missingness brightens toward target.

_seriate_columns(M, n1) -> Vector{Int}

Greedy nearest-neighbour seriation of the columns under the association matrix M (ϕ of the missingness masks): start at the column with the most missing values, then repeatedly append the unplaced column most associated with the one just placed. This is the cheap, deterministic ordering that makes a block structure legible — columns that vanish together end up adjacent — without pulling in a clustering dependency for what is a one-dimensional layout problem.

Columns with no missing values at all are held out of the walk and appended at the end in table order: they have no missingness pattern to sit next to, and ϕ against them is undefined, so letting them compete would park a blank column in the middle of the very block structure the ordering exists to expose.

Among the remaining columns NaN counts as zero association. Ties break on the larger missing count, then on the lower column index, so the result never depends on iteration order.

_table_info(tbl) -> (cols, colnames::Vector{String}, nrows, ncols)

Resolve any Tables.jl-compatible source to a column-accessible object plus its dimensions. This is the single entry point through which every public function consumes data, so the package works identically for DataFrames, CSV.File, NamedTuples of vectors, XLSX.gettable results, etc.

_use_color(io::IO) -> Bool

Canonical ecosystem convention for color-aware output: defer to the :color property of io via get(io, :color, false). On Julia >= 1.11 this already resolves correctly for a raw Base.TTY (terminfo-based detection is wired into Base.get for TTY). For older Julia versions (down to the package's 1.6 floor) a raw, unwrapped TTY doesn't carry that information yet, so we conservatively fall back to io isa Base.TTY. Callers who want to force (or suppress) color regardless of io's concrete type should wrap it explicitly, e.g. IOContext(io, :color => true) — exactly as any other Base/ecosystem show-like function expects.

compute_cooccurrence(tbl; method=:phi) -> (M, colnames, n_missing_per_col, nrows)

Pairwise association between the missingness masks of every pair of columns — the correlation-style answer to the question missingpatterns answers by enumeration: which columns go missing together?

Built on top of compute_pattern_stats: since each unique pattern already carries its row count, the pairwise tallies cost O(npatterns × k²) (k = missing columns per pattern) instead of O(nrows × ncols²) — for real data with few distinct patterns this is essentially free.

Methods:

• :phi — Pearson's ϕ coefficient of the two binary masks, in [-1, 1]. Positive: the columns tend to be missing together (rarely MCAR!); negative: their missingness repels.
• :jaccard — |A ∩ B| / |A ∪ B| of the missing-row sets, in [0, 1].

Degenerate pairs (a column with zero or all-missing rows) yield NaN.

compute_missing_stats(tbl; max_rows, max_cols) -> MissingGridStats

Compute all display-independent statistics for a Tables.jl-compatible tbl in a single pass per column, without ever materializing an nrows × ncols missing-value matrix. Memory footprint is O(dr*dc + dc + dr) — bounded by the display size (max_rows × max_cols), not by the data itself. Row-range labels are always built (they are at most max_rows tiny strings).

compute_missing_stats_grouped(tbl, by, period=nothing; max_rows, max_cols) -> MissingGridStats

Like compute_missing_stats, but rows are grouped by the by column's values instead of by position. Two modes:

• period is :year, :quarter, :month, :week or :day: by must hold Date/DateTime values, and rows are grouped by the calendar period they fall in (groups sorted chronologically). :week follows ISO-8601, so the label carries the ISO week-year, which at the turn of the year can differ from the calendar year (2024-12-30 is 2025-W01).
• period === nothing (default): by is treated as a categorical column — rows are grouped by exact value (groups sorted with isless, so any Real, AbstractString, Symbol, etc. column works).

In both modes, rows with a missing by value form a trailing ∅ group. When there are more groups than max_rows, consecutive groups (in sorted order) are merged into one block and labeled as a range (e.g. 2004-2005 for periods, A-C for categories), with proportions weighted by each group's true row count — so unequal-sized groups never distort the picture.

compute_pattern_stats(tbl; isna=ismissing) -> PatternStats

Compute the unique row-wise missingness patterns of a Tables.jl-compatible tbl and their frequencies, sorted most-common first. isna is the "counts as absent" predicate (see plotmissing).

render_grid_compact!(buf, stats, style; halfblock)

Compact grid: condensed chrome always; half-block vertical doubling when halfblock (requires style.use_color), classic glyphs otherwise.

render_pattern_table!(buf, stats, style, max_patterns;
                      show_bar=true, min_pct=0.0) -> (shown, nkept)

Draws the pattern table for stats, reusing the exact same border/cell primitives as render_grid! — one row per unique missingness pattern (already sorted most-common first), one column per variable plus trailing n/% columns and, when show_bar, an UpSet-style horizontal frequency bar scaled to the most common displayed pattern. Patterns whose relative frequency is below min_pct (percent of rows) are filtered out before the max_patterns cap is applied. Returns (shown, nkept): how many patterns were rendered and how many survived the min_pct filter, so the caller can report both kinds of omission.

render_summary_compact!(buf, stats, style)

Single-line summary carrying the same information as render_summary! (dimensions, compression ratio, missing/present counts and percentages).