"UnboundLocalError: counts_per_cols" — What Disabling Numba Actually Broke in scanpy

scanpy · normalize_total · Numba · UnboundLocalError · reproducible builds

"UnboundLocalError: counts_per_cols" — What Disabling Numba Actually Broke in scanpy

Status as of writing: present in scanpy 1.11.5 (and versions around it), fixed via PR #4231 (merged 2026-07-27), shipped in 1.12.4 (2026-08-27). Reported in scanpy#4228.

If you disable Numba JIT — a common workaround when a container or CI environment can't write Numba's cache files — sc.pp.normalize_total() could crash outright on CSR matrices with the (default) setting exclude_highly_expressed=False:

NUMBA_DISABLE_JIT=1 python - <<'PY'
import numpy as np
from scipy.sparse import csr_matrix
from anndata import AnnData
import scanpy as sc

adata = AnnData(csr_matrix(np.array([[1, 2, 0], [0, 3, 4]], dtype=np.int32)))
sc.pp.normalize_total(adata, target_sum=1e4)
PY

# UnboundLocalError: cannot access local variable 'counts_per_cols'
# where it is not associated with a value

Why disabling Numba even matters

This isn't an exotic setup. People turn off Numba JIT (via NUMBA_DISABLE_JIT=1) for a few ordinary reasons:

• The runtime environment can't write to Numba's cache directory (common on locked-down CI runners or read-only containers)
• Debugging — JIT-compiled functions are harder to step through
• Cold-start latency in short-lived jobs where JIT compilation overhead isn't worth it

The actual bug

The CSR-specific fast path, _normalize_csr, only initialized counts_per_cols inside the if exclude_highly_expressed: block, but returned it unconditionally at the end of the function. With Numba's JIT active, this quirk of the compiled code path apparently never surfaced the same way; with JIT disabled, plain Python's stricter local-variable scoping raised the UnboundLocalError the moment the default-off branch was taken.

The fix (confirmed by reading the current source) simply initializes counts_per_cols = np.zeros(columns, dtype=np.int32) unconditionally before the branch, so it always has a value to return:

# current scanpy main, _normalization.py
def _normalize_csr(mat, rows, columns, exclude_highly_expressed=False, ...):
    counts_per_cols = np.zeros(columns, dtype=np.int32)   # <- now unconditional
    ...
    if exclude_highly_expressed:
        counts_per_cols_t = np.zeros((n_threads, columns), dtype=np.int32)
        ...
    return counts_per_cell, counts_per_cols

What to do

Upgrade to scanpy ≥1.12.4. If you're stuck on an older pinned version and need Numba disabled, the workaround is to explicitly pass exclude_highly_expressed=True even if you don't need the exclusion logic — it forces the code path that does initialize the variable, sidestepping the crash (at the cost of a slightly different internal computation you should verify doesn't change your output if you truly don't want exclusion).

Verified against scanpy's main branch source and PR #4231, 2026-09. Source: scanpy#4228.

A Year-Old scanpy Bug That Could Silently Zero Out Your Data — Fixed Last Month

scanpy · normalize_total · silent data loss · CSR vs dense · preprocessing bug

A Year-Old scanpy Bug That Could Silently Zero Out Your Data — Fixed Last Month

We've written before about how easy it is to accidentally double-log-normalize spatial data (it happened across our own corpus — see our earlier workflow post). This one is a different bug in the same function, and it's worse: it can silently zero out your entire matrix with no error, just two easy-to-miss RuntimeWarnings.

Status as of writing: present in scanpy 1.11.2 through 1.12.3 (2025-05-28 → 2026-07-24). Fixed in 1.12.4 (released 2026-08-27) via PR #4256, reported in scanpy#4251. If you pinned scanpy any time in the last 14 months, check your version.

What target_sum=None is supposed to mean

sc.pp.normalize_total(adata, target_sum=None) — the default most tutorials use — documents its target as "the median of total counts for observations." Simple enough. But _normalize_total_helper computes that median two different ways depending on whether your matrix is CSR sparse or dense:

# CSR path (scanpy/preprocessing/_normalization.py, pre-1.12.4)
if target_sum is None:
    target_sum = np.median(counts_per_cell)            # median over ALL cells

# dense / everything-else path
if target_sum is None:
    target_sum = _compute_nnz_median(counts_per_cell)   # median over POSITIVE cells only

The two only disagree when a zero-total cell shifts the median — with row sums like [0, 10, 10, 20] both give 10, which is probably why this went unnoticed for so long. Empty cells are common in practice: an unfiltered raw matrix, an aggressive gene subset, a .raw slice with different filtering than the parent object.

The severe case: when at least half your cells are empty

If enough cells have zero total counts, the CSR branch's median lands on 0. Dividing by zero doesn't error — it silently produces an all-zero matrix, with only the warnings below as evidence anything went wrong:

import anndata as ad
import numpy as np
import scanpy as sc
import scipy.sparse as sp

# row sums [0, 0, 10] — the only populated cell has counts [4, 6]
dense2 = np.array([[0.0, 0.0], [0.0, 0.0], [4.0, 6.0]], dtype=np.float32)

for label, X in (("dense", dense2.copy()), ("csr", sp.csr_matrix(dense2))):
    a = ad.AnnData(X=X)
    sc.pp.normalize_total(a, target_sum=None)
    out = a.X if isinstance(a.X, np.ndarray) else a.X.toarray()
    print(f"{label:>5}: {out.ravel()}")

# dense: [0. 0.  0. 0.  4. 6.]   <- correct, target=10, populated cell untouched
#   csr: [0. 0.  0. 0.  0. 0.]   <- target=0, the ONLY real data point is destroyed

.../scanpy/preprocessing/_normalization.py:119: RuntimeWarning: divide by zero encountered in divide
.../scanpy/preprocessing/_normalization.py:119: RuntimeWarning: invalid value encountered in divide

No exception. No crash. Just a warning easy to filter out in a notebook with dozens of other warnings, and a matrix that's now all zeros.

Why the CSR path had a different rule at all

Before 1.11.2 there was a single code path, and it excluded zero-count cells on both branches. The split arrived with the numba CSR kernel in PR #3571 (merged 2025-05-19), which kept the old positive-cell-median rule for the non-CSR branch but gave the new CSR branch a plain np.median. Neither that PR nor its release notes mention a semantic change — this reads as an oversight, not a decision, and the reporter's one-line fix (route the CSR branch through the same _compute_nnz_median helper) is exactly what landed in 1.12.4.

What to actually do

• Upgrade to scanpy ≥1.12.4 if you can — this is the real fix.
• If you're pinned to an older version, pass an explicit target_sum (e.g. target_sum=1e4) instead of relying on the median default — this sidesteps the bug entirely regardless of version.
• Before normalizing, check (adata.X.sum(axis=1) == 0).sum() — if it's more than a small fraction of your cells, filter them out first rather than letting either branch guess a target around them.

Verified directly against scanpy's main branch source and the released 1.12.4 tag, 2026-09. Source: scanpy#4251.