NEBULA3D

NEBULA3D — Neutron Elastic Background Utility for Local Analysis and 3D-ΔPDF — is a Python toolkit (the nebula3d package) for cleaning 3D reciprocal-space neutron diffuse scattering volumes and preparing them for 3D-ΔPDF analysis.

The current workflow is built around symmetrised Mantid HKL volumes. It removes powder-ring backgrounds, punches sharp Bragg and satellite peaks, fills the punched holes with a diffuse-background estimate, Fourier-transforms the cleaned volume into a real-space 3D-ΔPDF, ends with a back-FFT consistency check, and can hand the result to an AI reasoning review — a local or cloud LLM that grades the reduction from metrics computed in the browser.

Mantid / symmetrised HKL volume
        |
        v
  1. powder-ring subtraction        examples/remove_rings_3d.py
  2. Bragg/satellite punch          examples/punch_bragg_3d.py
  3. Bragg-hole backfill            examples/backfill_bragg_3d.py
  4. radial-background flatten      examples/flatten_background_3d.py
  5. 3D-ΔPDF transform              examples/delta_pdf.py
  6. back-FFT consistency check     examples/delta_pdf_consistency.py
  7. cleanup / ΔPDF viewers         examples/explore_slice.py, examples/explore_delta_pdf_ortho.py
  8. AI reasoning review (optional) web AI Assistant — local or cloud LLM

For a 3D-PDF (total scattering with the Bragg peaks kept — a Patterson-like map) instead of the ΔPDF, use examples/run_pipeline_pdf.py, which skips the punch and backfill stages. To characterise magnetic diffuse that sits at the Bragg satellites, use examples/investigate_bragg_diffuse.py.

Web UI

nebula3d ships a single browser console (React) that unifies the cleanup and 3D-ΔPDF viewers and drives the whole pipeline from a raw .nxs with live progress. It has two interchangeable run modes that share the same UI and the same reduction code:

• In-browser (no install) — the hosted app at https://drthyang.github.io/nebula3d/ is a fully static GitHub Pages bundle that runs the complete nebula3d pipeline client-side via Pyodide, at full-resolution float64 (up to ~50 M voxels — a 301×401×401 volume fits). Load your own .nxs/.h5; nothing is uploaded, nothing is server-side.
• Native — pip install -e ".[web]" then nebula3d-web (serves http://127.0.0.1:8000); the same UI backed by FastAPI for local work with no size limit.

The browser build has full feature parity with the native backend — the static, hosted app is a first-class way to run nebula3d, not a reduced demo.

A sidebar console drives everything from one place — a global dataset switcher, a pipeline runner (the default landing view), reciprocal-space cleanup, a Bragg profile view, 3D-ΔPDF orthoslices, and the back-FFT consistency check — plus an AI Assistant that connects to your selected LLM, local (Ollama / LM Studio) or cloud (OpenAI / Gemini), and reasons over the reduction — grading ring removal, the Bragg punch, backfill, and the ΔPDF from metrics computed in the browser (nothing leaves your machine but the chat call to your own model server). See QUICKSTART.md to get running and docs/web.md for the reference, architecture, and development workflow.

Install

Requires Python 3.10 or newer.

git clone https://github.com/drthyang/nebula3d
cd nebula3d
pip install -e ".[dev]"

For local scripts run from the repository root, set:

export PYTHONPATH=src
export MPLCONFIGDIR=/tmp/mpl

MPLCONFIGDIR keeps Matplotlib cache files out of the repository.

Development

Before pushing, run the same checks as CI (tests, lint, type check):

bash scripts/check.sh

It runs pytest, ruff check src/ tests/, and mypy src/nebula3d. Set PY=/path/to/python to choose the interpreter. To run it automatically on every push, install it as a git hook in your clone:

ln -s ../../scripts/check.sh .git/hooks/pre-push

Input Data

Place a Mantid-exported NeXus file in data/raw/. Either variant works:

*_cc_sub_bkg.nxs   # correlation chopper, empty-can background subtracted
*_cc_data.nxs      # correlation chopper data

Here cc means CORELLI correlation chopper, and sub_bkg means the empty-can background has already been subtracted. Correlation-chopper (cc) and properly symmetrised data is recommended. You can also load nebula3d HDF5 files written by the package itself.

Quick Start

To run the app, see QUICKSTART.md. For concise CLI command recipes for batch workflows and viewers, see docs/commands.md.

Run the complete pipeline:

DATA_FILE=data/raw/your_volume_cc_sub_bkg.nxs \
PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl \
python3 examples/run_pipeline.py

examples/run_pipeline.py runs all compute stages, skips stages whose outputs already exist, writes the 3D-ΔPDF, runs the consistency check, then opens the cleanup and ΔPDF viewers. The stages are: (1) ring removal, (2) Bragg punch, (3) Bragg backfill, (4) radial-background flatten — the explicit background-removal step (default on; FLATTEN=0 to skip), (5) 3D-ΔPDF FFT, (6) back-FFT consistency check. The background is removed at step 4, not by a hidden blur inside the FFT: the transform's own Gaussian SUBTRACT_BG is off by default because it is the alternative remover (see step 5 below).

Useful overrides:

Variable	Effect
DATA_FILE=/path/to/file.nxs	Use a specific input file.
NO_VIEWER=1	Stop after writing outputs and the consistency check; do not open GUI viewers.
FORCE=1	Recompute every stage.
`FORCE_FROM=rings	punch
FLATTEN=0	Skip the radial-background flatten (step 4).
CONSISTENCY=0	Skip the final back-FFT consistency check.

Run Stages Manually

Run these from the repository root when you want to inspect or tune individual stages.

1. Remove Powder Rings

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl RING_PRESET=cc_on \
python3 examples/remove_rings_3d.py

Output:

data/processed/*_ringremoved.h5

2. Punch Bragg And Satellite Peaks

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl PUNCH_PRESET=cc_on MODE=both \
MIN_I=0.8 MIN_PROM=0.8 INTEGER_FIT_POSITION=1 INTEGER_FIT_SHAPE=1 \
INTEGER_H_GUARD=0.12 \
SEARCH_EXCLUDE_H=-0.6667,-0.3333,0.3333,0.6667 SEARCH_EXCLUDE_H_WIDTH=0.08 \
PREVIEW=0 \
python3 examples/punch_bragg_3d.py

Output:

data/processed/*_braggpunched.h5

3. Backfill Bragg Holes

PYTHONPATH=src METHOD=q_shell \
python3 examples/backfill_bragg_3d.py

Output:

data/processed/*_braggpunched_backfilled.h5

4. Flatten The Radial Background (Background Removal)

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl \
python3 examples/flatten_background_3d.py

Output:

data/processed/*_backfilled_flattened.h5

Sweeps spherical |Q| shells and subtracts a smooth, continuous per-shell background floor (default estimator floor/p25), so the isotropic radial pedestal flattens to ≈0 while the anisotropic diffuse and Bragg residuals are preserved. This is the explicit background-removal step. Use examples/validate_flatten.py to check isotropy, feature retention, and over-subtraction on your own volumes.

5. Compute The 3D-ΔPDF

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl APODIZE=gaussian \
python3 examples/delta_pdf.py

Outputs:

examples/_delta_pdf.h5
examples/_delta_pdf_hk0.png
examples/_delta_pdf_h0l.png
examples/_delta_pdf_0kl.png

By default the transform uses the full |Q| range (no crop), which gives the finest real-space ΔPDF — the real-space pixel size is fixed by the |Q| extent kept, not by zero-padding (see docs/algorithms/delta_pdf.md). This matches the back-FFT consistency view, which always uses the full range. CROP_H=4 CROP_K=8 CROP_L=15 is an optional band-limit (smaller transform, trims the noisier outer |Q| shells at the cost of a coarser real-space grid). The background has already been removed at step 4, so the transform's own smooth-background subtraction is off. SUBTRACT_BG=0,sigma,sigma is the legacy alternative: it subtracts a smooth per-H-plane Gaussian background inside the FFT. Use the flatten or SUBTRACT_BG, never both — running both removes the background twice, and the per-H-plane blur (σ_H=0) destroys the on-axis H-direction signal that the flatten preserves.

6. Check ΔPDF Consistency

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl \
DATA_FILE=data/processed/your_volume_ringremoved_braggpunched_backfilled_flattened.h5 \
python3 examples/delta_pdf_consistency.py

The check computes the ΔPDF with the standard settings, inverse-transforms it back to reciprocal space, and compares data | back-FFT | residual. A faithful transform gives Pearson r close to 1 with a small normalised RMS residual. The web UI exposes the same check interactively, including adjustable |Q| and real-space r bands.

Visual QA

Inspect cleanup across H:

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl USE_BACKGROUND=0 \
PUNCH_PRESET=cc_on MODE=both MIN_I=0.8 MIN_PROM=0.8 \
INTEGER_FIT_POSITION=1 INTEGER_FIT_SHAPE=1 INTEGER_H_GUARD=0.12 \
SEARCH_EXCLUDE_H=-0.6667,-0.3333,0.3333,0.6667 SEARCH_EXCLUDE_H_WIDTH=0.08 \
BACKFILL_METHOD=q_shell H_VALUE=0.3333 \
python3 examples/explore_slice.py

The viewer shows raw data, removed ring intensity, punched data, and backfilled data, with an H slider for scrubbing through the volume.

Inspect the real-space 3D-ΔPDF:

PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl RMAX=50 \
python3 examples/explore_delta_pdf_ortho.py

This opens three linked orthogonal real-space cuts with movable cut sliders, contrast control, and unit-cell gridlines.

Python API

import nebula3d
from nebula3d.analysis import BraggRemover, backfill_bragg, compute_delta_pdf

vol = nebula3d.load("data/processed/sample_ringremoved.h5")

remover = BraggRemover(
    mode="both",
    punch_radii=(0.09, 0.12, 0.45),
    min_intensity=0.8,
    min_prominence=0.8,
    integer_optimize_position=True,
    integer_optimize_shape=True,
    integer_h_guard_hkl=0.12,
    search_n_mad=4.0,
    search_min_intensity=0.8,
    search_min_prominence=0.8,
    search_exclude_h_centers=(-2 / 3, -1 / 3, 1 / 3, 2 / 3),
    search_exclude_h_half_width=0.08,
    incident_beam_ellipsoid_radii_hkl=(0.15, 0.50, 1.00),
)

punched = remover.apply(vol)
filled = backfill_bragg(punched, method="q_shell")
dpdf = compute_delta_pdf(filled, apodization="hann")

Documentation

Start with docs/README.md.

Key pages:

Page	Purpose
docs/algorithms/powder_rings.md	Powder-ring model and subtraction strategy.
docs/algorithms/bragg_cleanup.md	Bragg/satellite detection, punching, and backfill.
docs/algorithms/delta_pdf.md	3D-ΔPDF transform, centring, background subtraction, and consistency checks.
QUICKSTART.md	Get the app running (native or in-browser) in a few commands.
docs/commands.md	Concise CLI command recipes for batch workflows and viewers.
docs/web.md	Browser console: run modes, viewers, architecture, dev workflow.
docs/interactive.md	Matplotlib viewer usage and visualization API.
CHANGELOG.md	Release notes and version history.

Package Layout

src/nebula3d/
├── core.py              HKLVolume: 3D array, HKL axes, mask, sigma, UB matrix
├── io/                  Mantid NeXus, nebula3d HDF5, and ASCII HKL I/O
├── preprocessing/       powder-ring models, background handling, sampling
├── analysis/            Bragg punch/fill and 3D-ΔPDF
├── inpainting/          symmetry, TV, RBF, and biharmonic fallbacks
└── visualization/       slices, profiles, overview plots, interactive viewers

Tests And CI

Run locally in a Python 3.10+ environment with dev dependencies:

PYTHONPATH=src python3 -m pytest -o addopts=''
python3 -m ruff check src/ tests/
python3 -m mypy src/nebula3d --ignore-missing-imports

GitHub Actions runs the same checks on Python 3.10, 3.11, and 3.12.

Status

Version 0.3.0 (beta). The recommended workflow is operational and ends with the back-FFT consistency check: powder-ring removal, Bragg cleanup, Bragg-hole backfill, radial flatten, 3D-ΔPDF transform, consistency QA, and interactive viewers. The complete pipeline also runs fully client-side in the static GitHub Pages app, at full-resolution float64 with feature parity to the native backend, and the browser console now includes an AI Assistant that grades the reduction from in-browser metrics. The package remains pre-1.0/beta while the public API and file formats continue to evolve.

License and provenance

Copyright (C) 2026 Tsung-han Yang.

This project is licensed under the GNU Affero General Public License v3.0 or later.

This is an independent personal software project. It is not affiliated with, sponsored by, endorsed by, funded by, developed for, or owned by any employer, institution, laboratory, university, government agency, funding organization, facility, or other organization.

Commercial, closed-source, proprietary, or alternative licensing may be available only by separate written permission from the copyright holder.

See PROVENANCE.md for the project provenance statement.