Fix and refactor create_irma_vcf.py

[PauPascualMas]

PR Description

This PR improves IRMA-to-VCF conversion, with a focus on indel normalization, variant-filtering consistency, malformed-input handling, and VCF-safe output.

The previous implementation could discard contextual allele rows too early, fail while processing missing numeric values, produce invalid or incomplete indel representations when reference anchors were unavailable, and behave unpredictably with malformed alignment or allAlleles inputs.

Changes

Correct variant-filtering logic

Variant retention now follows the documented condition:

(AF >= min_freq AND total depth >= total_depth)
OR
(allele depth >= alt_depth)

Numeric conversion is handled safely. Missing values such as NA, ., empty strings, and non-numeric values no longer satisfy numeric filter conditions or cause comparison errors.

Preserve context required for indel normalization

All parseable allAlleles rows are retained during initial parsing.

Filtering is applied later, after allele information has been merged with the alignment. This preserves reference and low-frequency rows needed to identify the reference bases used to anchor insertions and deletions.

Improve insertion normalization

• Assemble insertions before reference position 1 in sample-coordinate order.
• Normalize initial insertions using reference position 1 as the required VCF padding base.
• Verify that an anchor base exists before normalizing internal insertions.
• Select minority insertions using numeric allele-frequency and quality comparisons.
• Avoid relying on dictionary insertion order when assembling multi-base initial insertions.
• Improve deletion normalization
• Validate the availability of the preceding or following reference base required for VCF normalization.
• Preserve separate handling for deletions at the beginning of the sequence.
• Continue merging adjacent consensus and minority deletion bases when their coordinates are compatible.
• Skip unnormalizable deletions explicitly rather than emitting malformed VCF records.
• Improve error and warning handling

The script now:

• Detects empty alignments.
• Rejects alignments containing one or more than two sequences.
• Detects duplicate FASTA record IDs.
• Verifies that aligned sequences have equal lengths.
• Detects missing input files.
• Rejects empty or unparseable allAlleles input.
• Handles incomplete wrapped records without reading indefinitely at EOF.
• Warns when an indel cannot be normalized because its reference context is unavailable.
• Reports the total number of skipped indel-normalization events.
• Produce safer VCF output
• Ignore missing values when calculating mean depth, quality, and frequency.
• Emit . when no numeric value is available.
• Omit DP from INFO when total depth is unavailable.
• Preserve the consensus flag only for variants represented in the consensus.
• Continue sorting variants by reference position and type before writing the VCF.

Motivation

IRMA represents variants using a combination of consensus alignments and allAlleles statistics. Insertions and deletions therefore require positional context that may be stored in rows which do not themselves pass the final reporting thresholds.

Applying filters during parsing could remove that context and prevent valid indels from being normalized. The revised workflow separates parsing, alignment merging, normalization, and final filtering more clearly and handles missing or malformed values defensively.

PR checklist

• This comment contains a description of changes (with reason).
• Make sure your code lints (black and flake8).
• If a new tamplate was added make sure:
  • Template's schema is added in templates/services.json.
  • Template's pipeline's documentation in assets/reports/md/template.md is added.
  • Results Documentation in assets/reports/results/template.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).
• If you know a new user was added to the SFTP, make sure you added it to templates/sftp_user.json

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72e6bf3d47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".