Support for Database Packaging

[isc-kiyer]

Feature Request

IPM should support packaging all document resources as a database (IRIS.DAT) file instead of source files/studio project.

Why do we want this?

A few reasons:

• This is in particular helpful when packaging up an entire application (not a reusable libraru)
• No compilation needed when installing the application. This is important when deploying in a customer environment to limit where things could go wrong.
• Swapping out an IRIS.DAT file makes for faster upgrades than compilation (lower down time)

Workflows

Packaging

Optionally flag based (e.g. package ModuleA -format database). Options for format: module, database, studio project (long term get rid of this as IPM becomes native to IRIS).

Considerations:

• Should we just package the existing IRIS.DAT or install in a fresh namespace and database to package that.
• What to do with the export-deps flag that currently exists?
• Keeping compatibility for any non document resources to behave as is today.
• What do with module.xml so that at install time, it knows its an IRIS.DAT file.
• IPM should be mapped and not installed in the current namespace or else you package up IPM itself as well.
• What to do with data? What data should be packaged (e.g. we want to keep IPM data so should map it to the routine db but should not package application data as it would wipe out existing app data on upgrade).

Installation

Developer Mode

Not supported for database packaged.

Non-developer Mode

Considerations:

• Warn user that IRIS.DAT file will be replaced
• Ensure IPM is mapped and not in current routine DB
• What to do with existing .DAT file? Likely rename and leave there by default. Have flags for cleanup or alternate commands.

Upgrades

Considerations:

• How to manage IPM update infrastructure persistent tables.

Storage

We need to carefully think about how storage of IPM should be handled:

• Module info: Need this to be available in routine database always and mapped accordingly in the namespace
• Repo info: Always keep in globals DB
• Update step info: Need this to be in globals database so what steps have been run don't get wiped out when routine DB is swapped out
• History log info: Always keep in globals DB

Think about what happens with different mapped storage when certain databases are mirrored but others aren't (e.g. globals db is mirrored and routine db is not)

[isc-dchui]

DATABASE PACKAGING

Background

Currently packaging of IPM modules occurs either as source files or as a studio project. The former is the default while the latter is useful for deployed code and installing in systems without IPM. However, these approaches still face limitations that database packaging would solve. Database packaging is the idea of packaging the entirety of a module and its dependencies into an IRIS.DAT file and delivering that file for installation. The benefits include but are not limited to:

• Packaging an entire application is much simpler
• No compilation on installation
• Faster upgrades through swapping out an IRIS.DAT file
• IPM is not required on the installation machine

Design

Overview

Add a new option to the package command to bundle up an entire module in an IRIS.DAT to function as the routines database. This database should not contain IPM but should keep some of the IPM data, e.g. module info. When installing, swapping the old routines database for a new one should "just work" with or without IPM installed in the namespace.

Resource Handling

Code Resources

Classes, routines, includes, globals, and documents are all compiled into the IRIS.DAT file. These are standard InterSystems resources that live in the routines database.

CSP Files & Static Resources

• CSP files and static web assets (JS, CSS, images) are packaged separately alongside IRIS.DAT in a csp/ subdirectory
• The package command will programatically modify the module.xml to add a <FileCopy> to move the resources to the correct location
• All <FileCopy> will occur as usual

Python Dependencies

• Python wheels can shipped in a wheels/ subdirectory alongside IRIS.DAT
• Installed during the Initialize phase using the existing PythonWheel resource processor
• requirements.txt recommended for most Python dependencies
• In general, will leverage existing IPM functionality

Test Resources

• Default: Test resources (Scope="test" or Scope="verify") are excluded from database packages
• Optional: Use -include-test-resources flag to include test resources for validation purposes
• Test resources are typically only needed during development and CI/CD, not production deployment

Generated Resources

• Resources marked Generated="true" are included in database packages
• They're already compiled and part of the application, so no need to regenerate

Other Resource Files

- E.g. CPF files
- Copy them over as needed and make sure they work

Platform Compatibility

Cross-Platform Support

IRIS.DAT files are designed to work across Windows, Linux, and macOS. This enables packaging once and deploying to multiple platforms.

Endian Conversion

• Use the cvendian utility when needed to convert byte order between different architectures
• The utility can detect and report on current byte order or perform conversion

IRIS Version Compatibility

• module.xml should specify IRIS version requirements via <SystemRequirements>
• Installation validates running IRIS version against these requirements

Packaging

CLI Flags

Change package into a series of flags:
- package-source, alias package (default) - creates .tgz with source files
- package-database, creates IRIS.DAT database package (inside a .tgz with other files)
- package-studio-project) - creates Studio project (will be removed in IPM 1.0.0) Add an optional -include-test-resources` flag to include test resources in database package

• Excluded by default
  Add an optional -use-current-db flag that tries to package into current DB:
• Default is to create a fresh new IRIS.DAT in a temporary location
• With flag: Package into the namespace's current routines DB
• Validation: Check that IPM is not in current routines DB (must be externally mapped)

Interactions with Other Flags

• -export-deps flag will not be available for package-database
• -export-py-deps flag for package-database will default to True, but can be toggled off

Database Creation

Pre-validation

• Check that IPM classes do not exist in current routines DB
• If IPM found: ERROR "IPM must be mapped from another namespace"
• This ensures IPM doesn't get packaged with the module
  Default Approach (without -use-current-db):
• Create new empty IRIS.DAT in temporary location
• Pre-allocate reasonable size
• Unmount current routines database
• Mount this new database
• Load module resources into this new database
• Kill extent on %IPM data and map appropriate data
• Unmount new database
• Remount old routines database
  Current DB Approach (with -use-current-db):
• Package into the namespace's current routines DB
• Let user perform without warning or checking
• Unmount current DB and rename/copy for backup
• Remount current DB and write into it

Checksumming & Integrity

SHA-256 Checksum:

• Generate SHA-256 checksum of final IRIS.DAT file
• Embed in module.xml: <Checksum Algorithm="SHA-256">abcdef123456...</Checksum>
• Used during installation to verify file integrity before mounting

Package Output Structure

mymodule-1.0.0-database.tgz/
├── module.xml             # Manifest with metadata and checksum
├── dependencies.xml       # Compilation of module.xml's of dependencies
├── IRIS.DAT               # Compiled database (routines DB)
├── csp/                   # CSP files (if present)
│   └── app/
│       ├── index.html
│       └── styles.css
├── wheels/                # Python wheels (if present)
│   └── mypackage-1.0.0-py3-none-any.whl
└── other resource files

Changes to module.xml

• Automatically set <Packaging>database</Packaging>
  • If it is already set, then package will perform database packaging even without -package database
• Add SHA-256 checksum of IRIS.DAT with <Checksum Algorithm="SHA-256">abcdef123456...</Checksum>
  • Used during installation to verify file integrity before mounting
• Dependencies will have their module.xml's compiled into a single dependencies.xml file
  • This will contain all the metadata required for install and obviate the need to map/copy globals, avoiding the potential issue of differing IPM versions

Installation (with IPM)

Commands

• install, load, and update all work with database packages
• New flag -swap-db to prevent unintended database swap
  • Without this flag: Default to other packaging methods, require user confirmation for database packages
  • With this flag: Only use database packaged modules (error if none found), skip confirmation

Pre-Installation Validation

File Checks:

• Verify module.xml exists and is readable (more validation will happen during install)
• Verify IRIS.DAT file exists and size is reasonable (not zero)
  Checksum Validation:
• Read <Checksum Algorithm="SHA-256"> from module.xml
• Compute SHA-256 of IRIS.DAT file
• Compare checksums - error if mismatch (corrupted file)
  IPM Mapping Check:
• Verify IPM is mapped from another namespace (not in target routines DB)
• If IPM found in target routines DB: ERROR
• This prevents IPM from being wiped out by the swap

User Confirmation

Swap Warning (unless confirmation skipped with -swap-db):

Warning: This will unmount and replace the routines database for namespace PROD.
Current DB: C:\InterSystems\IRIS\mgr\PROD\IRIS.DAT
Backup will be created: IRIS_PROD_20260309120000.DAT
All code in this namespace will be replaced.
Continue? [y/N]:

Database Swap Procedure

Detailed swap steps:

1. Pre-Validation: Run all validation checks above
2. Unmount: Unmount current routines DB from namespace
3. Backup: Rename current IRIS.DAT to IRIS_<db-name>_<timestamp>.DAT
4. Copy: Copy new IRIS.DAT into place (original name)
5. Mount: Mount new IRIS.DAT as routines DB
6. Post-Validation: Run post-install checks (see below)
7. On Success: Update IPM metadata in globals DB
8. On Failure: Rollback (see Rollback section)

Backup Strategy

Automatic Backup:

• Rename: IRIS.DAT → IRIS_<db-name>_<timestamp>.DAT
• Location: Same directory as original database
• Format: IRIS_PROD_20260309143022.DAT (YYYYMMDDHHmmss)
• User manages deletion of old databases

Post-Install Validation

• Should only be doable in %SYS, but double check
• Validate by switching to the namespace
• Regular IPM validation will handle the rest

Rollback on Failure

If mount fails:

1. Delete new IRIS.DAT file
2. Rename backup DB back to original name: IRIS_<db-name>_<timestamp>.DAT → IRIS.DAT
3. Remount original database
4. Log all errors for troubleshooting
5. Return namespace to original state
   If validation fails:

• Same rollback procedure as mount failure
• Preserve error messages for debugging
  Rollback Confirmation:
• Display: "Installation failed. Rolling back to previous version..."
• On successful rollback: "Rollback complete. Namespace restored."

Update IPM Metadata

On successful installation:

• Update ^IPM.Storage.Module in globals DB with new module info
• Record installation in ^IPM.General.HistoryD
• Any info about the newly installed module or its dependencies will be available in module.xml or dependencies.xml

Developer Mode

Not Supported: Developer mode cannot be enabled for database-packaged modules. Error if attempted.

Installation (without IPM)

• Swap out old routines DB IRIS.DAT with new one
• Manually move CSP and Python wheels to appropriate locations
• Should be scriptable (out of scope for this)

Uninstallation

IPM has existing uninstall capabilities that should be leveraged.

• Uninstalling the database-packaged module should essentially leave the namespace as it was originally
  • Make sure this still works post-module.xml changes
• If source packaged modules are installed in this database, uninstalling the original should not remove them

Upgrades

• New database will bring along new update step classes
• Update steps run stored in globals DB but will need to be mapped to routines DB

Storage

What Goes Where

Globals Database (persists across swaps):

• Module metadata: ^IPM.Storage.Module, ^IPM.Storage.ModuleD
• Resource references: ^IPM.Storage.ResourceReference
• Module dependencies: ^IPM.Storage.Module_Dependencies
• Repository info: ^IPM.Repo.* (always in globals DB)
• Update step tracking: ^IPM.UpdateStep.AnyMember, ^IPM.UpdateStep.PrimaryOnly
• History logs: ^IPM.General.HistoryD (installation/uninstall/load history)
• Settings: ^IPM.General.SettingsD
  Routines Database (gets swapped):
• Classes: %Dictionary.* entries for all module classes
• Routines: Standard routine storage (MAC, INT files)
• Includes: ^rINC global for include files
• User-defined globals: Module-specific globals

Metadata Mapping Strategy

• The important information is module metadata and update step tracking

  • Module metadata is (re)created from module.xml and dependencies.xml
  • Update step tracking should be mapped over to packaged database so info will be available when installed

• When installing, map module storage to routines database

• When upgrading,

  • For non-mirrored instances, also map update steps to routines database
  • For mirrored/run only on primary update steps, only need to run where needed
    ○ Also let IPM know to not run them elsewhere (write into globals DB maybe?)
    ○ After upgrade, can just leave mappings
    ○ Doesn't work if routines DB is mapped to multiple NS's :(

IPM External Requirement

IPM itself (%IPM.* classes) must not be in the routines DB being swapped:

• IPM must be mapped from %SYS or another namespace
• This ensures IPM survives database swaps
• Validated during packaging and installation

Mirroring

Configuration Assumptions

• Globals DB: Mirrored (metadata syncs between members)
• Routines DB: May or may not be mirrored
  Upgrade Sequence:

1. Install on backup first
2. Test backup thoroughly
3. Failover from primary to backup
4. Install on new backup (former primary)
   Metadata Synchronization:

• Installation on primary updates globals DB
• Metadata change propagates to secondary via mirroring
• Update isn't complete until primary and backup are updated
  • When backup updates, may need to store IPM related metadata temporarily in temp database
    Product Specifics
• IRIS for Health/HealthShare has some special routines for handling mirroring?
• Regular IRIS will need to handle it in %SYS

Testing

Test Case Categories

These test cases will be bundled into test methods (see "Test Methods" section below).

Packaging:

• Package simple module (just classes)
• Package with dependencies (direct and transitive)
• Package with all resource types (classes, routines, includes, globals, CSP, Python wheels, CPF, FileCopy, generated resources)
• Package with/without test resources (-include-test-resources flag)
• Package with -use-current-db flag
• Package with/without -export-py-deps flag
• Packaging validation: IPM in routines DB (error), IPM properly mapped (success)
• Output verification: .tgz structure, IRIS.DAT created, checksums, module.xml updates, dependencies.xml

Installation:

• Install via install command with -swap-db
• Install via load command with -swap-db
• Pre-installation validation: corrupted/missing/zero-sized IRIS.DAT, invalid module.xml, missing dependencies.xml (all should fail)
• IPM validation: IPM in target routines DB (should error)
• Database swap: backup creation, IRIS.DAT replacement
• Post-installation: verify classes, routines, includes, globals, CSP files, Python wheels, FileCopy resources all work
• Metadata: verify via zpm "list", dependencies, history
• Backup and rollback: corrupt backup file during rollback (should handle gracefully)
• Multiple database packages in same namespace (should fail)

Upgrading (existing Update.cls tests cover dependency changes, version validation, update step execution):

• Upgrade database package to newer database package with -swap-db
• Upgrade source package to database package with -swap-db
• Backup verification before upgrade
• Database swap during upgrade
• Update steps can access both globals DB and routines DB
• Update step classes from new database are available
• Developer mode handling: upgrade from source with dev mode to database package, attempt to enable dev mode on database package (should error)

Uninstallation:

• Uninstall database package completely
• Verify namespace returns to original state (backup still exists)
• Verify module removed from zpm "list" and history updated
• Uninstall with dependents (should fail or warn)
• Uninstall leaf dependency (should succeed)
• Mixed packaging: uninstall source package from namespace with database package (database should remain)

Validation & Error Handling:

• Checksum tampering: IRIS.DAT, module.xml checksum field, corrupted .tgz (all should fail)
• Data integrity: verify classes compile, routines execute, globals are correct
• Empty module edge case

Mixed Packaging:

• Database package with source package dependencies
• Install database package then source package in same namespace
• Multiple database packages in same namespace (should fail)
• Verify zpm "list" and other commands work with mixed types

Manual Testing (Not Automated in CI/CD)

These tests should be performed manually to avoid overloading CI/CD systems or because they require special setup/intervention.

Performance & Scalability:

• Package large module (thousands of classes)
• Package module with large globals (GB-sized database)
• Package module with many dependencies (dozens of dependencies)
• Verify packaging time is reasonable for large modules
• Verify installation time is reasonable for large databases

Interrupt & Recovery:

• Interrupt packaging midway (Ctrl+C or kill process), verify cleanup
• Interrupt installation midway (Ctrl+C or kill process), verify rollback

Mirroring Test Cases (Requires Special Setup)

These tests require a mirrored IRIS environment with primary and backup members. They are more complex to set up and run.

Basic Mirrored Installation:

• Install database package on primary member with -swap-db
• Verify globals DB metadata syncs to backup member
• Verify module appears in zpm "list" on backup after metadata sync

Upgrade in Mirrored Environment:

• Install initial version on primary
• Upgrade on backup first (recommended sequence)
• Verify backup database is swapped
• Failover from primary to backup
• Upgrade former primary (now backup)
• Verify both members have new version

Update Steps with Mirroring:

• Test "any member" update steps (run on both primary and backup)
• Test "primary only" update steps (run only on primary)
• Verify primary-only update steps don't run on backup
• Handle case where routines DB is mapped to multiple namespaces

Test Methods (Bundled Test Cases)

This section maps the test cases above into specific test methods that will be implemented in test classes.

Test.PM.Integration.DatabasePackage (new test class)

TestSimplePackageAndInstall:

• Package module with just classes in new namespace (simplest case)
• Verify .tgz structure matches expected format
• Verify IRIS.DAT file is created and not zero-sized
• Verify SHA-256 checksum generation
• Verify module.xml is updated with <Packaging>database</Packaging>
• Install database package via install command with -swap-db
• Verify all classes and routines are accessible
• Verify namespace can compile code and run basic operations
• Verify module appears in zpm "list" output with correct version
• Verify installation is recorded in history
• Package another simple module
• Install via load command with -swap-db
• Verify load command works correctly

TestPackageAndInstallWithDependencies:

• Package module with direct dependencies only
• Install and verify dependencies are tracked correctly via zpm "list -dependencies"
• Package module with transitive dependencies (deep dependency tree)
• Install and verify all dependencies are present
• Verify dependencies.xml is created correctly with all dependency metadata

TestPackageAndInstallAllResourceTypes:

• Package with classes, routines, include files, user-defined globals, generated resources, CSP files, Python wheels, CPF files, FileCopy resources
• Verify CSP files are in csp/ subdirectory
• Verify Python wheels are in wheels/ subdirectory
• Verify FileCopy resources are included
• Install database package with -swap-db
• Verify CSP files are copied to correct location
• Verify Python wheels are installed
• Verify FileCopy resources are in correct locations
• Verify generated resources are present and functional
• Verify all resource types work correctly

TestPackageAndInstallWithTestResources:

• Package with -include-test-resources flag
• Verify test resources are included in package
• Install and verify test resources are present
• Package without flag (default behavior)
• Install and verify test resources are excluded

TestPackageWithUseCurrentDbFlag:

• Package with -use-current-db flag
• Verify packaging into current namespace's routines DB works
• Install and verify package works correctly

TestPackageWithExportPyDepsFlag:

• Package with -export-py-deps toggled on
• Verify Python wheels are included in wheels/ subdirectory
• Install and verify Python wheels work
• Package with -export-py-deps toggled off
• Verify Python wheels are not included

TestPackagingAndInstallValidation:

• Attempt to package with IPM present in routines DB (should error)
• Attempt to package with IPM properly mapped (should succeed and package correctly)
• Package valid module, then tamper with IRIS.DAT (checksum mismatch, install should fail)
• Package valid module, then tamper with module.xml checksum field (install should fail)
• Package valid module, delete IRIS.DAT file (install should fail)
• Create package with zero-sized IRIS.DAT (install should fail)
• Create package with invalid module.xml (install should fail)
• Delete dependencies.xml from package (install should fail appropriately)
• Corrupt .tgz file (install should fail)
• Attempt install with IPM in target routines DB (should error and prevent)

TestBackupAndRollback:

• Package and install database package with -swap-db
• Verify backup creation with correct naming: IRIS_<db-name>_<timestamp>.DAT
• Verify backup is in same directory as original
• Corrupt backup file, trigger rollback scenario
• Verify appropriate error handling when backup is corrupted

TestUpgradeDatabaseToDatabase:

• Install database package v1.0
• Package and upgrade to database package v2.0 with -swap-db
• Verify old database is backed up before upgrade
• Verify new database replaces old one during upgrade
• Verify update steps can access both globals DB and routines DB
• Verify update step classes from new database are available

TestUpgradeSourceToDatabase:

• Install source package v1.0
• Package and upgrade to database package v2.0 with -swap-db
• Verify metadata migration between packaging types
• Verify database swap occurs correctly
• Install source package with dev mode, upgrade to database package (verify dev mode handling)
• Attempt to enable developer mode on database package after install (should error)

TestUninstallDatabasePackage:

• Package and install database package
• Uninstall database-packaged module completely
• Verify namespace returns to state before installation
• Verify backup database still exists (user manages deletion)
• Verify module no longer appears in zpm "list" output
• Verify uninstallation is recorded in history
• Package and install database package with dependencies
• Attempt to uninstall module that has dependents (should fail or warn)
• Uninstall leaf dependency module (should succeed)

TestMixedPackaging:

• Package and install module A (database package) with -swap-db
• Package module B and attempt to install (database package) in same namespace (should fail - can only have one routines DB)
• Install database package, then install source package in same namespace
• Uninstall source package, verify database package remains
• Package module as database package with source package dependencies
• Install and verify all dependencies work correctly
• Verify zpm "list" shows both types