Feature | CMG-703 | stack-to-stack migration with audit report#1078
Feature | CMG-703 | stack-to-stack migration with audit report#1078chetan-contentstack wants to merge 18 commits into
Conversation
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
7 similar comments
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
chetan-contentstack
requested review from
shradha-nahar,
umesh-more-cstk,
umeshmore45 and
vikrantraut-cstk
and removed request for
umeshmore45
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
There was a problem hiding this comment.
Pull request overview
Implements Contentstack stack-to-stack migration support end-to-end (export → validate → audit → map → test → final), adding new backend services/routes, upload-api validators/mappers, and UI flow updates (including cross-region source login/session handling and improved long-running migration UX).
Changes:
- Adds Contentstack export validation + locale extraction + upload-api validator/mapper support.
- Adds API support for source export/validation/audit (with sharded lowdb audit storage) and CLI import safeguards (link normalization + per-module fallback).
- Updates UI migration flow (audit step, long timeouts, better log parsing, and regional source-login token caching).
Reviewed changes
Copilot reviewed 67 out of 69 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/tests/unit/validators/contentstack.validator.test.ts | Unit tests for Contentstack zip/json validator. |
| upload-api/tests/unit/services/contentstack.service.test.ts | Unit tests for Contentstack mapper service POSTing locales. |
| upload-api/tests/unit/services/contentstack.locales.test.ts | Unit tests for resolving export root + extracting locales. |
| upload-api/tests/unit/config/index.config.test.ts | Updates config env var precedence + mysql default tests. |
| upload-api/src/validators/index.ts | Registers Contentstack validator for zip/json formats. |
| upload-api/src/validators/contentstack.ts | Adds Contentstack zip/json validation logic. |
| upload-api/src/services/createMapper.ts | Adds Contentstack mapper selection. |
| upload-api/src/services/contentstack/locales.ts | Implements export-root resolution + locale extraction for upload-api. |
| upload-api/src/services/contentstack/index.ts | Creates Contentstack mapper (posts extracted locales). |
| upload-api/src/routes/index.ts | Tightens file path handling + response typing; fixes comment typo. |
| upload-api/src/config/index.ts | Aligns local path env precedence + formatting. |
| upload-api/migration-aem/package-lock.json | Removes @types/uuid entry from lockfile. |
| ui/src/utilities/migrationSourceSession.ts | Session storage helpers for source-region token caching. |
| ui/src/utilities/functions.ts | Ensures failure notifications always have non-empty text. |
| ui/src/utilities/constants.ts | Adds constant for popup postMessage source. |
| ui/src/services/api/user.service.ts | Adds profile fetch for an override token (regional login). |
| ui/src/services/api/upload.service.ts | Hardens getCall error fallback when err.response is missing. |
| ui/src/services/api/stacks.service.ts | Adds optional source_region + app token override when listing stacks. |
| ui/src/services/api/migration.service.ts | Adds source-config update, audit/export endpoints, and long timeouts. |
| ui/src/pages/Migration/index.tsx | Adds audit step + source_details persistence + multiple step flow changes. |
| ui/src/pages/Login/index.tsx | Adds popup regional login flow + return-to handling + safer errors. |
| ui/src/context/app/app.interface.ts | Extends migration state shape with source_details + audit + stepValue. |
| ui/src/components/TestMigration/index.tsx | Improves disable logic + state persistence + completion handling. |
| ui/src/components/TestMigration/index.scss | Tweaks layout spacing for test migration UI. |
| ui/src/components/MigrationFlowHeader/index.tsx | Makes CTA label dependent on step/CMS; fixes step gating logic. |
| ui/src/components/LogScreen/MigrationLogViewer.tsx | Buffers socket log chunks + improves parsing + completion handling. |
| ui/src/components/LogScreen/index.tsx | Same log chunk buffering + parsing improvements for test migration logs. |
| ui/src/components/LegacyCms/legacyCms.scss | Styles new Contentstack source panel UI. |
| ui/src/components/LegacyCms/Actions/LoadSelectCms.tsx | Initializes source_details and sets source_mode for Contentstack. |
| ui/src/components/DestinationStack/Actions/LoadLanguageMapper.tsx | Supports locale arrays as strings or objects (Contentstack format). |
| ui/src/components/Common/DeleteProjectModal/index.tsx | Fixes React prop casing: tabIndex. |
| ui/src/components/AuditReport/index.tsx | Adds UI for running/reviewing audit and persisting exclusions/summary. |
| ui/src/components/AuditReport/index.scss | Styling for audit report UI/table. |
| ui/src/cmsData/migrationSteps.json | Adds Audit step and renumbers steps to 6-step flow. |
| api/tests/unit/utils/normalize-entry-links.utils.test.ts | Tests link normalization across schema shapes (group/blocks/multiple). |
| api/tests/unit/utils/contentstack-user-orgs.utils.test.ts | Tests org mapping across region-specific role payload shapes. |
| api/tests/unit/services/validation.service.test.ts | Tests export root resolution + structure validation. |
| api/tests/unit/services/user.service.test.ts | Ensures profile response includes region and mapped orgs. |
| api/tests/unit/services/projects.service.test.ts | Updates step-progression tests + adds source/audit update tests. |
| api/tests/unit/services/migration.service.test.ts | Adds tests for export/validate/audit endpoints and step/status updates. |
| api/tests/unit/services/exportCli.service.test.ts | Tests CLI export execution + region handling + error cases. |
| api/tests/unit/services/contentMapper.service.test.ts | Updates expected step gating for mapper operations. |
| api/tests/unit/services/auth.service.test.ts | Updates org membership requirements + aligns error message constant. |
| api/tests/unit/services/audit.service.test.ts | Tests audit generation paths and edge cases. |
| api/tests/unit/routes/projects.routes.test.ts | Adds route wiring tests for new project endpoints. |
| api/tests/unit/routes/migration.routes.test.ts | Adds route wiring tests for new migration endpoints. |
| api/tests/unit/models/audit-lowdb.test.ts | Tests sharded audit persistence + retrieval behavior. |
| api/tests/unit/controllers/migration.controller.test.ts | Updates controller behavior to await long-running service calls. |
| api/src/utils/sanitize-path.utils.ts | Adds allowlisted export-root path assertion helper. |
| api/src/utils/normalize-entry-links.utils.ts | Adds export-wide legacy link normalization utility. |
| api/src/utils/contentstack-user-orgs.utils.ts | Adds robust org mapping util across role shapes/regions. |
| api/src/services/validation.service.ts | Adds Contentstack export structure validation + root resolution. |
| api/src/services/user.service.ts | Uses org mapping util + includes region in profile payload. |
| api/src/services/runCli.service.ts | Adds link normalization + module fallback + export-root resolution. |
| api/src/services/projects.service.ts | Adds source config + audit selection updates; adds audit step gating. |
| api/src/services/exportCli.service.ts | Implements Contentstack stack export via CLI with structured logging. |
| api/src/services/auth.service.ts | Uses org mapping util for admin/owner/member determination. |
| api/src/services/audit.service.ts | Implements audit generation + persistence to sharded lowdb store. |
| api/src/server.ts | Fixes log tailing to avoid awaitWriteFinish delays; adds offset reset. |
| api/src/routes/projects.routes.ts | Adds project routes for source-config and audit-selections. |
| api/src/routes/migration.routes.ts | Adds migration routes for export/validate/audit. |
| api/src/models/project-lowdb.ts | Extends project schema to store source_details/audit/validation/source_locales. |
| api/src/models/audit-lowdb.ts | Adds sharded audit storage model. |
| api/src/controllers/projects.controller.ts | Adds controllers for source-config + audit-selections. |
| api/src/controllers/migration.controller.ts | Adds controllers for export/validate/audit; awaits long operations. |
| api/src/constants/index.ts | Adds CMS.CONTENTSTACK, audit DB config, and renumbers step constants. |
| .gitignore | Ignores export-stack/ output directory. |
Files not reviewed (1)
- upload-api/migration-aem/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
| // Default 30-minute cap on any single CLI invocation. Large stacks legitimately | ||
| // take many minutes to export, but a stuck child process should not hold an | ||
| // API request open indefinitely. Overridable via EXPORT_CLI_TIMEOUT_MS. | ||
| const DEFAULT_CLI_TIMEOUT_MS = 30 * 60 * 1000; |
There was a problem hiding this comment.
can you explain this why we are cli timeout?
There was a problem hiding this comment.
This was asked on Above comment By @vikrantraut-cstk.
#1078 (comment)
| break; | ||
| } | ||
| if (cms !== CMS.AEM) { | ||
| if (cms !== CMS.AEM && cms !== CMS.CONTENTSTACK) { |
There was a problem hiding this comment.
why we are not putting this in switch case
There was a problem hiding this comment.
@umesh-more-cstk These blocks are intentionally outside the switch (cms) — they run after the CMS-specific pre-processing. The first one applies to every CMS except AEM and Contentstack, and the second is Contentstack-only post-migration cleanup. Moving them into the switch would duplicate the logic across multiple cases (or several cases would fall through into Contentstack-style cleanup). Happy to refactor if you'd prefer, but the current shape avoids the duplication.
| ); | ||
|
|
||
| // Handle CMS-specific post-migration tasks | ||
| if (cms === CMS.CONTENTSTACK) { |
There was a problem hiding this comment.
why we are not putting this in switch case
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
1 similar comment
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
Adds Contentstack-to-Contentstack migration across regions/orgs: CLI-driven source export, export-structure validation, audit report (unused assets, unpublished entries, empty content types, unused global fields), and import via the official CLI with link-field normalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… and environment variable usage
- audit.service: support all regions in buildContentstackUrl, quoted-token asset reference detection, optional chaining, drop unused assetUidSet. - runCli.service: replace literal current_step=6 with STEPPER_STEPS.MIGRATION. - projects.service: gate audit-summary requirement to Contentstack source so non-CS migrations aren't blocked at step 2. - Migration UI: derive next step from isContentstackSource in content mapper and test migration handlers.
…tack - migration.service: skip non-CS pre-create steps (fieldAttacher, marketplace apps, extensions, taxonomies, global fields, delta asset index) when cms === CONTENTSTACK; the CLI imports these directly from the export. - migration.controller: default to 200 when service returns void so Express doesn't crash with "Invalid status code: undefined". - contentMapper + projects.service: align iteration default to 1 so reads match writers and step 4 finds the audit-generated mappers. - post-rebase cleanup: drop dead AutoMappedMerge modal reference, remove duplicate updateNewMigrationData import, fix dangling setHasShownCompletionNotification reset effect.
chetan-contentstack
force-pushed
the
feature/cmg-703-stack-to-stack
branch
from
4bf9336 to
f32cf05
Compare
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
| const allowedChars = | ||
| 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_.-' + | ||
| path.sep; | ||
| let safeRel = ''; | ||
| for (let i = 0; i < rel?.length; i++) { | ||
| const ch = rel?.charAt(i); | ||
| if (allowedChars.includes(ch)) { | ||
| safeRel += ch; | ||
| } | ||
| } |
| const KEY_TOKEN = 'migration_source_app_token'; | ||
| const KEY_REGION = 'migration_source_region_id'; | ||
|
|
||
| export function setMigrationSourceSession(region: string, appToken: string): void { | ||
| try { | ||
| sessionStorage.setItem(KEY_TOKEN, appToken); | ||
| sessionStorage.setItem(KEY_REGION, region.trim()); | ||
| } catch { | ||
| /* ignore */ | ||
| } | ||
| } | ||
|
|
||
| export function getMigrationSourceSession(): { region: string; appToken: string } | null { | ||
| try { | ||
| const appToken = sessionStorage.getItem(KEY_TOKEN) || ''; | ||
| const region = sessionStorage.getItem(KEY_REGION) || ''; | ||
| if (!appToken || !region) return null; | ||
| return { region, appToken }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| } | ||
|
|
||
| export function clearMigrationSourceSession(): void { | ||
| try { | ||
| sessionStorage.removeItem(KEY_TOKEN); | ||
| sessionStorage.removeItem(KEY_REGION); | ||
| } catch { |
| type FileProcessingResponse = { | ||
| status: number; | ||
| message: string; | ||
| file_details: any; | ||
| file?: string; | ||
| }; |
| const baseDir = path.join(__dirname, '..', '..', 'extracted_files'); | ||
| const filePath = path.join(baseDir, `${safeName}.json`); | ||
| // Validate path is within expected directory | ||
| if (isPathWithinBase(filePath, baseDir)) { | ||
| const filePath = (data as any)?.extractedPath as string | undefined; | ||
| if (filePath && isPathWithinBase(filePath, baseDir)) { |
| const baseDir = path.join(__dirname, '..', '..', 'extracted_files'); | ||
| let filePath = path.join(baseDir, safeName); | ||
| if (data?.file !== undefined) { | ||
| const safeFile = sanitizeFilename(data.file); | ||
| filePath = path.join(baseDir, safeName, safeFile); | ||
| } | ||
| // Validate path is within expected directory | ||
| if (isPathWithinBase(filePath, baseDir)) { | ||
| const filePath = (data as any)?.extractedPath as string | undefined; | ||
| if (filePath && isPathWithinBase(filePath, baseDir)) { | ||
| createMapper(filePath, projectId, app_token, affix, config); |
| const baseDir = path.join(__dirname, '..', '..', 'extracted_files'); | ||
| let filePath = path.join(baseDir, safeFileName); | ||
|
|
||
| // If the processor returned a specific file/folder, update the path | ||
| if (data?.file) { | ||
| const safeDataFile = sanitizeFilename(data.file); | ||
| filePath = path.join(baseDir, safeFileName, safeDataFile); | ||
| } | ||
|
|
||
| // Validate path is within expected directory | ||
| if (isPathWithinBase(filePath, baseDir)) { | ||
| const filePath = (data as any)?.extractedPath as string | undefined; | ||
| if (filePath && isPathWithinBase(filePath, baseDir)) { | ||
| createMapper(filePath, projectId, app_token, affix, config); |
| const safeExportPath = assertExportPathInAllowedRoot(exportPath); | ||
| const assetsPath = path.join(safeExportPath, 'assets', 'assets.json'); | ||
| const entriesDir = path.join(safeExportPath, 'entries'); | ||
| const contentTypesDir = path.join(safeExportPath, 'content_types'); | ||
| const globalFieldsDir = path.join(safeExportPath, 'global_fields'); | ||
|
|
||
| const assetIndex = await readJson(assetsPath); | ||
| const assets: any[] = []; | ||
| for (const value of Object.values(assetIndex as Record<string, string>)) { |
| const resolveSourcePathForImport = (project: any, stackUid: string): string => { | ||
| const defaultPath = path.join(process.cwd(), MIGRATION_DATA_CONFIG.DATA, stackUid); | ||
| if (!project?.legacy_cms) { | ||
| return defaultPath; | ||
| } | ||
|
|
||
| // For Contentstack-to-Contentstack, prefer the actual export path regardless of source_mode | ||
| if (project?.legacy_cms?.cms === CMS.CONTENTSTACK) { | ||
| const sourcePath = | ||
| project?.legacy_cms?.source_details?.export_path || | ||
| project?.legacy_cms?.source_details?.imported_data_path || | ||
| project?.extract_path; | ||
| return sourcePath || defaultPath; | ||
| } |
Summary
Implements CMG-703 — content migration from a source Contentstack stack to a destination stack across regions, organizations, or within the same org/region.
The flow: source-stack auth → CLI-driven export → export validation → audit report → destination config → content mapper → test migration → final migration.
Scope delivered
Source stack selection
LoadUploadFile.tsx).npx @contentstack/cli cm:stacks:exportvia the newexportCli.service.ts; exported data lands underapi/export-stack/<stackId>/.Export validation
validation.service.tsresolves the new branch-nested export layout (main/,master/, …) and reports missing modules (content_types,entries,assets,global_fields,locales) with a clean message.resolvedRooton the project so downstream steps use one canonical path.Audit report (new Step 3)
audit.service.ts+audit-lowdb.tsgenerate a per-project sharded audit covering:AuditReport/index.tsxlets users review and exclude items from the upcoming migration; exclusions are persisted on the project.GET /audit-data/:projectId.Destination configuration
getUserProfileWithToken) so users authenticate once per region and the resulting token is cached for reuse.contentstack-user-orgs.utils.ts.Test & final migration
runCli.service→cm:stacks:importpath with two new safeguards:normalize-entry-links.utils.tsrewrites legacy{ url, title }link values to{ href, title }(recursive throughgroup/blocks).MODULE_NOT_FOUND) trigger a per-module fallback so the rest of the import still succeeds.ERROR:lines from the CLI are treated as non-fatal warnings; only known fatal strings abort the run.Acceptance criteria mapping
audit-lowdb,/audit-data/:projectId)Known limitations (worth follow-up tickets)
source_branchonly; stacks with multiple branches need one project per branch. UI doesn't expose multi-branch selection yet.resolveContentstackExportRootis duplicated betweenapi/andupload-api/; consolidating requires a shared workspace package.