Skip to content

Enabling export of multiple instances #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fredrodlima merged 8 commits into from
Apr 16, 2025
Merged

Enabling export of multiple instances #12

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
596c055
InstanceExport.ps1: Updating script to enable multi instances export
fredrodlima Apr 11, 2025
8c27244
README.md: Updating texts for instructions on how to run the script
fredrodlima Apr 11, 2025
fe30b37
Update README.md
fredrodlima Apr 12, 2025
3411cc1
InstanceExport: Removing $instance on Log Cmdlet
fredrodlima Apr 15, 2025
4bbbe3c
Update README.md
fredrodlima Apr 15, 2025
054acb3
InstanceExport: Extracting ExportInstanceData
fredrodlima Apr 15, 2025
6d9e34e
Merge remote-tracking branch 'origin/enabling-export-multiple-instanc…
fredrodlima Apr 15, 2025
21f02b0
InstanceExport: log start message sooner
bigdogwillfeed Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
278 changes: 146 additions & 132 deletions InstanceExport/PowerShell/InstanceExport.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ Function Log {
[Parameter(Mandatory = $true)][String]$msg,
[Parameter(Mandatory = $false)][String]$displayMsg,
[Parameter(Mandatory = $true)][String]$logLevel,
[Parameter(Mandatory = $true)][String]$currentDate
[Parameter(Mandatory = $true)][String]$currentDate,
[Parameter(Mandatory = $true)][String]$instance
Copy link
Copy Markdown
Contributor Author

@fredrodlima fredrodlima Apr 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the $instance parameter so we could create different folders for each exported instance as well as update log texts and messages appropriately

Copy link
Copy Markdown
Contributor Author

@fredrodlima fredrodlima Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the $instance parameter in the Log and CreateDirectoryIfNotExits so the log is saved in the exportFilePath and has content for all instances that are exported

)
switch ($logLevel) {
"error" {
Expand All @@ -92,25 +93,26 @@ Function Log {
}
}
}
Add-Content "$($exportFilePath)/InstanceExport_$($currentDate)_log.txt" $msg
Add-Content "$($exportFilePath)/$instance/$($instance)_InstanceExport_$($currentDate)_log.txt" $msg
}

Function CreateDirectoryIfDoesNotExist {
param (
[Parameter(Mandatory = $true)][String]$directoryPath,
[Parameter(Mandatory = $true)][String] $currentDate
[Parameter(Mandatory = $true)][String] $currentDate,
[Parameter(Mandatory = $true)][String] $instance
)
$directoryExists = Test-Path -Path $directoryPath
if (!$directoryExists) {
try {
New-Item -Path $directoryPath -ItemType "directory" | Out-Null
$msg = "$directoryPath created"
Log -msg $msg -displayMsg $msg -logLevel "info" -currentDate $currentDate
Log -msg $msg -displayMsg $msg -logLevel "info" -currentDate $currentDate -instance $instance
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "Error creating directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate -instance $instance
}
}
}
Expand All @@ -121,7 +123,7 @@ Function GetPassword() {
return ConvertFrom-SecureString -SecureString $password -AsPlainText
}
catch {
Log -msg $_.Exception.Message -logLevel "displayInfo" -currentDate $currentDate
Log -msg $_.Exception.Message -logLevel "displayInfo" -currentDate $currentDate -instance $instance
}
# fallback to powershell 5
$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)
Expand All @@ -131,7 +133,7 @@ Function GetPassword() {
}
# we're probably on some non-windows environment... fall back to unsecured
$msg = "Warning: Unable to handle password securely. Falling back to plain text password"
Log -msg $msg -logLevel "displayInfo" -currentDate $currentDate
Log -msg $msg -logLevel "displayInfo" -currentDate $currentDate -instance $instance
return Read-Host "$msg. To continue, please enter the password again"
}

Expand Down Expand Up @@ -187,150 +189,162 @@ function Export {
This function exports an given instancce and save it to your disk in the specified exportFilePath from a JSON manifest file
#>
$currentDate = Get-Date -Format "MM_dd_yyyy_HH_mm_ss"

$manifestFileExists = Test-Path -Path $manifestFilePath

if ($manifestFileExists) {
$manifestJson = Get-Content -Raw -Path $manifestFilePath | ConvertFrom-Json
if ($manifestJson.CreateDate) {
$currentDate = Get-Date $manifestJson.CreateDate -Format "MM_dd_yyyy_HH_mm_ss"
}
else {
$msg = "Error in reading manifest file"
$displayMsg = "An error occurred when reading manifest file"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1
}

try {
CreateDirectoryIfDoesNotExist -directoryPath $exportFilePath -currentDate $currentDate
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1
}

try {
$accessTokenResponseModel = Login
}
catch {
Log -msg "An error occurred" -displayMsg "$($_.Exception.Response.StatusCode.value__): An error occurred when logging in" -logLevel "error" -currentDate $currentDate
Log -msg "Authorization error for Instance Export" -logLevel "error" -currentDate $currentDate
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate
Log -msg "Url: api/login" -logLevel "error" -currentDate $currentDate
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate
}

if ($accessTokenResponseModel) {

$accessToken = $accessTokenResponseModel.access_token

$header = @{
"authorization" = "Bearer $accessToken"
}

$currentCategory = "";
$i = 0;
for (; $i -lt $manifestJson.Entries.Length; $i = $i + 1) {
$entry = $manifestJson.Entries[$i]
if ($currentCategory -ne $entry.Category) {
$currentCategory = $entry.Category;
$message = "Starting extraction of $($entry.Category) category."
Log -msg $message -displayMsg $message -logLevel "displayInfo" -currentDate $currentDate

$manifestFilePathExists = Test-Path -Path $manifestFilePath

if ($manifestFilePathExists) {
$manifestFilePath = $manifestFilePath.TrimEnd('\') # Trim any trailing backslash if it exists

$manifesJsonFiles = Get-ChildItem -Path "$manifestFilePath\*" -Include "*.json"
Copy link
Copy Markdown
Contributor Author

@fredrodlima fredrodlima Apr 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the most important change in the script where we are getting all the .json files inside the manifestFilePath provided and running a foreach to export each one of the instances we would like to export in a single run of the script


foreach ($manifestFile in $manifesJsonFiles) {

$manifestFileExists = Test-Path -Path $manifestFile
Copy link
Copy Markdown
Contributor Author

@fredrodlima fredrodlima Apr 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From here is basically the same code we previously had for a single instance export. Just a few simple and small changes, nothing fancy to highlight if you see the changes hiding whitespaces


$instance = ($manifestFile.BaseName -split ' ')[0]
Copy link
Copy Markdown
Contributor Author

@fredrodlima fredrodlima Apr 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are getting the instance name from the BaseName of a given manifest json file. We should recommend users to not edit the name of the files we provide in order to things run smoothly for them.

Copy link
Copy Markdown
Contributor

@bigdogwillfeed bigdogwillfeed Apr 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each manifest has subdomain inside of it... probably better to use that than trust the filename


if ($manifestFileExists) {
$manifestJson = Get-Content -Raw -Path $manifestFile | ConvertFrom-Json
if ($manifestJson.CreateDate) {
$currentDate = Get-Date $manifestJson.CreateDate -Format "MM_dd_yyyy_HH_mm_ss"
}
else {
$msg = "Error in reading manifest file"
$displayMsg = "An error occurred when reading manifest file"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate -instance $instance
Comment thread
fredrodlima marked this conversation as resolved.
Outdated
Exit 1
}
$message = "Extracting data from $($entry.Url)"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$directoryPath = $exportFilePath + "/" + $entry.Path


try {
CreateDirectoryIfDoesNotExist -directoryPath $directoryPath -currentDate $currentDate
CreateDirectoryIfDoesNotExist -directoryPath "$exportFilePath\$instance" -currentDate $currentDate -instance $instance
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate -instance $instance
Exit 1
}

$msg = "$instance"
$displayMsg = "Starting exporting data for instance $instance"
Log -msg $msg -displayMsg $displayMsg -logLevel "success" -currentDate $currentDate -instance $instance

try {
$accessTokenResponseModel = Login
}
catch {
Log -msg "An error occurred" -displayMsg "$($_.Exception.Response.StatusCode.value__): An error occurred when logging in" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "Authorization error for Instance Export" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "Url: api/login" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate -instance $instance
}
$fileName = $entry.FileName.Split([IO.Path]::GetInvalidFileNameChars()) -join ''
$filePath = "$directoryPath/$fileName"
$fileAlreadyExists = Test-Path -Path $filePath -PathType Leaf
if (!$fileAlreadyExists -or $overwrite) {
$entryExportParameters = @{
Method = "GET"
Uri = "https://$($manifestJson.SubDomain).$($manifestJson.HostName)$($entry.Url)"
Headers = $header
ContentType = "application/json"

if ($accessTokenResponseModel) {

$accessToken = $accessTokenResponseModel.access_token

$header = @{
"authorization" = "Bearer $accessToken"
}

try {
if (!$entry.fileName.Contains(".json")) {
Invoke-RestMethod @entryExportParameters -OutFile $filePath | Out-Null

$currentCategory = "";
$i = 0;
for (; $i -lt $manifestJson.Entries.Length; $i = $i + 1) {
$entry = $manifestJson.Entries[$i]
if ($currentCategory -ne $entry.Category) {
$currentCategory = $entry.Category;
$message = "Starting extraction of $($entry.Category) category."
Log -msg $message -displayMsg $message -logLevel "displayInfo" -currentDate $currentDate -instance $instance
}
$message = "Extracting data from $($entry.Url)"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate -instance $instance
$directoryPath = $exportFilePath + "/" + $instance + "/" + $entry.Path

try {
CreateDirectoryIfDoesNotExist -directoryPath $directoryPath -currentDate $currentDate -instance $instance
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate -instance $instance
}
$fileName = $entry.FileName.Split([IO.Path]::GetInvalidFileNameChars()) -join ''
$filePath = "$directoryPath/$fileName"
$fileAlreadyExists = Test-Path -Path $filePath -PathType Leaf
if (!$fileAlreadyExists -or $overwrite) {
$entryExportParameters = @{
Method = "GET"
Uri = "https://$($manifestJson.SubDomain).$($manifestJson.HostName)$($entry.Url)"
Headers = $header
ContentType = "application/json"
}

try {
if (!$entry.fileName.Contains(".json")) {
Invoke-RestMethod @entryExportParameters -OutFile $filePath | Out-Null
}
else {
$GetEntriesResponse = Invoke-RestMethod @entryExportParameters
$GetEntriesResponse | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath
}
}
catch {
$message = "Error occurred when extracting data from $($entry.Url)"
Log -msg "An error occurred" -displayMsg $message -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg $message -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "Url: $($entry.Url)" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate -instance $instance
Log -msg "Exception: $($_.Exception)" -logLevel "error" -currentDate $currentDate -instance $instance
Write-Verbose $_.Exception
Write-Verbose $_.Exception.Response
if ($_.Exception.Response.StatusCode -eq 503) {
$message = "Lost connection to server. Waiting to restablish connection."
Write-ColorOutput red $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
do {
$response = ServerHealtCheck("https://$($manifestJson.SubDomain).$($manifestJson.HostName)/en/healthcheck");
} while ($response.Exception)
$message = "Server connection reestablished. Resuming export"
Write-ColorOutput green $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$i = $i - 1;
}
}
}
else {
$GetEntriesResponse = Invoke-RestMethod @entryExportParameters
$GetEntriesResponse | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath
$message = "Skipping $filePath because file already exists"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate -instance $instance
}
}
$msg = "Exporting data for instance $instance was succesful."
Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate -instance $instance

try {
Logout($header)
}
catch {
$message = "Error occurred when extracting data from $($entry.Url)"
Log -msg "An error occurred" -displayMsg $message -logLevel "error" -currentDate $currentDate
Log -msg $message -logLevel "error" -currentDate $currentDate
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate
Log -msg "Url: $($entry.Url)" -logLevel "error" -currentDate $currentDate
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate
Log -msg "Exception: $($_.Exception)" -logLevel "error" -currentDate $currentDate
Write-Verbose $_.Exception
Write-Verbose $_.Exception.Response
if ($_.Exception.Response.StatusCode -eq 503) {
$message = "Lost connection to server. Waiting to restablish connection."
Write-ColorOutput red $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
do {
$response = ServerHealtCheck("https://$($manifestJson.SubDomain).$($manifestJson.HostName)/en/healthcheck");

} while ($response.Exception)
$message = "Server connection reestablished. Resuming export"
Write-ColorOutput green $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$i = $i - 1;
}
$message = "Error occurred when logging out"
Log -msg $message -displayMsg $message -logLevel "error" -currentDate $currentDate -instance $instance
}
}
else {
$message = "Skipping $filePath because file already exists"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$msg = "Exporting data for instance $instance finished with errors."
$displayMsg = "Exporting data for instance $instance was not successful. Check the log file for more details."
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate -instance $instance
}

}
$msg = "Exporting Instance run finished."
Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate

try {
Logout($header)
}
catch {
$message = "Error occurred when logging out"
Log -msg $message -displayMsg $message -logLevel "error" -currentDate $currentDate
else {
$displayMsg = "An error occurred when looking for manifest folder"
Log -msg "Error in finding manifest folder" -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Copy link
Copy Markdown
Contributor

@bigdogwillfeed bigdogwillfeed Apr 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is displayed if the manifest file doesn't exist, which 1) is confusing, because it's a file not found but the message says folder and 2) should rarely happen because we're iterating over the list of json files in the folder to begin with.

}

}
else {
$msg = "Exporting Instance run finished with errors."
$displayMsg = "Exporting Instance run was not successful. Check the log file for more details."
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}

}
else {
try {
CreateDirectoryIfDoesNotExist -directoryPath $exportFilePath -currentDate $currentDate
}
catch {
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg "Error trying to create $directoryPath directory" -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}

$displayMsg = "An error occurred when looking for manifest file"
Log -msg "Error in finding manifest file" -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
$msg = "Exporting Instance script run finished."
Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate -instance $instance
Comment thread
fredrodlima marked this conversation as resolved.
Outdated
}

Comment thread
fredrodlima marked this conversation as resolved.
Outdated
}
Export
Export
Loading