Fix stuck export/upload progress caused by unindexed logs table - #1661
Open
caitmich wants to merge 4 commits into
Open
Fix stuck export/upload progress caused by unindexed logs table#1661caitmich wants to merge 4 commits into
caitmich wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
While diagnosing a hosted Pro instance with 47.2 million rows in its
logstable, we found the root cause is present in CE too: thelogstable has no index beyond its primary key, and the client-side progress-polling JS has no error handling. Together these mean that once alogstable grows large enough for lookups to slow down, progress polling can silently and permanently stop updating, with no indication to the user that anything went wrong.db/migrate/..._add_index_to_logs_on_uid.rb: adds an index onlogs.uid, the columnConsoleController#statusfilters on for every progress-polling request. Without it, each poll is a full table scan. Note: the equivalent Pro migration usesalgorithm: :inplaceto force an online, non-locking index build on MySQL, but that option isn't supported by the SQLite3 adapter, which CE defaults to — passing it here breaks migrations outright for most CE installs, so it's omitted for this version.app/assets/javascripts/hera/modules/console_updater.js.coffee: the polling loop only rescheduled its next check from inside the AJAX success callback, with no.fail()handler — a single dropped/timed-out request permanently stopped progress updates with no error shown. Now retries on failure (resetting the retry count on any success, so intermittent blips don't accumulate), and after 5 consecutive failures stops retrying and shows a "lost connection, please refresh" message instead of hanging silently forever.CHANGELOG: entries for the above.This will be synced into Pro (which has its own equivalent fixes plus two Pro-only changes: batching
LogCleanupJob's deletion, and the same polling fix in the Word-export validator).Testing steps
Confirm the logs table index exists after migrating
Trigger an export or upload and confirm progress updates appear live in the console without needing a manual refresh
Simulate a failed progress-polling request (e.g. block the endpoint temporarily) and confirm polling retries automatically, then shows the "lost connection" message after repeated failures instead of hanging indefinitely
Check List