Skip to content

Remove 29 dead videos from the data - #241

Merged
banool merged 1 commit into
masterfrom
banool/remove-dead-videos-data
Aug 1, 2026
Merged

Remove 29 dead videos from the data#241
banool merged 1 commit into
masterfrom
banool/remove-dead-videos-data

Conversation

@banool

@banool banool commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Reopens #240, rebased onto master now that #239 has merged. GitHub wouldn't let #240 be reopened because its base branch (banool/filter-dead-media) was deleted on merge, so this is a fresh PR with the same single commit. The diff is byte-identical to what #240 carried: 4 files, +214/-504.

First cleanup pass of filter_dead_media.py (see #239), run standalone against the committed data with extra-patient re-checks (5-minute round spacing), then independently verified ~40 minutes later by verify_removed_media.py (4 rounds, 2-minute spacing, different code path).

Result

  • 29 dead video URLs removed (0.52% of 5537 unique URLs), dropping 145 references.
  • No dictionary content lost: 0 entries and 0 sub_entries removed — every affected sub_entry has other videos that still work.
  • Every removed URL: 5/5 authentic Swift 404s in the filter (status + x-trans-id/x-openstack-request-id + exact not-found body), then 4/4 in the verifier: 29/29 confirmed, 0 false positives, 0 unconfirmed. All 30 sampled kept URLs alive.
  • All 29 are also 404 on the R2 mirror (they were never mirrorable), so no user was reaching these via fallback.
  • The removed set exactly matches the 29 paths archive_orphaned_media.py reports as referenced-but-missing from the mirror — two independent signals agreeing.

These 29 URLs are what has been driving video_resolve_failed/http_404 in ~5.6% of Auslan sessions (they're heavily shared Definition/Gloss videos). After this merges and clients refresh their data, that metric should fall toward zero.

The 29 removed URLs and the entries that referenced them
Path Referenced by
/mp4video/Definition/00/60.mp4 accommodate, accommodation, billet, boarder, board (live at), dormitory, hostel, hotel …
/mp4video/Definition/00/83.mp4 act, acting, actor, perform
/mp4video/Definition/19/19233.mp4 eliminate, elimination, exclude, exclusion, exempt, exemption, get rid of, reject …
/mp4video/Definition/20/20442.mp4 run (nose), runny nose
/mp4video/Definition/20/20443.mp4 cure, cured, heal, healed, recovery
/mp4video/Definition/20/20446.mp4 mince, mincemeat
/mp4video/Definition/20/20450.mp4 include, included, inclusion, incorporate, incorporation, involve, involved, involvement …
/mp4video/Definition/20/20451.mp4 alcoholic, alcoholism
/mp4video/Definition/20/20452.mp4 bain marie
/mp4video/Definition/20/20453.mp4 bain marie
/mp4video/Definition/20/20495.mp4 health, healthy, How are you, well (healthy)
/mp4video/Definition/20/20501.mp4 panic, petrified, petrify, scare, scared, terrified, terror
/mp4video/Definition/20/20502.mp4 block (stop), defence, defend, guard, prevent, prevention, protect, protection …
/mp4video/Definition/20/20503.mp4 rape, rapist
/mp4video/Definition/20/20504.mp4 rape, fuck (sexual act)
/mp4video/Definition/20/20510.mp4 doubtful, hazardous, perilous, risk, risky, venture, vulnerable
/mp4video/Definition/20/20514.mp4 chase, pursue, pursuit
/mp4video/Definition/20/20515.mp4 hit, slap
/mp4video/Definition/20/20516.mp4 make love, copulate, copulation, intercourse, sex, sexual, sexual intercourse
/mp4video/Definition/20/20518.mp4 critical, criticise, criticism
/mp4video/Definition/20/20530.mp4 eyewitness, saw (see), seen, witness
/mp4video/Definition/51/5108.mp4 bodyguard, guard, supervise, watch over
/mp4video/Definition/57/5773.mp4 describe, description, explain, explanation, inform, information
/mp4video/Definition/59/5923.mp4 include, included, inclusion, incorporate, incorporation, involve, involved, involvement …
/mp4video/Definition/91/9184.mp4 eliminate, elimination, exclude, exclusion, exempt, exemption, get rid of, reject …
/mp4video/Definition/91/9185.mp4 bond, connected, connection, link, linked, related, relationship
/mp4video/Definition/96/9661.mp4 afraid, fear, fright, frighten, frightened, scare, scared
/mp4video/Gloss/79/7944.mp4 intersectionality
/mp4video/Gloss/82/8271.mp4 opinion

Full per-attempt evidence (headers, body hashes, timings) is in the filter/verify reports; ask if you want them attached.


Generated by Claude Code

First cleanup pass of filter_dead_media.py against the committed data:
29 URLs answered 5/5 authentic Swift 404s across spaced rounds and were
independently re-confirmed 4/4 by verify_removed_media.py ~40 minutes
later. No entries or sub_entries were lost — every affected sub_entry
has other working videos. These URLs drove the video_resolve_failed
http_404 errors seen in ~5.6% of sessions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017bum5SFYEZD4gqX8PZhT6i
@banool
banool marked this pull request as ready for review August 1, 2026 03:55
@banool
banool merged commit 9999e6d into master Aug 1, 2026
5 checks passed
@banool
banool deleted the banool/remove-dead-videos-data branch August 1, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant