Skip to content

Remove 29 dead videos from the data - #240

Closed
banool wants to merge 1 commit into
banool/filter-dead-mediafrom
banool/remove-dead-videos-data
Closed

Remove 29 dead videos from the data#240
banool wants to merge 1 commit into
banool/filter-dead-mediafrom
banool/remove-dead-videos-data

Conversation

@banool

@banool banool commented Aug 1, 2026

Copy link
Copy Markdown
Owner

First cleanup pass of filter_dead_media.py (see #239), run standalone against the committed data with extra-patient re-checks (5-minute round spacing), then independently verified ~40 minutes later by verify_removed_media.py (4 rounds, 2-minute spacing, different code path).

Result

  • 29 dead video URLs removed (0.52% of 5537 unique URLs), dropping 145 references.
  • No dictionary content lost: 0 entries and 0 sub_entries removed — every affected sub_entry has other videos that still work.
  • Every removed URL: 5/5 authentic Swift 404s in the filter (status + x-trans-id/x-openstack-request-id + exact not-found body), then 4/4 in the verifier: 29/29 confirmed, 0 false positives, 0 unconfirmed. All 30 sampled kept URLs alive.
  • All 29 are also 404 on the R2 mirror (they were never mirrorable), so no user was reaching these via fallback.
  • The removed set exactly matches the 29 paths archive_orphaned_media.py reports as referenced-but-missing from the mirror — two independent signals agreeing.

These 29 URLs are what has been driving video_resolve_failed/http_404 in ~5.6% of Auslan sessions (they're heavily shared Definition/Gloss videos). After this merges and clients refresh their data, that metric should fall toward zero.

The 29 removed URLs and the entries that referenced them
Path Referenced by
/mp4video/Definition/00/60.mp4 accommodate, accommodation, billet, boarder, board (live at), dormitory, hostel, hotel …
/mp4video/Definition/00/83.mp4 act, acting, actor, perform
/mp4video/Definition/19/19233.mp4 eliminate, elimination, exclude, exclusion, exempt, exemption, get rid of, reject …
/mp4video/Definition/20/20442.mp4 run (nose), runny nose
/mp4video/Definition/20/20443.mp4 cure, cured, heal, healed, recovery
/mp4video/Definition/20/20446.mp4 mince, mincemeat
/mp4video/Definition/20/20450.mp4 include, included, inclusion, incorporate, incorporation, involve, involved, involvement …
/mp4video/Definition/20/20451.mp4 alcoholic, alcoholism
/mp4video/Definition/20/20452.mp4 bain marie
/mp4video/Definition/20/20453.mp4 bain marie
/mp4video/Definition/20/20495.mp4 health, healthy, How are you, well (healthy)
/mp4video/Definition/20/20501.mp4 panic, petrified, petrify, scare, scared, terrified, terror
/mp4video/Definition/20/20502.mp4 block (stop), defence, defend, guard, prevent, prevention, protect, protection …
/mp4video/Definition/20/20503.mp4 rape, rapist
/mp4video/Definition/20/20504.mp4 rape, fuck (sexual act)
/mp4video/Definition/20/20510.mp4 doubtful, hazardous, perilous, risk, risky, venture, vulnerable
/mp4video/Definition/20/20514.mp4 chase, pursue, pursuit
/mp4video/Definition/20/20515.mp4 hit, slap
/mp4video/Definition/20/20516.mp4 make love, copulate, copulation, intercourse, sex, sexual, sexual intercourse
/mp4video/Definition/20/20518.mp4 critical, criticise, criticism
/mp4video/Definition/20/20530.mp4 eyewitness, saw (see), seen, witness
/mp4video/Definition/51/5108.mp4 bodyguard, guard, supervise, watch over
/mp4video/Definition/57/5773.mp4 describe, description, explain, explanation, inform, information
/mp4video/Definition/59/5923.mp4 include, included, inclusion, incorporate, incorporation, involve, involved, involvement …
/mp4video/Definition/91/9184.mp4 eliminate, elimination, exclude, exclusion, exempt, exemption, get rid of, reject …
/mp4video/Definition/91/9185.mp4 bond, connected, connection, link, linked, related, relationship
/mp4video/Definition/96/9661.mp4 afraid, fear, fright, frighten, frightened, scare, scared
/mp4video/Gloss/79/7944.mp4 intersectionality
/mp4video/Gloss/82/8271.mp4 opinion

Full per-attempt evidence (headers, body hashes, timings) is in the filter/verify reports; ask if you want them attached.

🤖 Generated with Claude Code

https://claude.ai/code/session_017bum5SFYEZD4gqX8PZhT6i

First cleanup pass of filter_dead_media.py against the committed data:
29 URLs answered 5/5 authentic Swift 404s across spaced rounds and were
independently re-confirmed 4/4 by verify_removed_media.py ~40 minutes
later. No entries or sub_entries were lost — every affected sub_entry
has other working videos. These URLs drove the video_resolve_failed
http_404 errors seen in ~5.6% of sessions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017bum5SFYEZD4gqX8PZhT6i
@banool
banool deleted the branch banool/filter-dead-media August 1, 2026 03:48
@banool banool closed this Aug 1, 2026

banool commented Aug 1, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #241 — same branch, rebased onto master now that #239 has merged. This PR couldn't be reopened because its base branch banool/filter-dead-media was deleted on merge.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant