diff --git a/demo/10-stale-error-slot.html b/demo/10-stale-error-slot.html new file mode 100644 index 00000000..b53f2e7e --- /dev/null +++ b/demo/10-stale-error-slot.html @@ -0,0 +1,151 @@ + + + + + +The error that belonged to someone else + + + +
+ +
All problems  /  10
+ +

The error that belonged to someone else

+

The native error slot is never cleared, so a call that fails without writing a message reports the previous one.

+ +
+ +
+
backgroundhow two threads come to share one object
+
+ + + + + + a Python object has no owning thread — it belongs to whoever holds a reference + + + one Reader + one native handle inside it + + thread A + thread B + + + same reference + same reference + + + reads, and reads the error slot + + reads, and reads its own error slot + + nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough + + + the GIL stops both threads running Python at the same instant — it does not stop them + touching the same object, and it is given away entirely during a native call + +
The object is shared, but the error slot is not: each thread has its own. That is why one thread's marker cannot clear another thread's pending error.
+
+
+ +
+
beforemain: the slot is only ever read
+
+ + + + + + one thread, over time + + error slot + + Reader(bad file) + fails, writes message + + + "NotSupported: type is unsupported" + + read, raised + + nothing clears it + + load_settings(bad) + fails, writes nothing + + + still the old message + + raises the wrong error + + +
The second failure inherits the first failure's message, and its exception type.
+
+
+ +
+
afterreading consumes: a marker is written back
+
+ + + + + + one thread, over time + + error slot + + Reader(bad file) + + + "NotSupported: type is unsupported" + + read, raised + then marker written back + + + + marker = "no error of our own" + + load_settings(bad) + fails, writes nothing + + + marker still there + + marker reads as None: "Unknown error" + + +
Reading the slot marks it. A failure that writes nothing now reads back "no error" instead of a stale message.
+
+
+ +
+ +
+

Notes

+

Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.

+

The slot is thread-local and sticky. Python cannot empty it — the library exposes no call for that — so the branch writes a known value in instead, produced by asking the library to free address 2, which it can never be tracking. That free fails predictably and leaves a message the wrapper recognises.

+

The exact text is learned at import rather than hardcoded, so it matches the build actually loaded. If the learned text does not contain 0x2, the mechanism switches itself off and the library behaves as it did on main.

+

Why it matters beyond a wrong message. When a consuming call fails, the wrapper decides who owns the pointer by reading this slot. A stale UntrackedPointer: message makes it conclude the pointer is still Python's, and the object is kept alive holding memory the native side already freed.

+
+c2pa.py:986  _NO_ERROR_MARKER_ADDR
+c2pa.py:1013-1037  _read_native_error — marks on both exit paths
+c2pa.py:1428-1457  _learn_no_error_marker_text
+main:696-716  _read_native_error — "Peeks: the error stays in the native slot"
+tests  test_stale_error_not_misattributed_after_preset_error, test_reading_the_native_error_consumes_it +
+
+ + + +
+ + diff --git a/demo/20-native-section.html b/demo/20-native-section.html new file mode 100644 index 00000000..ae2e5875 --- /dev/null +++ b/demo/20-native-section.html @@ -0,0 +1,212 @@ + + + + + +The native section + + + +
+ +
All problems  /  20  ·  concept
+ +

The native section

+

Not a lock protecting data from other threads — a marked stretch of time on one thread, between a call returning and its error being read. Any free inside it destroys the message.

+ +
+ +
+
backgroundshared objects, per-thread windows
+
+ + + + + + a Python object has no owning thread — it belongs to whoever holds a reference + + + one Resource + one native handle inside it + + thread A + thread B + + + same reference + same reference + + + may open a native section + + may open its own, separately + + nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough + + + the GIL stops both threads running Python at the same instant — it does not stop them + touching the same object, and it is given away entirely during a native call + +
The section is thread-local because the error slot it protects is. A window open on thread A must not gate thread B's frees, or the two would block each other.
+
+ +
+

A Python object lives on the process heap — one region of memory shared by every thread. A thread is not a container that holds objects; it is a separate execution position, with its own call stack, walking through that same shared memory.

+ +

So nothing switches threads and nothing is handed over. In the probe behind this figure, a Reader created on MainThread and used from worker-1 stayed at the same address the whole time, with the same id(). Two threads simply looked at the same place.

+ +

What is shared is the name. An ordinary closure is enough:

+ +
r = Reader("image/jpeg", io.BytesIO(img))
+
+def worker():
+    return r.json()          # closure captures r
+
+threading.Thread(target=worker).start()
+ +

No serialisation, no copy, no transfer step. Compare multiprocessing, where a genuinely separate heap forces objects to be pickled across — there id(r) would differ and mutations would not be visible. Threads have no such boundary, and that is exactly what separates them from processes.

+
+
+ +

The glossary sentence that follows is about executing bytecode: it says nothing about which objects a thread may reach. The GIL keeps MainThread and worker-1 from running Python instructions in the same instant. It does not stop them both holding a reference to one Reader, and it does not stop one calling close() while the other is mid-call.

+ +
+
backgroundwhat the GIL is, and why it does not save you here
+
+ + + + + + the global interpreter lock — a token inside CPython. Only its holder runs Python bytecode. + + + "assure that only one thread executes Python bytecode at a time" — Python glossary + + thread A holds it + + thread B holds it + + thread A holds it + taking turns, never simultaneously + + + + 1  it is handed away for the whole of a native call + + "the GIL is always released when doing I/O" — Python glossary + + thread A: inside the native call + + thread B: running Python + at once + + + + 2  and it never made a statement indivisible + + "Python does not guarantee that high-level statements are atomic" — Python glossary + self._ensure_valid_state() ← the handle is checked + _lib.c2pa_reader_json(self._handle) ← it is used + + another thread + runs in this gap + +
Two independent reasons the GIL does not prevent these bugs: it is given up entirely during a native call, and it was never a guarantee that a check and the use that follows it happen as one step.
+
+
+ +
+
beforemain: a finalizer's free lands in the window
+
+ + + + + one thread — no second thread involved + + + + the window + + + call returns + message now in slot + + + error read + ownership decided from it + + + unrelated object collected → __del__ → c2pa_free + + + slot overwritten: "Other: UntrackedPointer: 0x..." + the real message is gone, and the wrong one steers the decision + +
A free of an untracked pointer writes its own complaint into the same slot. CPython runs finalizers at any bytecode boundary, so this needs no threads at all.
+
+
+ +
+
afterthe window is marked; frees inside it are queued
+
+ + + + + one thread + + + + _native_section: depth > 0 + + + call returns + + error read + message intact + + + same __del__ → teardown sees depth > 0 + + + queued on pending_resources — no free yet + + + depth 0: + queue drains + + the free still happens — just after the message has been read + nested sections raise the depth; only the outermost close drains + +
The free is postponed, not skipped. Without the pending list it would never happen at all, because nothing else would touch that object again.
+
+
+ +
+ +
+

Notes

+

Objects do not belong to threads. If they did, a close() on thread B could not reach thread A's handle, and most of this branch would be unnecessary.

+

Why the window exists. A C interface cannot raise an exception, so failure arrives in two pieces: a return value, and a message fetched by a separate c2pa_error() call. Between the two, arbitrary Python runs.

+

Thread-local because the native slot is: one thread's section must not gate another's frees. Depth-counted because native calls nest — and _read_native_error is itself one. A boolean would be cleared by the innermost exit while an outer classification was still reading.

+

The drain never hides your exception. It keeps only the first failure and returns it rather than raising; the bare raise re-raises whatever the body threw. A cleanup problem is logged, never substituted for the error you were reporting.

+
+c2pa.py:1040-1101  _native_section, _in_native_section, _register_for_section_flush
+c2pa.py:473-474  registration  ·  c2pa.py:517-521  re-registration
+c2pa.py:1071-1081  _drain — swaps the list, isolates each failure
+main  none of these symbols exist
+tests  test_native_section_defers_unrelated_finalizer_free, test_section_drain_error_does_not_mask_the_body_error +
+
+ + + +
+ + diff --git a/demo/30-close-during-call.html b/demo/30-close-during-call.html new file mode 100644 index 00000000..3c2e1196 --- /dev/null +++ b/demo/30-close-during-call.html @@ -0,0 +1,157 @@ + + + + + +Closing something another thread is using + + + +
+ +
All problems  /  30
+ +

Closing something another thread is using

+

A close() frees a handle another thread has already passed into a native call.

+ +
+ +
+
backgroundhow thread B gets hold of thread A's reader
+
+ + + + + + a Python object has no owning thread — it belongs to whoever holds a reference + + + one Reader + one native handle inside it + + thread A + thread B + + + same reference + same reference + + + reader.json() + + reader.close() + + nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough + + + the GIL stops both threads running Python at the same instant — it does not stop them + touching the same object, and it is given away entirely during a native call + +
No handover happens. Both threads simply hold the same reference, so both may call methods on it at any time.
+
+
+ +
+
beforemain: close frees immediately
+
+ + + + + + thread A + native + thread B + + + + + reader.json() + handle checked: valid + + hands the GIL away + + + running, holding + the pointer + + close() + + frees now + + + memory released + native still reading it + + + crash, or garbage + +
The validity check passed before the free. Nothing re-checks it, and the fault happens inside native code with no Python traceback.
+
+
+ +
+
aftercalls are counted; the close is recorded and deferred
+
+ + + + + + thread A + native + thread B + + + + + reader.json() + _inflight = 1 + + + + running, holding + the pointer + + close() + + marks CLOSED now, + records the free + frees nothing yet + + + call returns intact + + _inflight = 0 + + recorded free runs here, once + +
The object is unusable from the moment close is called, but the memory outlives the call that is using it.
+
+
+ +
+ +
+

Notes

+

Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.

+

Why not just hold a lock. These native calls run caller-supplied stream callbacks, which can call back into this API, possibly from a new thread. A lock held across the call would deadlock against that re-entry. So the lock is held only long enough to change a counter, never across the call itself — page 70 shows the case that makes this unavoidable.

+

Two independent reasons defer a teardown — this object's own call being in flight, and the native section. The recorded flag merges with and, so a "close without freeing" can never be upgraded to a free by a later caller who does not know the pointer already moved.

+
+c2pa.py:265-273  the new per-resource state
+c2pa.py:346-363  _native_call — counts, does not lock across the call
+c2pa.py:464-476  _teardown — the deferring branch
+main:264-267  __init__ was three assignments; main:330-337  _teardown freed at once
+tests  test_close_inside_callback_defers_free, test_deferred_consume_is_not_upgraded_to_free +
+
+ + + +
+ + diff --git a/demo/40-borrow-vs-consume.html b/demo/40-borrow-vs-consume.html new file mode 100644 index 00000000..dc047055 --- /dev/null +++ b/demo/40-borrow-vs-consume.html @@ -0,0 +1,152 @@ + + + + + +A consume that starts during a borrow + + + +
+ +
All problems  /  40
+ +

A consume that starts during a borrow

+

A borrowing call validates its pointer once. A consuming call on another thread then hands that same pointer to the library to be freed.

+ +
+ +
+
backgroundone builder, two threads, two kinds of call
+
+ + + + + + a Python object has no owning thread — it belongs to whoever holds a reference + + + one Builder + one native handle inside it + + thread A + thread B + + + same reference + same reference + + + to_archive() — borrows + + with_archive() — consumes + + nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough + + + the GIL stops both threads running Python at the same instant — it does not stop them + touching the same object, and it is given away entirely during a native call + +
Both calls act on the same native handle. Nothing in Python prevents the second from starting while the first is still running.
+
+
+ +
+
beforemain: no entry guard, the consume proceeds
+
+ + + + + + thread A — borrows + native + thread B — consumes + + + + + to_archive(out) + + pointer checked once + + + reading through it, + never re-checks + + with_archive(data) + + hands ownership over + + + native frees the pointer + inside the call + + + A reads freed memory + +
The registry check that normally catches a freed pointer already passed, at the top of thread A's call.
+
+
+ +
+
afterthe consume is refused while a borrow is in flight
+
+ + + + + + thread A — borrows + native + thread B — consumes + + + + + to_archive(out) + _inflight = 1 + + + + reading, undisturbed + + with_archive(data) + + _ensure_not_borrowed() + sees _inflight > 0 + C2paError raised + nothing crosses the boundary + + + A completes normally + +
Refused rather than queued: waiting would mean waiting on caller-supplied callbacks of unbounded duration.
+
+
+ +
+ +
+

Notes

+

Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.

+

Why deferring does not work here. A racing close() can be postponed because Python performs that free. A consuming call's free happens inside the library, during the call, as part of taking ownership. Python neither schedules it nor can postpone it.

+

Two checks cover the two orderings. _ensure_not_borrowed() catches a borrow already running. Marking the object CLOSED before releasing the lock catches one arriving afterwards, because every borrowing call re-checks validity under that same lock.

+

A separate counter tracks mutating calls, so two mutations cannot overlap and a read cannot run during one.

+
+c2pa.py:310-320  _ensure_not_borrowed  ·  c2pa.py:322-330  _ensure_no_mutating_call
+c2pa.py:722-730  _begin_consume — check, then mark CLOSED under the lock
+main:501-510  _consume_and_swap called straight through, no guard
+tests  test_consume_during_foreign_borrow_raises, test_unborrowed_consume_proceeds +
+
+ + + +
+ + diff --git a/demo/50-context-sign-callback.html b/demo/50-context-sign-callback.html new file mode 100644 index 00000000..93a24fc5 --- /dev/null +++ b/demo/50-context-sign-callback.html @@ -0,0 +1,158 @@ + + + + + +The signer callback freed mid-signature + + + +
+ +
All problems  /  50
+ +

The signer callback freed mid-signature

+

What gets freed is a function pointer Python created, which the library calls through while signing.

+ +
+ +
+
backgroundthe context is shared, and so is its callback
+
+ + + + + + a Python object has no owning thread — it belongs to whoever holds a reference + + + one Context + one native handle inside it + + thread A + thread B + + + same reference + same reference + + + signs through its callback + + closes it, dropping that reference + + nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough + + + the GIL stops both threads running Python at the same instant — it does not stop them + touching the same object, and it is given away entirely during a native call + +
The trampoline is kept alive by one attribute on the shared Context. Either thread can drop the last reference to it.
+
+
+ +
+
beforemain: close drops the callback, unguarded
+
+ + + + + + thread A + native + thread B + + + + + builder.sign() + + + + signing + + + calls back through + the trampoline + + context.close() + _signer_callback_cb = None + + last reference gone + + + trampoline collected + + next callback enters freed memory + +
Native holds the trampoline's address but no reference to it, so ordinary Python reference counting can free it mid-call.
+
+
+ +
+
afterthe context is held in flight for the duration of the sign
+
+ + + + + + thread A + native + thread B + + + + + builder.sign() + _context_guard(context) + + + + signing + + + trampoline pinned + for the whole call + + context.close() + + marks CLOSED, records + the release + + + sign completes + + + callback released here, by the last to leave + +
The same guard also refuses a sign that starts on an already-closed context, instead of signing without the signer.
+
+
+ +
+ +
+

Notes

+

Both threads reach the same object because both hold a reference to it; the GIL keeps them from running Python at the same instant but is handed away entirely during a native call — the GIL figure on page 20 shows what it does and does not cover.

+

What a trampoline is. To let native code call a Python function, ctypes builds a small object native can call like a C function. It is an ordinary Python object with ordinary reference counting, and nothing on the native side holds a reference to it. Keeping it alive as long as native might call it is the caller's job; here the Context holds that reference.

+

The quiet failure. If the callback is already gone when the sign begins, native signs without calling it and reports success. You get a file that looks signed and is not. The guard's validity check turns that into an exception.

+

The guard is duck-typed, so a caller-supplied context implementing only the published contract still works — at the cost of no in-flight protection. A test exists to ensure the built-in Context never falls into that unprotected branch.

+
+c2pa.py:4362-4372  the guarded context-sign
+c2pa.py:1945-1957  _context_guard — duck-typed on _native_call
+main:1722-1724  _release dropped the callback unconditionally
+tests  test_context_sign_after_close_raises_rather_than_skipping_signer, test_built_in_context_still_gets_in_flight_protection +
+
+ + + +
+ + diff --git a/demo/60-third-thread-gc.html b/demo/60-third-thread-gc.html new file mode 100644 index 00000000..c058c5a2 --- /dev/null +++ b/demo/60-third-thread-gc.html @@ -0,0 +1,145 @@ + + + + + +The thread that frees it never used it + + + +
+ +
All problems  /  60
+ +

The thread that frees it never used it

+

Every other page shows two threads doing something deliberate. This one shows a third thread that never touched the object and frees it anyway.

+ +
+ +
+
backgrounda free is not always something someone asked for
+
+ + + + + + nobody calls close() here — the free happens wherever the last reference dies + + thread A + thread B + thread C + + + + + + creates the Reader + + + reference + + + uses it, then returns + + + last reference + + + drops it — never used it + __del__ runs here, c2pa_free + + + measured: 12 readers created on four pool threads were all freed on four different pool threads + +
Reference counting destroys an object wherever its count reaches zero. Thread C may be a worker that only returned a value; the free still runs on its stack.
+
+
+ +
+
beforemain: that free is immediate, and lands wherever C happens to be
+
+ + + + + thread C, meanwhile, is busy with its own unrelated work + + + + C's own native-error window + + + C's call returns + + C reads its error + + + someone else's Reader is collected on C → c2pa_free + + + if untracked: slot overwritten + + C now reports a failure it never had + +
The object being freed and the thread doing the freeing are unrelated. C is damaged by work it never asked for.
+
+
+ +
+
afterthe gate is C's own state, not the object's
+
+ + + + + the same finalizer, on the same uninvolved thread + + + + C's window: depth > 0 + + + C's call returns + + C reads its error + message intact + + + teardown asks: is this thread in a section? + + + yes → queued on C's pending list + + + freed here + +
The check is on the freeing thread's state, never on the object's. That is the only thing that works when the freeing thread is arbitrary.
+
+
+ +
+ +
+

Notes

+

The freeing thread is arbitrary. It is tempting to assume whoever frees the object is one of the threads using it. A pool worker that merely returned a value can be the one running c2pa_free, so the design cannot rely on that assumption anywhere.

+

This is what forces the deferral gate to be thread-local. If the section were a property of the resource, thread C's finalizer would consult the wrong object's state entirely — the reader being freed is not the one C was working with.

+

The same applies to _released: several threads can reach the same teardown at once, so idempotency has to key on a flag, not on the lifecycle state, which the deferred path sets while the free is still owed.

+
+c2pa.py:439-477  _teardown — the gate is _in_native_section(), a thread property
+c2pa.py:479-502  _finish_teardown — idempotent via _released
+c2pa.py:1040  _native_section_state = threading.local()
+tests  test_third_thread_gc_of_dropped_reference_frees_exactly_once (200 resources, 4 workers,
+      asserts every handle freed exactly once), test_json_racing_finalizer_does_not_crash,
+      test_cross_thread_create_and_close_frees_exactly_once +
+
+ + + +
+ + diff --git a/demo/70-blocking-callback.html b/demo/70-blocking-callback.html new file mode 100644 index 00000000..86adec42 --- /dev/null +++ b/demo/70-blocking-callback.html @@ -0,0 +1,146 @@ + + + + + +The callback that waits for another thread + + + +
+ +
All problems  /  70
+ +

The callback that waits for another thread

+

This is the scenario that rules out the fix everyone reaches for first. A per-object lock deadlocks here, and so does a reentrant one.

+ +
+ +
+
backgroundwhat a stream callback is allowed to do
+
+ + + + + when you pass a file-like object, the library calls back into your Python code to read it + + + Reader(..., stream) + + + native runs + + + your readinto() is called + + + that callback is ordinary user code. It may block, take locks, start threads, wait for them, + or call back into this same library — the library cannot constrain any of it + + + so any lock held for the duration of such a call is a lock held across arbitrary user code, + for an unbounded time, that the user code itself may need + +
The library gives up control to user code in the middle of its own operation, and cannot bound what that code does.
+
+
+ +
+
beforethe obvious fix: hold the object's lock across the call
+
+ + + + + + thread A + helper thread + + + + + takes the lock, keeps it + + + its callback is called + + + starts a helper + + + helper: target.json() + wants the same lock + + + callback waits: helper.join() + + + helper waits for a lock A holds. A waits for the helper. Neither can move. + a reentrant lock does not help: the waiting party is a different thread + +
Reentrancy solves the same-thread case only. Here the blocked party is a second thread, so an RLock blocks it exactly as a plain lock would.
+
+
+ +
+
aftercount the call instead of locking across it
+
+ + + + + + thread A + helper thread + + + + + lock, _inflight += 1, unlock + + + callback runs, no lock held + + + starts a helper + + + helper: target.json() + acquires freely, finishes + + + join() returns, callback completes + + + the counter still tells a racing close() that work is in progress — the protection is kept, + without anything for the helper to block on + +
The counter provides the same guarantee as the lock without being something another thread can wait on.
+
+
+ +
+ +
+

Notes

+

A per-object lock does not work. It is the natural first answer to the races page 30 describes, and this is the case that rules it out. The test's own docstring puts it plainly: "A lock held across construction deadlocks here, whether it is global or per-object."

+

Note what is not the problem. This is not re-entrancy — that case is real and an RLock handles it. Here the callback does not take the lock itself; it waits for a different thread that needs it. No lock design survives that, because the deadlock is between two threads with a cycle through user code the library never sees.

+

test_stream_callback_blocking_on_other_thread_does_not_deadlock builds it exactly: a readinto that starts a helper touching the same reader, joins it with a ten-second timeout, and records a failure if the helper is still alive. It runs the construction five times over.

+
+c2pa.py:346-363  _native_call — lock only around the counter, never across the call
+c2pa.py:332-344  _lock — "Never hold this across a native call that drives stream callbacks"
+c2pa.py:3303-3307  with_fragment — the same reasoning, via a non-blocking acquire
+tests  test_stream_callback_blocking_on_other_thread_does_not_deadlock,
+      test_stream_callback_reentering_api_does_not_deadlock, test_concurrent_storm_terminates +
+
+ + + +
+ + diff --git a/demo/index.html b/demo/index.html new file mode 100644 index 00000000..aca83671 --- /dev/null +++ b/demo/index.html @@ -0,0 +1,67 @@ + + + + + +What this branch fixes + + + +
+ +

What this branch fixes

+

Seven problems in the Python C2PA wrapper, one page each: a diagram of what went wrong before, a diagram of what the fix does, and a short note underneath.

+ +

Most of these corrupt native memory rather than raise an exception, so the visible symptom is a crash somewhere unrelated, a hang, or output that is quietly wrong. On main, ManagedResource.__init__ is three assignments: no lock, no record of calls in progress, no deferred cleanup.

+ +
+ + +
10
+
The error that belonged to someone else memory
+

The library leaves error messages in a slot nothing ever clears, so a call that fails without writing its own reports the previous one — and a stale message about pointer ownership makes the wrapper free memory twice.

+
+ + +
20
+
The native section concept
+

A critical section protects data from other threads. This protects a stretch of time on one thread: the gap between a call returning and its error being read. A finalizer for an unrelated object, running in that gap, destroys the message.

+
+ + +
30
+
Closing something another thread is using memory
+

A close() on one thread frees a handle another thread has already passed into a native call. The obvious fix — hold a lock — deadlocks, because those calls run your own code.

+
+ + +
40
+
A consume that starts during a borrow memory
+

Deferring a close does not help when the free happens inside the library, during a different call. A borrowing call validates its pointer once and never re-checks, so the usual protection never fires.

+
+ + +
50
+
The signer callback freed mid-signature memory
+

What gets freed is a function pointer Python created, which native calls through while signing. The quiet version signs the file without ever calling the signer, and reports success.

+
+ + +
60
+
The thread that frees it never used it memory
+

Every other page shows two threads doing something deliberate. Reference counting frees an object wherever its last reference dies — which can be a pool worker that only returned a value and never touched it.

+
+ + +
70
+
The callback that waits for another thread
+

The scenario that rules out the fix everyone reaches for first. A callback starts a helper thread that needs the same object and waits for it; a per-object lock deadlocks, and a reentrant one does too.

+
+ +
+ +

Code references are to src/c2pa/c2pa.py; the “before” quotes are from git show main:src/c2pa/c2pa.py.

+ +
+ + diff --git a/demo/style.css b/demo/style.css new file mode 100644 index 00000000..1f9727af --- /dev/null +++ b/demo/style.css @@ -0,0 +1,354 @@ +:root { + color-scheme: light dark; + --bg: #fbfaf8; + --fg: #1c1a17; + --muted: #5d5750; + --rule: #ddd7cf; + --card: #ffffff; + --code-bg: #f4f1ec; + --accent: #b3261e; + --accent-soft: rgba(179, 38, 30, 0.12); + --ok: #1c6b4a; + --ok-soft: rgba(28, 107, 74, 0.12); +} + +@media (prefers-color-scheme: dark) { + :root:not([data-theme="light"]) { + --bg: #16151a; + --fg: #eae7e2; + --muted: #a49e97; + --rule: #35323a; + --card: #1e1d23; + --code-bg: #232228; + --accent: #ff8f84; + --accent-soft: rgba(255, 143, 132, 0.16); + --ok: #6cc79b; + --ok-soft: rgba(108, 199, 155, 0.16); + } +} + +:root[data-theme="dark"] { + --bg: #16151a; + --fg: #eae7e2; + --muted: #a49e97; + --rule: #35323a; + --card: #1e1d23; + --code-bg: #232228; + --accent: #ff8f84; + --accent-soft: rgba(255, 143, 132, 0.16); + --ok: #6cc79b; + --ok-soft: rgba(108, 199, 155, 0.16); +} + +* { box-sizing: border-box; } + +body { + margin: 0; + background: var(--bg); + color: var(--fg); + font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif; + -webkit-font-smoothing: antialiased; +} + +.wrap { + max-width: 46rem; + margin: 0 auto; + padding: 3rem 1.25rem 6rem; +} + +.crumb { + font-size: 0.8rem; + color: var(--muted); + margin-bottom: 2rem; + letter-spacing: 0.02em; +} + +.crumb a { color: var(--muted); } + +h1 { + font-size: 1.95rem; + line-height: 1.2; + margin: 0 0 0.4rem; + letter-spacing: -0.02em; +} + +.standfirst { + font-size: 1.08rem; + color: var(--muted); + margin: 0 0 2.6rem; + line-height: 1.55; +} + +h2 { + font-size: 1.18rem; + margin: 3rem 0 0.9rem; + padding-top: 1.4rem; + border-top: 1px solid var(--rule); + letter-spacing: -0.01em; +} + +h3 { + font-size: 1rem; + margin: 2rem 0 0.6rem; +} + +p { margin: 0 0 1rem; } + +a { color: inherit; text-decoration-color: var(--rule); text-underline-offset: 2px; } +a:hover { text-decoration-color: currentColor; } + +code { + font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + font-size: 0.87em; + background: var(--code-bg); + padding: 0.1em 0.34em; + border-radius: 3px; +} + +pre { + background: var(--code-bg); + border: 1px solid var(--rule); + border-radius: 6px; + padding: 0.9rem 1rem; + overflow-x: auto; + margin: 0 0 1rem; +} + +pre code { background: none; padding: 0; font-size: 0.8rem; line-height: 1.55; } + +.filename { + font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + font-size: 0.74rem; + color: var(--muted); + margin-bottom: 0.35rem; + letter-spacing: 0.01em; +} + +figure { margin: 2rem 0; } + +figure svg { + display: block; + width: 100%; + max-width: 100%; + height: auto; + color: var(--fg); +} + +figcaption { + font-size: 0.85rem; + color: var(--muted); + margin-top: 0.85rem; + line-height: 1.5; +} + +ol, ul { margin: 0 0 1rem; padding-left: 1.4rem; } +li { margin-bottom: 0.5rem; } + +.steps { counter-reset: step; list-style: none; padding-left: 0; } + +.steps li { + counter-increment: step; + position: relative; + padding-left: 2.1rem; + margin-bottom: 0.8rem; +} + +.steps li::before { + content: counter(step); + position: absolute; + left: 0; + top: 0.08rem; + width: 1.45rem; + height: 1.45rem; + border-radius: 50%; + background: var(--code-bg); + border: 1px solid var(--rule); + color: var(--muted); + font-size: 0.76rem; + font-weight: 600; + display: flex; + align-items: center; + justify-content: center; +} + +.steps li.bad::before { + background: var(--accent-soft); + border-color: var(--accent); + color: var(--accent); +} + +.note { + border-left: 3px solid var(--rule); + padding: 0.15rem 0 0.15rem 1rem; + margin: 1.5rem 0; + color: var(--muted); + font-size: 0.94rem; +} + +.note.warn { border-left-color: var(--accent); } +.note strong { color: var(--fg); } + +.tests { list-style: none; padding-left: 0; } + +.tests li { + padding: 0.6rem 0; + border-bottom: 1px solid var(--rule); + font-size: 0.93rem; +} + +.tests li:first-child { border-top: 1px solid var(--rule); } + +.tests .tname { + font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + font-size: 0.8rem; + display: block; + margin-bottom: 0.15rem; +} + +.tests .twhat { color: var(--muted); font-size: 0.88rem; } + +table { border-collapse: collapse; width: 100%; font-size: 0.9rem; margin: 0 0 1rem; } +th, td { text-align: left; padding: 0.55rem 0.7rem 0.55rem 0; border-bottom: 1px solid var(--rule); vertical-align: top; } +th { font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted); font-weight: 600; } + +.scroll { overflow-x: auto; } + +.pagenav { + display: flex; + justify-content: space-between; + gap: 1rem; + margin-top: 4rem; + padding-top: 1.4rem; + border-top: 1px solid var(--rule); + font-size: 0.9rem; +} + +.pagenav a { color: var(--muted); } +.pagenav a:hover { color: var(--fg); } + +/* index */ +.cards { display: grid; gap: 0; margin-top: 2rem; } + +.card { + display: block; + padding: 1.3rem 0; + border-top: 1px solid var(--rule); + text-decoration: none; + color: inherit; +} + +.card:last-child { border-bottom: 1px solid var(--rule); } +.card:hover .card-title { text-decoration: underline; text-underline-offset: 3px; } + +.card-num { + font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + font-size: 0.74rem; + color: var(--muted); +} + +.card-title { font-size: 1.05rem; font-weight: 600; margin: 0.2rem 0 0.35rem; } +.card-desc { font-size: 0.92rem; color: var(--muted); margin: 0; line-height: 1.55; } + +.tag { + display: inline-block; + font-size: 0.7rem; + letter-spacing: 0.04em; + text-transform: uppercase; + padding: 0.15rem 0.45rem; + border-radius: 3px; + border: 1px solid var(--rule); + color: var(--muted); + margin-left: 0.5rem; + vertical-align: 0.1rem; +} + +.tag.crash { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); } + +/* diagram-first page format */ +.diagrams { margin: 2.5rem 0 0; } + +.panel { margin: 0 0 2.6rem; } + +.panel-label { + display: flex; + align-items: baseline; + gap: 0.6rem; + margin-bottom: 0.7rem; +} + +.panel-tag { + font-size: 0.7rem; + letter-spacing: 0.08em; + text-transform: uppercase; + font-weight: 700; + padding: 0.18rem 0.5rem; + border-radius: 3px; +} + +.panel-tag.before { color: var(--accent); background: var(--accent-soft); } +.panel-tag.after { color: var(--ok); background: var(--ok-soft); } + +.panel-claim { font-size: 0.95rem; color: var(--muted); } + +.panel figure { margin: 0; } +.panel figcaption { margin-top: 0.6rem; } + +.footnote { + margin-top: 3rem; + padding-top: 1.4rem; + border-top: 1px solid var(--rule); + font-size: 0.92rem; + color: var(--muted); +} + +.footnote p { margin: 0 0 0.7rem; } +.footnote strong { color: var(--fg); } +.footnote code { font-size: 0.85em; } + +.refs { + margin-top: 1.2rem; + font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + font-size: 0.74rem; + color: var(--muted); + line-height: 1.9; +} + +.panel-tag { color: var(--muted); background: var(--code-bg); } + +.panel-note { + margin-top: 1.1rem; + font-size: 0.92rem; + color: var(--muted); + line-height: 1.6; +} + +.panel-note p { margin: 0 0 0.8rem; } +.panel-note p:last-child { margin-bottom: 0; } +.panel-note strong { color: var(--fg); } + +.panel-note pre { + margin: 0.9rem 0; + background: var(--code-bg); +} + +.bridge { + margin: 0 0 2.6rem; + padding-left: 1rem; + border-left: 3px solid var(--rule); + font-size: 0.94rem; + color: var(--muted); + line-height: 1.6; +} + +.panel-tag.also { color: var(--muted); background: var(--code-bg); } + +.footnote h2 { + font-size: 0.78rem; + text-transform: uppercase; + letter-spacing: 0.08em; + color: var(--muted); + margin: 0 0 0.9rem; + padding: 0; + border: 0; + font-weight: 600; +} diff --git a/docs/context-settings.md b/docs/context-settings.md index 35692a82..bd950fb9 100644 --- a/docs/context-settings.md +++ b/docs/context-settings.md @@ -752,6 +752,8 @@ ctx = Context() assert isinstance(ctx, ContextProvider) # True ``` +A provider that does not derive from `ManagedResource` runs without in-flight teardown protection. `Reader` and `Builder` check `is_valid` before use, but nothing defers a teardown that arrives mid-construction, so closing such a provider on another thread while a `Reader` or `Builder` is being built from it can free the native context while that construction is still using it. The built-in `Context` carries that protection. Custom providers that share a context across threads should keep it alive for the duration of any construction that uses it. + ## Migrating from load_settings The `load_settings()` function is deprecated. Replace it with `Settings` and `Context` APIs: diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index 1cf057f0..20a669c9 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -99,22 +99,195 @@ Python owns and frees two kinds of things: the **single current native handle** Therefore, the managed resources have the following principles: -- Each `ManagedResource` holds exactly one `_handle`. `_swap_handle()` replaces it with the pointer a consuming call returned and does not free the old value, since the native side took it (see [Consume-and-swap](#consume-and-swap)). +- Each `ManagedResource` holds exactly one `_handle`. `_consume_and_swap()` replaces it with the pointer a consuming call returned and does not free the old value, since the native side took it (see [Consume-and-swap](#consume-and-swap)). - `_teardown(free_handle=False)`, `_consume_no_replacement()`, and `_consume_into()` all close or advance the object without calling `c2pa_free`, because ownership moved to the native side. -- Only a few sites free a live handle. Two of them free a pointer this layer still provably owns: normal teardown (`_teardown(free_handle=True)`), and the create-then-validate path, which frees a freshly created pointer if activation fails. The third, `_release_handle()`, is a *guarded* free used only when ownership is genuinely unknown (a consuming call failed without setting an error, or a Python exception was raised before the native side reported anything): if the native side already took the pointer, its address is no longer in the registry and `c2pa_free` is a `-1` no-op, so the free touches no memory. No path frees a pointer known to have been consumed and reallocated (see [Why an ownership-taken failure does not free](#why-an-ownership-taken-failure-does-not-free)). +- Only a few sites free a live handle, and most free a pointer this layer still provably owns: normal teardown (`_teardown(free_handle=True)`); the create-then-validate path, which frees a freshly created pointer if activation fails; and the constructors that free a raw pointer when wrapping it raises, since no instance took ownership (`Signer.from_info`, `Signer.from_callback`, `Builder.from_archive`). The exception is `_release_handle()`, a *guarded* free used only when ownership is unknown (a consuming call failed without setting an error, or a Python exception was raised before the native side reported anything): if the native side already took the pointer, its address is no longer in the registry and `c2pa_free` is a `-1` no-op, so the free touches no memory. No path frees a pointer known to have been consumed and reallocated (see [Why an ownership-taken failure does not free](#why-an-ownership-taken-failure-does-not-free)). - `_release()` drops stream wrappers, callbacks, and caches before the native pointer is freed (see [Subclass-specific cleanup with `_release()`](#subclass-specific-cleanup)). ### Double-free risk mitigations -Three distinct risks. Two have a mechanism in this layer; the third is the caller's to synchronize: +Freeing the same native pointer twice corrupts the allocator's bookkeeping. Allocators that detect it stop the process (see [How a native memory bug reports itself](#how-a-native-memory-bug-reports-itself)). Where they do not, the damage surfaces later, somewhere unrelated to the code responsible. Separate situations lead here: a single flow misreading who owns a pointer, a forked child freeing its parent's memory, and two threads racing. + +Each risk and its mechanism: | Hazard | Covered by | How | | --- | --- | --- | -| Freeing a pointer a consuming call already took (single flow) | `_swap_handle` / `_teardown(free_handle=False)` triage | The consumed pointer is abandoned, never freed. The retained-vs-consumed decision reads the native error tag (`UntrackedPointer:` / `WrongPointerType:` mean not taken). | +| Freeing a pointer a consuming call already took (single flow) | `_consume_and_swap` / `_teardown(free_handle=False)` triage | The consumed pointer is abandoned, never freed. The retained-vs-consumed decision reads the native error tag (`UntrackedPointer:` / `WrongPointerType:` / `NullParameter:` / `InvalidBufferSize:` mean not taken). | | A forked child freeing a pointer its parent owns | PID stamp (`record_owner_pid` / `is_foreign_process`) | Cleanup in a process that did not allocate the pointer nulls the handle and marks `CLOSED` without freeing (see [Fork safety](#fork-safety)). | -| Two **threads** in one process racing frees on distinct objects, where the allocator recycles a just-freed address | Not covered here | `ManagedResource` has no lock and no thread stamping. The PID stamp cannot see it: sibling threads share a PID. Safety for genuinely shared handles must come from the caller's own synchronization or from the native registry, not this layer. | +| Two **threads** racing a `close()` against an in-flight native call on the same object, where the allocator recycles a just-freed address | `_op_lock` / `_native_call()` / `_pending_teardown` | A close arriving while a native call is in flight is recorded rather than applied. The last caller to leave `_native_call()` performs the deferred free (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)). | + +The PID stamp is fork-only: it compares process IDs, and two threads in the same process always match on that PID. Sharing one `ManagedResource` instance across threads still needs locks: nothing here protects two threads racing on genuinely distinct objects that happen to share an allocator. + +## Streams + +Bytes reach the native library through a `Stream`. + +`Stream` wraps a Python stream-like object (file stream or memory stream) so the native library can read from and write to it via callbacks. It does not inherit from `ManagedResource`, and it uses `c2pa_release_stream()` instead of `c2pa_free()` for cleanup. + +### Why is `Stream` not a `ManagedResource`? + +A `Reader` or `Builder` holds a native resource that Python code calls methods on. A `Stream` holds a native handle that the native library calls back into (read, seek, write, flush). The native library needs a different release function to tear down the callback machinery, since ownership has a different meaning here (due to the callbacks). + +`Stream` tracks its own state with `_closed` and `_initialized` flags rather than `LifecycleState`, but it supports the same three cleanup paths: context manager, explicit `.close()`, and `__del__` fallback. + +### Callbacks re-enter Python + +A `Stream` registers four ctypes callbacks (`_read_cb`, `_seek_cb`, `_write_cb`, `_flush_cb`). When the native library reads from a stream, it calls one of them, which runs caller-supplied Python. + +A native call driving a stream callback cannot hold `_op_lock`. The callback may re-enter this API on the same thread and deadlock against the lock already taken (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)). + +The callback objects must also outlive the native side's use of them, covered in [Preventing garbage collection of live references](#preventing-garbage-collection-of-live-references). + +Each callback checks `_initialized` and `_closed` before touching the underlying Python stream, and returns `-1` if the stream is not initialized or already closed. A callback arriving after teardown reports an error instead of reading through dropped state. + +### `Stream` cleanup + +`Stream` holds `_close_lock`, an `RLock`. It serializes the three cleanup paths: `close()`, `__del__`, and a `close()` on another thread. Without it, two of them reach the same stream and call `c2pa_release_stream` twice on one native handle. `Stream` needs its own lock since it does not inherit the `_op_lock` machinery. + +The lock is reentrant for the same reason `_op_lock` is. `close()` sets the four callback attributes to `None` inside the locked region, which can drop the last reference to an object whose finalizer runs at that bytecode boundary. `__del__` takes the same lock. A plain `Lock` deadlocks against itself when that finalizer belongs to the stream being closed. + +Cleanup runs in the direction of dependency: whatever can still invoke or reach the other is torn down first. Because callbacks run the opposite way for a `Stream`, its close order is the reverse of `ManagedResource`'s: + +| | Order | Why | +| --- | --- | --- | +| `ManagedResource` | `_release()` first (dropping streams and callbacks), then `c2pa_free` | The native pointer depends on the Python-side resources, so those are torn down while the pointer is still valid. | +| `Stream` | `c2pa_release_stream` first, then drop the callbacks | The native stream invokes the callbacks. Releasing it first guarantees none can fire, and the callback objects are dropped after that. | + +`close()` performs both steps: it calls `c2pa_release_stream`, then sets `_read_cb`, `_seek_cb`, `_write_cb`, and `_flush_cb` to `None`. + +`__del__` performs only the first. It calls `c2pa_release_stream` and leaves the four attributes pointing at their callbacks. Nothing leaks, because `__del__` runs when the `Stream` is being collected: the attributes are freed with the object that holds them. Explicit cleanup drops them itself rather than waiting for collection. + +`Stream` does not own the Python object it wraps and never closes it. The caller that opened a file owns that file. A `Reader` that opened a file itself tracks it as `_backing_file` and closes it during its own `_release()`. + +Both `close()` and `__del__` take the foreign-process branch, marking the stream closed without calling into the native library. [Fork safety](#fork-safety) covers why. + +Both check `is_foreign_process()` before acquiring `_close_lock`, as `_teardown()` and `_lock()` do. A child inherits the lock in whatever state it had at `fork()`, and the thread holding it does not exist there to release it, so a child that acquired first would wait on it forever. + +### Reference cycles in the callbacks + +Each ctypes callback closes over the `Stream` it belongs to. Captured directly, that forms a cycle: the `Stream` holds the callback, the callback's closure holds the `Stream`. Nothing in that loop reaches a refcount of zero, so cleanup falls to the cycle collector. [Why `__del__` is not reliable enough](#why-__del__-is-not-reliable-enough) covers why that timing cannot be relied on. + +The closures capture a `weakref` to the `Stream` instead, resolving it on each call. The reference count can then reach zero, keeping cleanup on the deterministic path. + +## Threads, the GIL, and locks + +### How a native memory bug reports itself + +A Python bug raises an exception with a traceback. A native memory bug terminates the process, and the operating system reports it as a signal. + +SIGSEGV (segmentation violation) comes from the hardware. Every process has a map of which address ranges it may touch and how. The CPU traps when asked to read or write an unmapped address, or to write one mapped read-only. The kernel then sends SIGSEGV. A segmentation fault, or segfault, is that event. SIGSEGV is the signal delivered for it. It fires whenever a dereferenced pointer does not point at accessible memory. A null pointer does it. So does a pointer into a stack frame that has already returned, or into memory that was freed and unmapped. Freeing memory does not always unmap it. Allocators keep the pages and reuse them. Reading through a freed pointer therefore often succeeds, returning whatever occupies that address now. The read returns another object's bytes, corrupting state that surfaces far from the code responsible. SIGSEGV is another outcome when the pages happen to be gone. + +SIGABRT (abort) has a different origin. No hardware raises it. The program raises it against itself by calling `abort()`, on detecting that its own invariants are broken. Allocators do this. A `free()` handed a pointer it never issued, or the same pointer twice, or a heap whose bookkeeping a stray write has damaged, stops the process rather than continuing on a corrupt heap. + +Which signal arrives depends on the allocator. glibc can print a diagnostic and raises SIGABRT; the macOS allocator traps instead, giving SIGTRAP. A use-after-free that reaches unmapped pages gives SIGSEGV on both. + +Both terminate the process from inside native code. No exception is raised, no `finally` block runs, no traceback is printed. + +### A close arriving mid-call + +A race here is two threads, one object, one native pointer, and operations that take several steps. Failure follow roughly these steps: + +1. Thread A calls `reader.json()` (for instance), which validates the handle and enters the native call. +2. While that call is running, thread B calls `reader.close()` (for instance, close on same object), which frees the native pointer. +3. Thread A's native code, still running, reads through the pointer it was given. + +Step 3 reads freed memory. The process takes SIGSEGV if those pages are gone. If the allocator kept them and reissued that address, the read succeeds and returns another object's bytes. Both happen inside native code. + +### What the GIL can guarantee + +CPython compiles source into bytecode: a sequence of small instructions the interpreter executes one at a time. A single line of Python becomes several of them. `self._ensure_valid_state()` compiles to four (load `self`, load the attribute, call, discard the result). + +CPython has a Global Interpreter Lock, a single interpreter-wide lock. Only one thread executes bytecode at a time. One instruction cannot interleave with another, so built-in containers do not corrupt structurally under concurrent access. + +Anything longer than one instruction gets no such guarantee. The interpreter can switch threads between any two instructions, so a line of Python can be interrupted partway. + +### Why the GIL is not enough here + +ctypes releases the GIL for the duration of a foreign function call. Other Python threads then run in parallel with the native code. Every native call in this layer opens that window, and a `close()` can arrive inside it. [A close arriving mid-call](#a-close-arriving-mid-call) walks through that race. + +The check-then-use sequences here span many instructions. `_ensure_valid_state()` validates the handle; a later instruction loads `self._handle` and passes it to native. The interpreter can switch threads in between, so a passing check does not promise the handle is still live at the point of use. `_op_lock` makes those pairs one unit. + +`_native_call()` closes the window by making the free wait for the call to finish: + +```mermaid +flowchart TD + subgraph guarded [Guarded by _native_call] + direction TB + C1["Thread A: enters guard
_inflight becomes 1"] --> C2["Thread A: native call runs"] + D1["Thread B: close()"] --> D2{"_inflight nonzero?"} + D2 -->|yes| D3["Record _pending_teardown,
mark CLOSED, free nothing"] + C2 --> C3["Thread A: leaves guard
_inflight becomes 0"] + D3 -.->|"free deferred to
whoever leaves last"| C3 + C3 --> C4["Deferred free runs
pointer no longer in use"] + end +``` + +### Why races are observed + +A race on a native pointer produces a use-after-free or a double-free, which is a crash or silent memory corruption, and the corruption can surface arbitrarily far from its cause. [Double-free risk mitigations](#double-free-risk-mitigations) covers the three specific hazards and the mechanism for each. + +> [!NOTE] +> Free-threaded CPython builds remove the GIL entirely, so code that was accidentally relying on it for atomicity loses that protection anyway. + +## Locking and in-flight tracking -The PID stamp is fork-only: it compares process IDs, and two threads in the same process always match. Sharing one `ManagedResource` instance across threads without external synchronization is outside what this layer protects against. +Each `ManagedResource` holds a reentrant lock, `_op_lock`, and a counter, `_inflight`, that together serialize teardown against concurrent use from other threads. + +A lock (Python's `threading.Lock`) can be acquired once, and a second `acquire()` from the same thread on that lock blocks forever, waiting on a lock that thread itself is holding. A reentrant lock (`threading.RLock`) tracks which thread holds it and how many times: the owning thread can acquire it again without blocking, and the lock is only released once that thread has released it the same number of times it acquired it. A different thread still blocks until the owner releases fully. + +`_op_lock` is an `RLock` rather than a plain `Lock` for two reasons specific to this code. First, a finalizer (`__del__`) can run at any bytecode boundary — including one in the middle of a method that has already acquired the lock on this same thread — so `__del__` calling back into locked code must not deadlock against itself. Second, a consuming call tears the handle down from inside the locked region it is already holding: `_teardown()` is called while `_op_lock` is held, and it needs to acquire the same lock again rather than re-entering as a different, blocked acquisition. `_lock()` returns it, except in a forked child: there it raises `C2paError` immediately rather than blocking, because the thread that might hold the lock at fork time does not exist in the child to release it, and waiting on it would hang forever (see [Fork safety](#fork-safety)). + +The lock is never held across a native call that drives a stream callback: construction, `resource_to_stream`, the Builder stream methods, and signing all release the GIL and re-enter caller-supplied Python, which may itself call into this API on another thread (see [Callbacks re-enter Python](#callbacks-re-enter-python)). Holding `_op_lock` there would deadlock against that reentry. Those calls go through `_native_call()` instead: a context manager that increments `_inflight` under the lock, yields to run the native call unlocked, then decrements `_inflight` on the way out. If `_teardown()` runs while a call is in flight, it records the requested `free_handle` value in `_pending_teardown` and marks the resource `CLOSED` immediately, so no other caller can start using it, but defers the actual free. The last `_native_call()` to exit picks up `_pending_teardown` and runs `_teardown()` for real. + +A recorded `_pending_teardown` only ever moves from `True` to `False`, never back. When one caller records a teardown that frees and another records one that does not, the one that does not wins. A pointer the native side already took is never freed on the way out. + +Cleanup idempotency keys on a `_released` flag rather than on the `CLOSED` state. The deferred path marks the resource `CLOSED` when it records the teardown, while still owing the release that runs later. + +`Context.__init__` does not wrap the signer hand-off in `signer._native_call()`: the consuming call marks the signer `CLOSED` under its own lock before calling native, which is what stops a `signer.close()` on another thread from freeing the handle mid-transfer (see [Borrowing versus consuming](#borrowing-versus-consuming)). The Builder's `close()` after signing runs outside its own `_native_call()` block, so a teardown deferred during the call still executes once the call returns. + +### Lock ordering + +Two threads acquiring the same pair of locks in opposite orders deadlock: each holds what the other waits for. A single order of lock acquisition avoids the issue. + +An operation holds guards by role, from outermost to innermost: + +1. A method-specific lock, where a method serializes itself against other calls to itself. +2. The guard of the object the method is called on. +3. The guard of any object it borrows for the duration of the call. + +Rules that follow from the order: + +- A borrowed object's guard is always taken inside the operating object's guard, never the reverse. Two concurrent operations sharing a borrowed object then acquire in one direction. +- A lock held across a native call must be one no callback path acquires. +- `Stream._close_lock` is a leaf: nothing is acquired while holding it, so it cannot join a cycle. + +No code path holds two resources' `_op_lock` at once. `_native_call()` counts a call as in flight instead of holding the lock across it, which is what keeps that true. `test_no_nested_op_locks` checks it over the Reader read path by intercepting `_lock()` and recording any acquisition made while another is held. + +### Borrowing versus consuming + +Deferring a teardown protects a consuming call against a racing `close()`. It does not protect a borrowing call against a racing consume, which is a different risk with a different mitigation. + +A borrowing call passes the handle to native and gets it back unchanged. A consuming call hands ownership over, and the native side frees the pointer during the call. A borrowing call validates the pointer once on entry, then holds it for the duration of the operation. The pointer registry is never consulted again. So a consume starting midway through a borrow frees memory the borrowing call is still reading, and the usual `-1` rejection never happens because validation already succeeded. + +`ManagedResource` therefore refuses the consume rather than allowing it to start. `_begin_consume()` runs `_ensure_not_borrowed()`, which rejects the call when `_inflight` is nonzero, and then marks the resource `CLOSED` before releasing `_op_lock`. + +The check catches a borrow already in flight. The `CLOSED` mark catches one arriving afterwards: `_native_call()` calls `_ensure_valid_state()` under the same lock, so a borrow that starts later is refused instead of reaching a pointer about to be freed. The lock cannot simply be held across the native call, because those calls run caller-supplied stream callbacks that re-enter this API. + +The mark is provisional. `_abort_consume()` restores the previous state when the native call turns out not to have taken the handle, which keeps the retained branch of the [ownership-taken triage](#why-an-ownership-taken-failure-does-not-free) handing back a usable object. + +`_raise_consume_failure()` performs that restore, on the pre-consume branch only. The reservation is held until the branch is known. `_read_native_error()` is itself a native call and releases the GIL, so a resource restored to `ACTIVE` before the error is classified is visible as usable to another thread while the native side may already own its handle. + +`_consume_and_swap()` is excluded. `_consume_and_swap()` requires the resource to stay `ACTIVE` and the object remains usable with its replacement pointer, so there is no `CLOSED` mark to make and no check. Its callers (`Reader.with_fragment`, `Builder.with_archive`) pass streams whose callbacks re-enter this API, so they hold their own `_native_call()`. `Reader.with_fragment()` additionally serializes itself with a lock of its own, described in [`Reader.with_fragment()`](#readerwith_fragment). + +### Context lifetime during a context-sign + +A `Context` built with a signer keeps that signer alive after consuming it. `Context.__init__` copies the signer's ctypes callback into `_signer_callback_cb`, because the `Signer` object is closed by the consuming call while the native side still needs the trampoline to invoke. `Context._release()` drops that reference. + +`c2pa_builder_sign_context` calls back into that trampoline for the duration of the sign, so the Context has to stay alive across it just as a borrowed `Signer` does. `Builder._sign_internal` therefore wraps the call in `self._context._native_call()`. A `close()` arriving on another thread then takes the deferred branch: it records the teardown, marks the Context `CLOSED`, and leaves `_release()` for whichever caller leaves the guard last. The callback stays pinned and the handle stays valid until the sign finishes. + +A sign already in flight is never cut short. Closing a Context mid-sign is safe; the close takes effect once the call returns. Without the guard, `_release()` frees the ctypes trampoline while native is calling through it, and the process takes SIGSEGV rather than raising. + +A sign cannot start once the Context is closed, and raises `C2paError` instead. A released Context has already dropped the callback, so the native side signed without ever invoking it. An error is the only way that is visible to the caller. ## Guarantees provided by ManagedResource @@ -126,18 +299,20 @@ The PID stamp is fork-only: it compares process IDs, and two threads in the same | **Cleanup is idempotent** | Calling `close()` (or exiting a `with` block) multiple times is safe; after the first successful cleanup, further calls do nothing. | | **Cleanup never raises (ordinary errors)** | The cleanup path catches and logs `Exception`, never re-raising it. `_release()` runs inside `_safe_release()`, which logs and swallows; the `c2pa_free` call has its own handler; and `_cleanup_resources()` wraps both. The original exception from the `with` block (if any) is never masked. **Asynchronous interrupts are the deliberate exception.** The cleanup handlers catch `Exception`, which excludes the `BaseException` signals the interpreter raises to unwind a process (a cancellation request or an exit in progress). Those propagate through cleanup untouched, and the remaining free may not run. Such a signal means the process is being torn down and its address space, native allocations included, is about to be reclaimed as a whole. Catching it would suppress a shutdown the caller asked for in order to complete a free that is about to become irrelevant, so the handlers stay scoped to `Exception`. | | **State transitions are one-way** | Lifecycle moves only from UNINITIALIZED to ACTIVE to CLOSED. A closed resource cannot be reactivated. | -| **Transitions go through helper methods** | Subclasses call `_activate()`, `_swap_handle()` or `_teardown()` and never assign `_handle` or `_lifecycle_state` directly. `_activate()` and `_swap_handle()` validate before mutating, so an object cannot end up active with a null handle. | +| **Transitions go through helper methods** | Subclasses call `_activate()`, `_consume_and_swap()` or `_teardown()` and never assign `_handle` or `_lifecycle_state` directly. `_activate()` and `_consume_and_swap()` validate before mutating, so an object cannot end up active with a null handle. | | **Ownership transfer is safe** | When a pointer is transferred elsewhere (e.g. via `_teardown(free_handle=False)`), the object stops managing it and does not call `c2pa_free` on it. | -| **Public methods validate lifecycle state** | Every public API calls `_ensure_valid_state()` before use; closed or invalid state yields `C2paError` instead of undefined behavior or crashes. | +| **Public methods validate lifecycle state** | Every public method that uses the handle calls `_ensure_valid_state()` before doing so; closed or invalid state yields `C2paError` instead of undefined behavior or crashes. The exceptions touch no handle: `is_valid` reports the state rather than requiring it, and the `get_supported_mime_types` classmethods query the library itself. | ## Preventing garbage collection of live references When a Python object passes a callback or pointer to the native library, that reference must stay alive for as long as the native side might use it. Python's garbage collector has no way to know that native code is still holding a reference to a Python callback. -The SDK solves this by storing these references as instance attributes on the owning object. For example, `Stream` stores its four callback objects (`_read_cb`, `_seek_cb`, `_write_cb`, `_flush_cb`) as instance attributes. As long as the `Stream` object is alive, its callbacks have a nonzero reference count and will not be collected. Similarly, when a `Signer` is consumed by a `Context`, the Context copies the signer's `_callback_cb` to its own `_signer_callback_cb` attribute so the callback survives even though the Signer object is now closed. +The SDK solves this by storing these references as instance attributes on the owning object. For example, `Stream` stores its four callback objects (`_read_cb`, `_seek_cb`, `_write_cb`, `_flush_cb`) as instance attributes. As long as the `Stream` object is alive, its callbacks have a nonzero reference count and will not be collected (see [Streams](#streams) for how those callbacks avoid forming a reference cycle with the `Stream` itself). Similarly, when a `Signer` is consumed by a `Context`, the Context copies the signer's `_callback_cb` to its own `_signer_callback_cb` attribute so the callback survives even though the Signer object is now closed. During cleanup, `_release()` sets these attributes to `None`, which drops the reference count on the callback objects and allows them to be collected. In the cleanup sequence, `_release()` runs first, then `c2pa_free` frees the native pointer. `_release()` goes first so that subclass-specific resources (open file handles, stream wrappers) are torn down before the native pointer they depend on is freed. +This ordering applies to `ManagedResource`. `Stream` releases in the opposite order, and [`Stream` cleanup](#stream-cleanup) explains why each side is correct for the direction its callbacks run. + ## How native memory is freed The native Rust library exposes a single C FFI function, `c2pa_free`, that deallocates memory it previously allocated. `ManagedResource` wraps this in a static method: @@ -164,7 +339,7 @@ stateDiagram-v2 [*] --> UNINITIALIZED : __init__() UNINITIALIZED --> ACTIVE : _activate(handle) UNINITIALIZED --> CLOSED : close() before activation - ACTIVE --> ACTIVE : _swap_handle(new_handle) + ACTIVE --> ACTIVE : _consume_and_swap(new_handle) ACTIVE --> CLOSED : close() / __exit__ / __del__ / _teardown() ``` @@ -179,9 +354,9 @@ Each transition has one method that performs it, and subclasses must go through | Method | Transition | What it enforces | | --- | --- | --- | | `_activate(handle)` | UNINITIALIZED to ACTIVE | Rejects a null handle, and refuses to run on an already-activated resource. A rejected activation leaves the object exactly as it was. | -| `_swap_handle(new_handle)` | ACTIVE to ACTIVE | Requires the resource to already be active and the replacement to be non-null. Used when an FFI call consumed the old handle and returned a new one. | -| `_teardown(free_handle=False)` | ACTIVE to CLOSED | Drops the handle without freeing it, for when ownership passed to the native side (e.g. `Signer` into `Context`). Runs `_release()` first, so subclass cleanup still happens. Unlike the other two, it validates nothing. | -| `_release_handle()` | ACTIVE to CLOSED | Frees the handle (guarded, via `_teardown(free_handle=True)`) and closes the object. Same post-state as the consumed teardown. | +| `_consume_and_swap(new_handle)` | ACTIVE to ACTIVE | Requires the resource to already be active and the replacement to be non-null. Used when an FFI call consumed the old handle and returned a new one. | +| `_teardown(free_handle=False)` | ACTIVE to CLOSED | Drops the handle without freeing it, for when ownership passed to the native side (e.g. `Signer` into `Context`). Runs `_release()` first, so subclass cleanup still happens. Unlike the other two it enforces no precondition on the current state: it closes whatever it is given. | +| `_release_handle()` | ACTIVE to CLOSED | Frees the handle (guarded, via `_teardown(free_handle=True)`) and closes the object. Same post-state as the consumed teardown. A resource that is already non-ACTIVE takes the other branch, which clears the handle without freeing it; the reserved consume paths call `_teardown()` directly for that reason. | Because activation is the only way in, no code path can leave an object ACTIVE while holding a null handle. @@ -194,7 +369,7 @@ Two terms recur throughout this document. An **owned free** calls `c2pa_free` on | `True` | Either the pointer is still provably ours (normal `close()`, `__del__`) — an owned free — or ownership is unknown after a failure (`_release_handle()`) — a guarded free. | Calls `c2pa_free`. On the owned paths the pointer is really freed; on the unknown-ownership path the registry returns `-1` without touching memory if the native side already took it. | | `False` | The native side already took ownership: a consuming FFI call swallowed the pointer, or it passed to another object. | Frees nothing; the new owner does. A `c2pa_free` here would double-free (or hit the guarded `-1` no-op that dirties the error slot and risks racing a recycled address). | -Every public method calls `_ensure_valid_state()` before doing any work, which raises `C2paError` unless the resource is ACTIVE with a non-null handle. +Every public method that uses the handle calls `_ensure_valid_state()` before doing any work, which raises `C2paError` unless the resource is ACTIVE with a non-null handle. ## Ways to clean up @@ -232,10 +407,10 @@ If neither the context manager nor an explicit `.close()` is used, `__del__` att Cleanup must not raise an *ordinary* exception. A failure during cleanup (for example, the native library crashing on free) should not mask the original exception that caused the `with` block to exit. `ManagedResource` enforces this: -- `close()` delegates to `_cleanup_resources()`, which wraps the entire cleanup sequence in a try/except that catches and silences `Exception`. +- `close()` delegates to `_cleanup_resources()`, which wraps the entire cleanup sequence in a try/except that catches and silences `Exception`. It performs no teardown itself: after the foreign-process and already-closed checks it calls `_teardown(free_handle=True)`, which does the remaining work. - `_release()` is never called directly during cleanup. It runs inside `_safe_release()`, which logs any `Exception` with a traceback and returns normally, so a subclass whose `_release()` raises an ordinary error cannot stop the native pointer from being freed afterwards. - If freeing the native pointer fails, the error is logged via Python's `logging` module but not re-raised. -- The state is set to `CLOSED` as the very first step, before attempting to free anything. If cleanup fails halfway, the object is still marked closed, preventing a second attempt from doing further damage. +- `_teardown()` sets `CLOSED` before running `_release()` or freeing anything. If cleanup fails halfway, the object is still marked closed, preventing a second attempt from doing further damage. It then moves the pointer into a local and nulls `_handle` before calling `c2pa_free`, so no other caller can read a handle that is about to be freed. - Cleanup is idempotent. Calling `close()` on an already-closed object returns immediately. These handlers catch `Exception`, not `BaseException`. The signals the interpreter raises to unwind a process (a cancellation request, or an exit already in progress) are `BaseException`, so they pass through cleanup untouched and the remaining free may not run. That is intentional: the signal means the whole process is going away, and its address space, native allocations included, is reclaimed on exit. Holding the interpreter in cleanup to finish a free that is about to become irrelevant would only delay the shutdown the caller asked for. @@ -249,11 +424,13 @@ flowchart TD FP -->|yes| N["null the handle, set CLOSED,
do not free"] --> DONE([return]) FP -->|no| ST{"already CLOSED?"} ST -->|yes| DONE - ST -->|no| SET["set CLOSED first"] + ST -->|no| TD["_teardown(free_handle=True)"] + TD --> SET["mark _released, set CLOSED"] SET --> REL["_safe_release()
logs and swallows"] - REL --> H{"handle set?"} + REL --> NULL["take the pointer into a local,
set _handle = None"] + NULL --> H{"pointer was set?"} H -->|no| DONE - H -->|yes| FREE["_free_native_ptr()
logs on failure"] --> NULL["_handle = None"] --> DONE + H -->|yes| FREE["_free_native_ptr()
logs on failure"] --> DONE ``` The `foreign process` branch is explained under [Fork safety](#fork-safety). @@ -276,7 +453,7 @@ with open("photo.jpg", "rb") as file: manifest = reader.json() ``` -The order matters because resources often depend on each other. In both examples, the `Reader` holds a native pointer that references the file's data through a `Stream` wrapper. If the file handle were closed first, the native library would still hold a pointer into the stream's read callbacks, and any subsequent access (including cleanup) could read freed memory or trigger a segfault. By closing the Reader first, the native pointer is freed while the underlying file is still open and valid. Python's `with` statement guarantees this ordering: resources listed later (or nested deeper) are torn down first. +The order matters because resources often depend on each other. In both examples, the `Reader` holds a native pointer that references the file's data through a [`Stream`](#streams) wrapper, and the native library reads that file by calling back into the stream's callbacks. If the file handle were closed first, those callbacks would still be reachable from native code but would be reading a closed file, and any subsequent access (including cleanup) could read freed memory or segfault. By closing the Reader first, the native pointer is freed while the underlying file is still open and valid. Python's `with` statement guarantees this ordering: resources listed later (or nested deeper) are torn down first. ## Reader lifecycle @@ -313,10 +490,12 @@ stateDiagram-v2 end note ``` -While `ACTIVE`, callers can use `.add_ingredient()`, `.add_action()`, etc. repeatedly. `.sign()` closes the Builder when it returns, on both the success and the failure path. Closing without signing frees the pointer the same way. +While `ACTIVE`, callers can use `.add_ingredient()`, `.add_action()`, etc. repeatedly. `.sign()` closes the Builder when it returns, whether the signing succeeded or failed. A call rejected on its arguments before signing starts, such as a first argument that is neither a `Signer` nor a format string, raises without closing: nothing was signed, so the Builder is still usable. Closing without signing frees the pointer the same way. The native sign call borrows the builder's pointer rather than taking ownership of it, so `Builder` never marks it consumed and the pointer is freed normally through `c2pa_free`. The close enforces single use; it is not a memory-management requirement. +The sign call runs inside `self._native_call()`, nesting `signer._native_call()` when the caller passes an explicit `Signer`, and `close()` runs after that block exits (see [Locking and in-flight tracking](#locking-and-in-flight-tracking) for why the order matters and what it protects against). + ## Ownership transfer Some operations transfer a native pointer from one object to another. When this happens, the original object must stop managing the pointer (e.g. so it is not freed twice). @@ -341,16 +520,26 @@ sequenceDiagram X->>X: copy signer._callback_cb to _signer_callback_cb Note right of X: Pin the callback first:
the Signer is about to be consumed X->>S: _consume_no_replacement(set_signer) + S->>S: _begin_consume(): under _op_lock,
refuse if borrowed, then mark CLOSED + Note right of S: The CLOSED mark is the protection:
a close() on another thread finds it
already closed and frees nothing S->>N: c2pa_context_builder_set_signer(builder_ptr, handle) alt status 0 (success) S->>S: _teardown(free_handle=False) Note right of S: Consumed: native took the signer - else pre-consume rejection (UntrackedPointer / WrongPointerType) - Note right of S: Rejected before ownership moved:
Signer retained, typed error raised - else other error - S->>S: _teardown(free_handle=False) - Note right of S: Native took it then failed and dropped it + else non-zero status + S->>S: _raise_consume_failure() reads the native error + Note right of S: The error is read before any free,
so a free's own error cannot overwrite it.
The reservation is held until the branch is known + alt error carries a pre-consume tag + S->>S: _abort_consume(): restore the previous state + Note right of S: Rejected before ownership moved:
Signer stays ACTIVE, typed error raised + else any other error + S->>S: _teardown(free_handle=False) + Note right of S: Native took it, then failed and
dropped the value itself: free nothing + else error slot empty + S->>S: _teardown(free_handle=True) guarded free + Note right of S: Ownership unknown: a real free if still
ours, a -1 no-op if native took it + end end X->>B: _consume_into(build) @@ -362,7 +551,8 @@ sequenceDiagram Details in that sequence that are easy to get wrong: - The callback is copied to the Context *before* the transfer. A successful consume runs `_release()`, which drops the Signer's reference to the callback; a Context that copied it afterwards would be pointing at a callback nothing keeps alive. -- `set_signer` does not always take the pointer. A pre-consume rejection (`UntrackedPointer:` / `WrongPointerType:`) leaves the Signer `ACTIVE` and retained, so the triage must read the native error before deciding to close it. Treating every failure as "consumed" would close a signer the native side never took. +- The transfer is not wrapped in `signer._native_call()`. It is protected by the `CLOSED` mark that `_begin_consume()` makes under `_op_lock` before the native call starts. A racing `signer.close()` finds the resource already closed and frees nothing, and a later borrow is refused for the same reason. That check also refuses the consume outright when another thread is already borrowing the handle to sign with (see [Borrowing versus consuming](#borrowing-versus-consuming)). +- `set_signer` does not always take the pointer. A pre-consume rejection (one of `_PRE_CONSUME_ERROR_TAGS`) leaves the Signer `ACTIVE` and retained, so the triage must read the native error before deciding to close it. Treating every failure as "consumed" would close a signer the native side never took. - A `ctypes.ArgumentError` from `set_signer` is re-raised untouched by `_invoke_consume`: marshalling failed, the native function never ran, and the Signer still owns its handle. Only calls that reached native go through the consumed/retained triage. - The builder is never held as a raw local across the signer and build calls. `_NativeBuilder`'s `with` block owns it: a settings error, a retained-signer error, a build rejection, or an async interrupt all free it through `close()`, and a successful build consumes it so `close()` is then a no-op. The old raw-pointer recovery block that used to free `builder_ptr` on the un-reached-build path is gone. @@ -396,6 +586,44 @@ stateDiagram-v2 On success the object stays `ACTIVE` because the Python-side object is still valid: it has a live native pointer, its public methods still work, and callers may continue using it (e.g. reading the updated manifest or feeding in another fragment). The lifecycle state does not change because from `ManagedResource`'s perspective nothing has closed. Only the underlying native pointer has been swapped. This is different from a consumed teardown (`_teardown(free_handle=False)`), where the object transitions to `CLOSED` and becomes unusable. On the success path the old pointer must not be freed by `ManagedResource` because the native library already consumed it as part of the FFI call. The failure path is different and is covered by the triage in [`_consume_and_swap()`](#_consume_and_swap). +### `Reader.with_fragment()` + +One `with_fragment()` call does two things: + +1. The FFI call consumes the Reader's current handle and returns a replacement, which `_consume_and_swap()` stores. +2. The Reader updates its own Python-side fields: the `Stream` wrappers it owns and the manifest caches. Both still describe the consumed handle. + +`_fragment_streams` holds the `Stream` wrapper for the current fragment. Each call replaces that list rather than appending to it, closing the previous wrapper immediately. The native reader never reads a superseded fragment back, and each open wrapper pins a native stream, its callbacks, and the caller's buffer. + +#### Serializing `with_fragment()` against itself + +Those two steps have to run as one unit. Two threads interleaving them can close a `Stream` the native reader is still reading through. They can also leave the Reader holding wrappers and caches that belong to a handle another thread has already replaced. + +`Reader._fragment_lock` covers both steps. It is an `RLock`, and `with_fragment()` is the only method that takes it. + +The guard spans the native call, which drives caller-supplied stream callbacks. `with_fragment()` therefore takes it with `acquire(blocking=False)` and releases it in a `finally`. A second thread finding it held is refused with `C2paError` rather than parked behind a native call that is waiting on a callback to return. A callback that starts a thread of its own and waits for it would otherwise deadlock: the new thread waits for the guard, and the call holding the guard waits for the callback. + +A refusal leaves the Reader untouched. No stream is built and no handle is consumed, so the call succeeds once the other thread returns. + +Reentrancy applies to the owning thread only. A callback that calls `with_fragment()` synchronously passes the guard and reaches the native call, which rejects the handle it has already consumed. + +The two locks nest in a [fixed order](#lock-ordering): `_fragment_lock` outside, then `_native_call()` and `_op_lock` inside it. + +The fork check runs before `_fragment_lock` is taken, for the same reason `_teardown()` checks first. A forked child cannot wait on a lock no surviving thread will release, so it reports the error `_lock()` reports rather than hanging (see [Fork safety](#fork-safety)). + +#### Cache invalidation + +A `Reader` caches the manifest data. The caches describe the handle they were read from, so a successful swap invalidates them. Both the invalidation and the cache reads in `json()` run under `_op_lock`, so a reader never observes a half-updated cache. + +Replacing the handle and clearing the caches are two separate steps, and `_op_lock` is not held between them: + +1. `_consume_and_swap()` replaces the handle. This runs inside `_native_call()`, which counts the call as in flight rather than holding `_op_lock`. +2. `with_fragment()` acquires `_op_lock` and clears the caches. + +Between the two, the Reader has the new handle and the old manifest. A `json()` on another thread acquires `_op_lock` in that gap, finds the cache populated, and returns it without consulting the handle at all. The manifest it serves belongs to the fragment that was just replaced. + +`_fragment_lock` does not prevent this. It serializes `with_fragment()` against other calls to `with_fragment()`, and readers never take it. Callers sharing a `Reader` across threads have to serialize `with_fragment()` against their own reads. + ### `_consume_and_swap()` Every call of this shape goes through one helper, which takes the FFI call as a callable and handles the outcomes: @@ -403,18 +631,20 @@ Every call of this shape goes through one helper, which takes the FFI call as a ```python # Reader.with_fragment() internally does: self._consume_and_swap( - lambda handle: _lib.c2pa_reader_with_fragment(handle, format_bytes, stream), - Reader._ERROR_MESSAGES['reader_error']) + lambda handle: _lib.c2pa_reader_with_fragment( + handle, format_arg, main_obj._stream, frag_obj._stream, + ), + Reader._ERROR_MESSAGES['fragment_error']) ``` -The call is passed as a lambda because the helper supplies the handle and, on success, replaces it via `_swap_handle()`. +The call is passed as a lambda because the helper supplies the handle and, on success, replaces it via `_consume_and_swap()`. The helper exists because a failed return can be ambiguous. The native functions run in phases: it validates the **borrowed pointer** (passed in without transferring ownership; the caller still owns it unless the callee explicitly takes it over), then takes ownership, then does the work. A failure in the first phase and a failure after the second come back to Python as the same value (a null pointer, or a non-zero status), but they leave ownership in opposite places. ```mermaid flowchart TD - CALL["FFI call(handle)"] --> V{"validate borrowed handle"} - V -->|invalid| R["reject: handle NOT taken
sets UntrackedPointer / WrongPointerType"] --> F1["returns a failure value
(null, or non-zero status)"] + CALL["FFI call(handle)"] --> V{"validate arguments,
then the borrowed handle"} + V -->|invalid| R["reject: handle NOT taken
sets one of _PRE_CONSUME_ERROR_TAGS"] --> F1["returns a failure value
(null, or non-zero status)"] V -->|valid| TAKE["take ownership of handle"] TAKE --> WORK{"execute function logic"} WORK -->|fails| DROP["native drops the value itself
sets some other error"] --> F2["returns a failure value
(null, or non-zero status)"] @@ -428,9 +658,9 @@ The two failure paths are indistinguishable from the return value alone. Only th | Native error | Who owns the handle | What the helper does | | --- | --- | --- | -| `UntrackedPointer:` or `WrongPointerType:` | Still ours: rejected before ownership moved | Handle kept, resource stays `ACTIVE`, typed error raised. Normal cleanup frees it later. | +| One of `_PRE_CONSUME_ERROR_TAGS` | Still ours: rejected before ownership moved | Handle kept, resource stays `ACTIVE`, typed error raised. Normal cleanup frees it later. | | Any other error | Taken, then the operation failed | `_teardown(free_handle=False)`: the native side already dropped the value, so nothing is freed here. Resource goes `CLOSED`, error typed from the native message. | -| No error at all | Unknown | `_release_handle()` guarded free, the caller's message is raised with `"Unknown error"` filled in. | +| No error at all | Unknown | Guarded free, the caller's message is raised with `"Unknown error"` filled in. A reserved consume frees through `_teardown(free_handle=True)`, because `_release_handle()` treats a reserved resource as one it does not own. | This error and ownership triage relies on the native error still being readable (and correctly being the last error encountered) after the call returns. Reading an error copies the message out and frees the copy, but leaves the native slot set until the next error overwrites it. @@ -438,7 +668,7 @@ Three consume helpers share this triage; they differ only in what the FFI call r | Helper | Success return | Success action | | --- | --- | --- | -| `_consume_and_swap()` | a replacement pointer | `_swap_handle()`, resource stays `ACTIVE` | +| `_consume_and_swap()` | a replacement pointer | installs the replacement, resource stays `ACTIVE` | | `_consume_no_replacement()` | a status code (`0` = ok) | `_teardown(free_handle=False)`, resource `CLOSED` | | `_consume_into()` | a *different* object's pointer | `_teardown(free_handle=False)`, the pointer returned for the caller to own | @@ -448,30 +678,31 @@ Three consume helpers share this triage; they differ only in what the FFI call r A consuming FFI call can fail. It may reject the borrowed pointer before taking it, or it may take ownership first and then, on a later failure, drop the value itself. -The native error message indicates which of the errors happened. A rejection carries one of the `_PRE_CONSUME_ERROR_TAGS` (`UntrackedPointer:` or `WrongPointerType:`), which means the handle was never taken and is retained. Any other error message means the native side may have taken ownership and already dropped the value. On top of those, preparing the call's own arguments can fail in Python before the native function ever runs (for example, encoding a bad value or a ctypes marshalling error other than `ArgumentError`), and that outcome is handled separately. +The native error message indicates which of the errors happened. A rejection carries one of the `_PRE_CONSUME_ERROR_TAGS`, which means the handle was never taken and is retained. Any other error message means the native side may have taken ownership and already dropped the value. On top of those, preparing the call's own arguments can fail in Python before the native function ever runs (for example, encoding a bad value or a ctypes marshalling error other than `ArgumentError`), and that outcome is handled separately. -The two settled branches each take the exact action their ownership implies. A pre-consume rejection (an error prefixed `UntrackedPointer:` or `WrongPointerType:`) means the handle is still the caller's, so it is retained and freed later by normal cleanup. Any other native error means the value is already gone, so `_teardown(free_handle=False)` runs the Python-side cleanup without freeing anything. +The two settled branches each take the exact action their ownership implies. A pre-consume rejection (one of the `_PRE_CONSUME_ERROR_TAGS`) means the handle is still the caller's, so it is retained and freed later by normal cleanup. Any other native error means the value is already gone, so `_teardown(free_handle=False)` runs the Python-side cleanup without freeing anything. Always calling the guarded free instead, even where the value is known to be gone, is tempting because a stale free looks like a harmless `-1` no-op. It is only harmless while the freed address stays unclaimed. The native registry rejects an address it no longer tracks, but once another thread allocates a fresh tracked object at that recycled address, the registry does track it again — and a stale free aimed at the old value would now find a live entry and destroy a different thread's object. The scenario is unlikely, but not unreachable: it needs a second thread inside its own FFI call, an allocator that hands back the exact address just freed, and that reuse to happen during the (narrow) window between the native drop and this free. But the window is real under concurrent use. The failure is a silent cross-thread corruption rather than a clean error, and the free is not needed in the first place on this branch. So where the value is known to be consumed, the free is skipped rather than issued and left to the registry to reject. The native error slot stays sticky: it holds whatever it last held until the next error overwrites it, and nothing clears it in between. Issuing an unneeded free would set an untracked-pointer error there that a later caller could mistake for the failure it actually asked about, so skipping the free keeps the slot free for the next real error. `_release_handle()` (a guarded free) is reserved for the two branches where ownership is not known for certain: a Python exception raised before native reports anything, and a failure that leaves the error slot empty (which no defined native failure is expected to produce). In both, a guarded free is a good default, since it is a real free when the handle is still ours and a `-1` no-op when the native side already took it. -None of this is protected by a lock on the Python side: `ManagedResource` has no thread-safety mechanism of its own, and the retained-vs-consumed guarantee comes entirely from the native pointer registry and its thread-local error slot. As noted under [Which double-free risks this layer guards](#double-free-risk-mitigations), sharing one instance across threads without external synchronization is the caller's responsibility. This is a different hazard from [Fork safety](#fork-safety), which concerns a forked child process, not a thread within the same process. +The retained-vs-consumed decision itself comes entirely from the native pointer registry and its thread-local error slot, not from a Python-side lock: `_op_lock` guards concurrent teardown against a call still in flight (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)), but it plays no part in reading which rejection prefix the native side set. That is a different hazard from [Fork safety](#fork-safety), which concerns a forked child process, not a thread within the same process. -A consuming C FFI function first removes the pointer from its registry, then reconstructs the owned value from it. `untrack_or_return!` runs ahead of `Box::from_raw` in `c2pa_c_ffi`. If the address is unknown or the wrong type, the untrack step fails before ownership is taken and sets an error whose prefix (`UntrackedPointer:` or `WrongPointerType:`) identifies it as a pre-consume rejection. Once the value has been reconstructed, a later failure simply drops it, the same as any owned value going out of scope. The Python side stays defensive (and as generic as possible) rather than assuming any exact behavior: it retains the handle when it recognizes one of those rejection prefixes, and where the outcome is unclear it falls back to the guarded free. A native side that behaved differently would degrade in one of two bounded ways: If it kept a pointer the Python side treated as consumed, nothing would free that pointer and it would leak. If it had already released a pointer the Python side then tried to free, the registry would not find the address and the free would return `-1` without touching memory. +A consuming C FFI function first removes the pointer from its registry, then reconstructs the owned value from it. `untrack_or_return!` runs ahead of `Box::from_raw` in `c2pa_c_ffi`. If the address is unknown or the wrong type, the untrack step fails before ownership is taken and sets an error whose prefix (one of the `_PRE_CONSUME_ERROR_TAGS`) identifies it as a pre-consume rejection. Once the value has been reconstructed, a later failure simply drops it, the same as any owned value going out of scope. The Python side stays defensive (and as generic as possible) rather than assuming any exact behavior: it retains the handle when it recognizes one of those rejection prefixes, and where the outcome is unclear it falls back to the guarded free. A native side that behaved differently would degrade in one of two bounded ways: a pointer the native side still held but Python treated as consumed would leak, since nothing would free it; a pointer the native side had already released but Python then tried to free would return `-1` from the registry without touching memory. ### Adopting the handle before giving it away `Reader._init_from_context` and `Builder._init_from_context` both create a native object, immediately activate it, and only then make the consuming call. `_create_and_activate()` handles the create-then-activate half: it calls the FFI constructor, validates the result with `_check_ffi_operation_result`, and `_activate()`s it, freeing the pointer if either step fails so a rejected creation leaks nothing. Reduced to its shape: ```python -self._create_and_activate( - lambda: _lib.c2pa_reader_from_context(context.execution_context), - Reader._ERROR_MESSAGES['reader_error']) +with context._native_call(): + self._create_and_activate( + lambda: _lib.c2pa_reader_from_context(context.execution_context), + Reader._ERROR_MESSAGES['reader_error']) self._consume_and_swap( lambda handle: _lib.c2pa_reader_with_stream( - handle, format_bytes, self._own_stream._stream, + handle, format_arg, self._own_stream._stream, ), Reader._ERROR_MESSAGES['reader_error']) ``` @@ -531,19 +762,13 @@ sequenceDiagram Both `_cleanup_resources()` and the consumed teardown take this branch. Neither simply skips the work: they null the handle and mark the object `CLOSED` so the child cannot go on to use it or try to free it later. Mutating the child's copy has no effect on the parent's, which is untouched and still valid. +`_teardown()` checks `is_foreign_process()` before taking `_op_lock`, not after, so the foreign-process branch never tries to acquire a lock in the child. The lock itself would raise there anyway (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)), but `_teardown()` needs to finish its cleanup rather than raise. Therefore, it settles the fork case first and only reaches for the lock once it knows this process owns the pointer. + The memory the child skips is not lost for good. A child that calls `exec()` replaces its address space; a child that exits has its memory reclaimed by the OS. Even a long-lived child (a `multiprocessing` worker using the fork start method) retains at most the objects it inherited at fork time, which is a bounded, one-off amount rather than a growing leak. Anything the child allocates itself carries the child's own PID and is freed normally. > [!NOTE] > `is_foreign_process()` returns `False` when no owner PID was ever recorded, so an object that somehow missed the stamp is cleaned up as before rather than leaking silently. -## Why is `Stream` not a `ManagedResource`? - -`Stream` wraps a Python stream-like object (file stream or memory stream) so the native library can read from and write to it via callbacks. It does not inherit from `ManagedResource`, and it uses `c2pa_release_stream()` instead of `c2pa_free()` for cleanup. - -The reason is that ownership runs in the opposite direction. A `Reader` or `Builder` holds a native resource that Python code calls methods on. A `Stream` holds a native handle that the native library calls *back into* (read, seek, write, flush). The native library needs a different release function to tear down the callback machinery. - -`Stream` tracks its own state with `_closed` and `_initialized` flags rather than `LifecycleState`, but it supports the same three cleanup paths: context manager, explicit `.close()`, and `__del__` fallback. - ## Which method to use when? `_create_and_activate`, `_consume_and_swap`, `_consume_no_replacement`, @@ -560,7 +785,7 @@ different situation when writing a new subclass: | A Python instance needs to wrap a handle a native call already returned, without creating a new one | `_wrap_native_handle(handle)` (classmethod) | | Ordinary teardown (`close()`, `__del__`) | Neither: these already route through `_cleanup_resources()` and `_teardown()`. Nothing outside `ManagedResource` itself calls `_teardown()` directly. | -`_activate()` and `_swap_handle()` are two low-level primitives this +`_activate()` and `_consume_and_swap()` are two low-level primitives this situation table builds on. ## Implementing a subclass of `ManagedResource` @@ -597,7 +822,7 @@ class NativeResource(ManagedResource): "Failed to create MyResource: {}") def _release(self): - # 4. Clean up class-specific resources. + # 3. Clean up class-specific resources. # Never let this method raise. Must be idempotent. # # Consider defining a simple lifecycle for native resources @@ -615,7 +840,7 @@ class NativeResource(ManagedResource): self._my_stream = None def do_something(self): - # 5. Check state at the start of every public method. + # 4. Check state at the start of every public method. # This raises C2paError if the resource is closed. self._ensure_valid_state() return _lib.c2pa_my_resource_do_something(self._handle) @@ -623,11 +848,11 @@ class NativeResource(ManagedResource): ### Troubleshooting -- An attribute set only in `__init__` is missing on an instance built by `_wrap_native_handle()`, because that path never runs `__init__`. The failure shows up later as an `AttributeError` from whichever method reads the attribute, often `_release()` during cleanup. Attributes belong in `_init_attrs()`, which `__init__` calls. +- An attribute set only in `__init__` is missing on an instance built by `_wrap_native_handle()`, because that path never runs `__init__`. The failure shows up later as an `AttributeError` from whichever method reads the attribute, often `_release()` during cleanup. Attributes belong in `_init_attrs()`, which each subclass `__init__` calls and which `_wrap_native_handle()` calls in its place. `ManagedResource.__init__` does not call it, so a subclass that omits the call gets neither path. - `_init_attrs()` called after an FFI call that can raise leaves `_release()` accessing attributes that do not exist yet when that call fails, crashing with `AttributeError`. It belongs immediately after `super().__init__()`, before anything that can fail. -- Assigning `self._handle` or `self._lifecycle_state` directly bypasses the checks that make the lifecycle safe. `_activate()` refuses a null handle and refuses to run on an already-active object; `_swap_handle()` requires the resource to be active and the replacement non-null. Direct assignment gives up both, and the resulting bugs (an ACTIVE object with a null handle, or a silently discarded pointer) surface far from their cause. +- Assigning `self._handle` or `self._lifecycle_state` directly bypasses the checks that make the lifecycle safe. `_activate()` refuses a null handle and refuses to run on an already-active object; `_consume_and_swap()` requires the resource to be active and the replacement non-null. Direct assignment gives up both, and the resulting bugs (an ACTIVE object with a null handle, or a silently discarded pointer) surface far from their cause. - A `_release()` that raises has its exception silently swallowed by `_cleanup_resources()`, visible only in the logs. A small lifecycle for managed resources would let `_release()` check whether they need releasing; the actual release call wrapped in try/except is a fallback for unexpected failures. diff --git a/src/c2pa/c2pa.py b/src/c2pa/c2pa.py index 537135a4..8d3fb24c 100644 --- a/src/c2pa/c2pa.py +++ b/src/c2pa/c2pa.py @@ -13,12 +13,14 @@ # Version: 0.37.9 +import contextlib import ctypes import enum import json import logging import sys import os +import threading import warnings import weakref from abc import ABC, abstractmethod @@ -233,9 +235,9 @@ class ManagedResource: - Call `_activate(handle)` once the native pointer is created and validated, which takes ownership of it and marks the resource active. Never assign `self._handle` or `self._lifecycle_state` directly. - - Call `_swap_handle(new_handle)` instead when an FFI call consumed the - current handle and returned a replacement (the success side of - `_consume_and_swap`). + - Call `_consume_and_swap(ffi_call, message)` when an FFI call consumes + the current handle and returns a replacement: reserve the handle, + run the call, setup the new handle. - Call `_teardown(free_handle=False)` when an FFI call took ownership of the handle without returning a replacement: the new owner frees it, so this does not. @@ -264,8 +266,150 @@ def _init_attrs(self): def __init__(self): self._lifecycle_state = LifecycleState.UNINITIALIZED self._handle = None + self._op_lock = threading.RLock() + self._inflight = 0 + self._mut_inflight = 0 + self._pending_teardown = None + self._teardown_lock = threading.Lock() + self._released = False record_owner_pid(self) + def _live_op_lock(self): + """Return this resource's operation lock, for mutual exclusion. + + Reentrant: a finalizer can run at any bytecode boundary, including + inside a region this thread already locked, and a consuming call + tears the handle down from inside the locked region. + + Falls back to a fresh lock when the attribute is missing. + + Never hold this across a native call that drives stream callbacks + (construction, resource_to_stream, the Builder stream methods, + signing). + Those calls release the Global Interpreter Lock (GIL) + and re-enter caller-supplied Python code, which may call back into + this API on another thread. + + Raises in a forked child instead of returning the lock: a child + inherits it in whatever state it had at fork(), and no thread in + the child exists to release it, so acquiring there hangs forever. + The child's copy is as unusable as a closed resource, so it + reports the same error. + """ + if is_foreign_process(self): + raise C2paError(f"{type(self).__name__} is closed") + + lock = getattr(self, '_op_lock', None) + if lock is None: + lock = threading.RLock() + try: + self._op_lock = lock + except Exception: + pass + return lock + + def _live_teardown_lock(self): + """Lock to protect teardowns. + + Held only for plain attribute updates, never across a native call or + an acquisition of the operation lock, so it can be taken either alone + or inside the operation lock without an ordering cycle. + + Falls back to a fresh lock when the attribute is missing. + """ + lock = getattr(self, '_teardown_lock', None) + if lock is None: + lock = threading.Lock() + try: + self._teardown_lock = lock + lock = self._teardown_lock + except Exception: + pass + return lock + + def _ensure_not_borrowed(self): + """Raise if a native call is in flight on this handle. + + Raises: + C2paError: If a native call is in flight on this resource. + """ + if getattr(self, '_inflight', 0) > 0: + name = type(self).__name__ + raise C2paError( + f"{name} is in use by another operation and " + f"cannot be consumed") + + def _ensure_no_mutating_call(self): + """Raise if a mutating native call is in flight on this handle. + + Raises: + C2paError: when a mutating native call is in progress. + """ + if getattr(self, '_mut_inflight', 0) > 0: + raise C2paError( + f"{type(self).__name__} is running a mutating operation") + + @contextlib.contextmanager + def _guarded_op(self, *, refuse_mut=True): + """Hold this resource's operation lock its duration, + and mark this thread as inside a native-error section. + + Note: Ordering is important and as the native section opens first + for the native call and closes last. + + Never hold this across a native call that drives stream callbacks. + Those calls release the Global Interpreter Lock + and re-enter caller-supplied code, which may call back into this API + on another thread. + """ + with _native_section(): + try: + with self._live_op_lock(): + if refuse_mut: + self._ensure_no_mutating_call() + yield + finally: + self._maybe_flush_pending() + + @contextlib.contextmanager + def _native_call(self): + """Hold the handle valid across a native call that runs + caller-supplied stream callbacks, so _live_op_lock() can't be held. + Count the call as in-flight/in-progress. + A free intent (teardown) is registered and the last caller frees. + A free intent marks the resource as closed, preventing further use. + """ + with self._live_op_lock(): + self._ensure_valid_state() + self._inflight = getattr(self, '_inflight', 0) + 1 + try: + with _native_section(): + yield + finally: + with self._live_op_lock(): + self._inflight -= 1 + self._maybe_flush_pending() + + @contextlib.contextmanager + def _exclusive_native_call(self): + """Exclusively marks this handle as being mutated. + A free intent (teardown) is registered and the last caller frees. + A free intent marks the resource as closed, preventing further use. + """ + with self._live_op_lock(): + self._ensure_valid_state() + self._ensure_no_mutating_call() + self._mut_inflight = getattr(self, '_mut_inflight', 0) + 1 + self._inflight = getattr(self, '_inflight', 0) + 1 + try: + with _native_section(): + yield + finally: + with self._live_op_lock(): + self._mut_inflight -= 1 + self._inflight -= 1 + self._maybe_flush_pending() + @staticmethod def _free_native_ptr(ptr): """Free a native pointer by passing it to c2pa_free. @@ -284,8 +428,10 @@ def _free_native_ptr(ptr): result = _lib.c2pa_free(ptr) if result != 0: logger.debug( - "c2pa_free returned %s for an untracked pointer ", + "c2pa_free returned %s for an untracked pointer", result) + # Reset error slot. + _write_no_error_marker() return result def _ensure_valid_state(self): @@ -321,12 +467,94 @@ def _safe_release(self): def _teardown(self, free_handle: bool): """Close the object: run _release, optionally free the handle, null it. free_handle=False (consumed) frees nothing, the new owner needs to free. + + The frees run under an operation lock. + Deferred when any gate is blocking: + - this resource's own handle is in flight in a native call + - this thread is inside a native-error section for some call + - someone else holds the operation lock (free intent gets queued) + + The forked-child case is handled before the lock is taken, because + _live_op_lock() raises in a child: this path has to finish rather + than report an error, so it cannot rely on acquiring. + """ + if is_foreign_process(self): + self._handle = None + self._lifecycle_state = LifecycleState.CLOSED + return + + if getattr(self, '_released', False): + return + self._record_pending_intent(free_handle) + + lock = self._live_op_lock() + if not lock.acquire(blocking=False): + self._close_lifecycle() + _register_for_section_flush(self) + return + + try: + if getattr(self, '_released', False): + # Checks released as it recorded possible free intents. + return + if getattr(self, '_inflight', 0) > 0 or _in_native_section(): + # Closes the resource so it can't be used anymore. + # Records also pending actual frees. + self._close_lifecycle() + if _in_native_section(): + _register_for_section_flush(self) + return + + with self._live_teardown_lock(): + pending = getattr(self, '_pending_teardown', None) + if pending is not None: + free_handle = pending and free_handle + self._pending_teardown = None + self._finish_teardown(free_handle) + finally: + lock.release() + + def _record_pending_intent(self, free_handle: bool): + """Queue a teardown intent, leaving the resource usable until + the intent runs. + A queued consume wins over a free, since native already owns a + consumed handle: freeing it again corrupts memory, where a missed + free only leaks. + """ + with self._live_teardown_lock(): + pending = getattr(self, '_pending_teardown', None) + if pending is None: + self._pending_teardown = free_handle + else: + self._pending_teardown = pending and free_handle + + def _close_lifecycle(self): + """Close the resource so it can no longer be used.""" + with self._live_teardown_lock(): + self._lifecycle_state = LifecycleState.CLOSED + + def _record_pending_teardown(self, free_handle: bool): + """Record a teardown intent and queue it. + Also closes the resource, and a queued consume wins + over a (new) teardown request. + """ + self._record_pending_intent(free_handle) + self._close_lifecycle() + + def _finish_teardown(self, free_handle: bool): + """Once teardown can run, runs the actual release. + Steps: release, null the handle, free if requested. """ if is_foreign_process(self): self._handle = None self._lifecycle_state = LifecycleState.CLOSED return + if getattr(self, '_released', False): + # Already done by another caller (concurrent caller). + return + + self._released = True self._lifecycle_state = LifecycleState.CLOSED self._safe_release() @@ -338,13 +566,56 @@ def _teardown(self, free_handle: bool): logger.error("Failed to free native %s resources", type(self).__name__, exc_info=True) + def _has_pending_teardown(self) -> bool: + """Check if a teardown request is waiting for the resource.""" + return getattr(self, '_pending_teardown', None) is not None + + def _flush_pending_pass(self): + """Attempt to run pending teardowns. + """ + + with self._live_op_lock(): + if getattr(self, '_pending_teardown', None) is None: + return + if getattr(self, '_inflight', 0) > 0: + return + if _in_native_section(): + _register_for_section_flush(self) + return + with self._live_teardown_lock(): + free_handle = self._pending_teardown + self._pending_teardown = None + self._finish_teardown(free_handle) + + def _maybe_flush_pending(self): + """Recheck if a teardown can run after something + that blocked it cleared. + """ + if is_foreign_process(self): + return + + self._flush_pending_pass() + if self._has_pending_teardown() and not getattr( + self, '_released', False): + self._flush_pending_pass() + def _release_handle(self): - """Free this handle, then close the object. Used only where ownership is - unknown (a guarded free is a real free if ours, a no-op if not). + """Free this handle and close the object, unless a queued teardown + already owns the free. Used only where ownership is unknown + (a guarded free is a real free if ours, a no-op if not). + Nulling the handle under a queued teardown would leave it nothing + to free. """ - if self._lifecycle_state != LifecycleState.ACTIVE: - self._handle = None - self._lifecycle_state = LifecycleState.CLOSED + with self._live_op_lock(): + owned_elsewhere = getattr( + self, '_pending_teardown', None) is not None + if not owned_elsewhere and ( + self._lifecycle_state != LifecycleState.ACTIVE): + self._handle = None + self._lifecycle_state = LifecycleState.CLOSED + owned_elsewhere = True + if owned_elsewhere: + self._maybe_flush_pending() return self._teardown(free_handle=True) @@ -373,22 +644,21 @@ def _activate(self, handle): def _create_and_activate(self, ffi_call, error_message, *, check=lambda r: not r): - """Obtain a fresh native pointer, validate it, and take ownership. - On any failure before ownership transfers, the pointer is freed - and the error re-raised. + """Get a new pointer/handle, validate, take ownership. Args: ffi_call: Zero-arg callable returning a fresh native pointer. error_message: Message for the C2paError raised on failure. - check: Predicate marking a result invalid - (default: a falsy pointer). + check: Lambda determining result invalidity. Raises: - C2paError: If the pointer fails validation; it is freed first. + C2paError: If the pointer fails the validation step. """ - ptr = ffi_call() + ptr = None try: - _check_ffi_operation_result(ptr, error_message, check=check) + with _native_section(): + ptr = ffi_call() + _check_ffi_operation_result(ptr, error_message, check=check) self._activate(ptr) except Exception: if ptr: @@ -396,36 +666,31 @@ def _create_and_activate(self, ffi_call, error_message, *, raise return ptr - def _swap_handle(self, new_handle): - """Replace the handle after an FFI call consumed the old one and - returned a replacement. - A null return from such a call is ambiguous (the callee may have - failed validation before taking ownership, or failed the operation - after), so callers must not call this with a null replacement. - Requires the resource to be active. - - Args: - new_handle: Non-null native pointer returned by the FFI call - - Raises: - C2paError: If the resource is not ACTIVE or new_handle is null - """ - name = type(self).__name__ - if self._lifecycle_state != LifecycleState.ACTIVE: - raise C2paError( - f"{name}: cannot swap the handle of a resource that is not " - f"active ({self._lifecycle_state.name})") - if not new_handle: - raise C2paError(f"{name}: cannot swap in a null handle") - - self._handle = new_handle - # Errors set by native lib, hinting at the cause of the error # These errors here means the pointer got somehow rejected by the lib, # so it is still ours to deal with. - _PRE_CONSUME_ERROR_TAGS = ("UntrackedPointer:", "WrongPointerType:") + _PRE_CONSUME_ERROR_TAGS = ( + "UntrackedPointer:", + "WrongPointerType:", + ) + + # An error tag starts the message or follows this one wrapper. + _NATIVE_ERROR_WRAPPER = "Other: " + + @staticmethod + def _is_pre_consume_rejection(error: str) -> bool: + """True when native rejected the handle before taking ownership. + + Anchored, not a substring search: native quotes caller text verbatim, + so a tag mid-message describes the caller's input. + """ + body = error + if body.startswith(ManagedResource._NATIVE_ERROR_WRAPPER): + body = body[len(ManagedResource._NATIVE_ERROR_WRAPPER):] + return any(body.startswith(tag) + for tag in ManagedResource._PRE_CONSUME_ERROR_TAGS) - def _invoke_consume(self, ffi_call, error_message): + def _invoke_consume(self, ffi_call, error_message, *, reserved=False): """Run an FFI call that consumes this handle, returning its raw result. A marshalling ArgumentError is re-raised untouched (call never reached @@ -439,11 +704,16 @@ def _invoke_consume(self, ffi_call, error_message): result (a replacement pointer, a status code, ...). error_message: Format string with one placeholder, used to wrap a callback exception. + reserved: True when the caller reserved the handle with + _begin_consume(), which frees here rather than through + _release_handle(). Raises: ctypes.ArgumentError: If marshalling failed; handle untouched. C2paError: If the call raised any other exception. """ + # Same thread that makes the call, same thread-local slot. + _write_no_error_marker() try: return ffi_call(self._handle) except ctypes.ArgumentError: @@ -451,40 +721,51 @@ def _invoke_consume(self, ffi_call, error_message): # is untouched and still ours. Re-raise as-is. raise except Exception as e: - self._release_handle() + if reserved: + # Resource left close (handle set). + self._teardown(free_handle=True) + else: + self._release_handle() raise C2paError(error_message.format(e)) from e - def _raise_consume_failure(self, error_message): + def _raise_consume_failure(self, error_message, previous_state=None): """Raise the error from an FFI handler consuming call. The native error is read before any free so a free's own - pointer-tracking error cannot overwrite it: the native error slot is - sticky and thread-local and the SDK does not clear it before the call, - so this trusts that the failing native path set its own error. - - That ordering is required: - c2pa_free on a handle the registry no longer tracks returns -1 and - overwrites the slot with its own "Other: UntrackedPointer: 0x..." - message. Freeing first would therefore replace the real failure - with another one and, because that substitute carries a pre-consume - tag, invert the retain/consume decision made below. + pointer-tracking error cannot overwrite it. + The native error slot is sticky and thread-local, + and the native SDK does not clear it before the call. + _invoke_consume marks the slot as carrying no error right before a + consuming call, so a failure that sets no error of its own reads back + as no error rather than as a stale one left by an earlier call. + + A caller that reserved the handle with _begin_consume() passes + previous_state and stays reserved until this classification finishes. + _read_native_error() is a native call and releases the GIL, so a + resource restored to ACTIVE before the tags are examined is visible as + usable to another thread while native may already own its handle. Only + the pre-consume branch hands the resource back. Args: error_message: Format string with one placeholder, used when the native layer offers no error of its own. + previous_state: Lifecycle state to restore if the handle turns out + to have been rejected before native took ownership. None when + the caller holds no reservation. Raises: C2paError: Always; typed by the native error when there is one. """ error = _read_native_error() if error: - if any(tag in error - for tag in ManagedResource._PRE_CONSUME_ERROR_TAGS): + if ManagedResource._is_pre_consume_rejection(error): logger.warning( "%s: native call rejected the handle before taking " "ownership (%s); handle retained", type(self).__name__, error) + if previous_state is not None: + self._abort_consume(previous_state) _raise_typed_c2pa_error(error) # A non-tag error means the native side took ownership then failed, @@ -494,20 +775,114 @@ def _raise_consume_failure(self, error_message): self._teardown(free_handle=False) _raise_typed_c2pa_error(error) - # No error in the slot: ownership is unknown, so free defensively. - self._release_handle() + # No error of its own: ownership is unknown, so free defensively. + # c2pa_free returns -1 for an address native already reclaimed. + # A reservation leaves the resource CLOSED with the handle set, + # which _release_handle() nulls without freeing. + if previous_state is not None: + self._teardown(free_handle=True) + else: + self._release_handle() raise C2paError(error_message.format("Unknown error")) + def _begin_consume(self): + """Reserve this handle for a consuming call, or raise. + This is the initiation of an exclusive borrow. + + Marks the resource as closed, stopping other borrows. + After this, the call is considered in-flight. + The caller owns the matching decrement. + + Returns: + The lifecycle state to restore if the call turns out not to have + consumed the handle. + + Raises: + C2paError: Unusable resource or native call in progress. + """ + with self._live_op_lock(): + # A consumed or closed resource has no handle left to hand over; + # without this the call would pass a null pointer to native. + self._ensure_valid_state() + self._ensure_not_borrowed() + previous = self._lifecycle_state + self._lifecycle_state = LifecycleState.CLOSED + self._inflight = getattr(self, '_inflight', 0) + 1 + return previous + + def _abort_consume(self, previous_state): + """Undo _begin_consume() after a call that did not take the handle. + + A pre-consume tag usually means the handle is still ours, so the + resource becomes usable again. The tag can also name another tracked + argument, which this does not distinguish. + + A deferred free still happens when the section drains, so a resource + with a queued teardown stays closed. + """ + with self._live_op_lock(): + if self._pending_teardown is not None: + return + if self._lifecycle_state == LifecycleState.CLOSED and self._handle: + self._lifecycle_state = previous_state + def _consume_and_swap(self, ffi_call, error_message): - """Run an FFI call that consumes this handle and returns a replacement. - On success the native lib consumed the handle and returned a new one, - which we swap in. A null return is a failure. + """Run an FFI call consuming the handle, reserving it. + A replacement handle will be swapping in on success + (a returned null value is a failure). """ - new_ptr = self._invoke_consume(ffi_call, error_message) - if new_ptr: - self._swap_handle(new_ptr) - return - self._raise_consume_failure(error_message) + + previous_state = self._begin_consume() + try: + with _native_section(): + new_ptr = self._invoke_consume( + ffi_call, error_message, reserved=True) + if new_ptr: + with self._live_op_lock(): + self._handle = new_ptr + if self._pending_teardown is None: + self._lifecycle_state = previous_state + return + self._raise_consume_failure(error_message, previous_state) + except BaseException: + self._abort_consume(previous_state) + raise + finally: + # Decrement to handle parallel potential in-flight consumers. + with self._live_op_lock(): + self._inflight -= 1 + self._maybe_flush_pending() + + def _consume_reserved(self, ffi_call, error_message, *, succeeded): + """Run a reserved consuming call and mark the handle consumed on + success. + + Args: + succeeded: Reads the call's raw result and returns whether it + succeeded. Each entry point has its own convention: a status + code, or a replacement pointer. + + Returns: + The call's raw result, for callers that hand it on. + """ + previous_state = self._begin_consume() + try: + with _native_section(): + result = self._invoke_consume( + ffi_call, error_message, reserved=True) + if succeeded(result): + self._teardown(free_handle=False) + return result + self._raise_consume_failure(error_message, previous_state) + except BaseException: + self._abort_consume(previous_state) + raise + finally: + # Same order as _consume_and_swap: drop _inflight under the + # lock, then flush. + with self._live_op_lock(): + self._inflight -= 1 + self._maybe_flush_pending() def _consume_no_replacement(self, ffi_call, error_message): """Run an FFI call that consumes this handle on success, when the native @@ -515,11 +890,9 @@ def _consume_no_replacement(self, ffi_call, error_message): handle. A non-zero status is a failure routed to _raise_consume_failure. """ - result = self._invoke_consume(ffi_call, error_message) - if result == 0: - self._teardown(free_handle=False) - return - self._raise_consume_failure(error_message) + self._consume_reserved( + ffi_call, error_message, + succeeded=lambda status: status == 0) def _consume_into(self, ffi_call, error_message): """Run an FFI call that consumes this handle and returns a *different* @@ -527,11 +900,10 @@ def _consume_into(self, ffi_call, error_message): and the new pointer is returned for the caller to own. A null return is a failure routed to _raise_consume_failure. """ - result = self._invoke_consume(ffi_call, error_message) - if result: - self._teardown(free_handle=False) - return result - self._raise_consume_failure(error_message) + # A null pointer is falsy. + return self._consume_reserved( + ffi_call, error_message, + succeeded=lambda pointer: bool(pointer)) @classmethod def _wrap_native_handle(cls, handle): @@ -572,10 +944,8 @@ def _cleanup_resources(self): if hasattr(self, '_lifecycle_state'): self._lifecycle_state = LifecycleState.CLOSED return - if ( - hasattr(self, '_lifecycle_state') - and self._lifecycle_state != LifecycleState.CLOSED - ): + if hasattr(self, '_lifecycle_state'): + # Closes here must defer to the teardown checks. self._teardown(free_handle=True) except Exception: pass @@ -693,25 +1063,123 @@ class C2paStream(ctypes.Structure): ] +# Address passed to c2pa_free to plant a marker in the native error slot. +# 2 is not an allocatable address. +_NO_ERROR_MARKER_ADDR = 2 + +# Exact text the native lib writes for a failed free of _NO_ERROR_MARKER_ADDR. +_NO_ERROR_MARKER_TEXT = None + + +def _write_no_error_marker(): + """A c2pa_free of an address the registry does not track writes + an expected error message learned at import into the + thread-local error slot and returns -1. + + This marker mechanism exists to distinguish a consuming call that + failed without setting its own error from a stale message left + by an earlier call on the same thread. + + No-op when the marker text could not be learned at import. + """ + if _NO_ERROR_MARKER_TEXT is None: + return + _lib.c2pa_free(_NO_ERROR_MARKER_ADDR) + + +def _is_no_error_marker(message: str) -> bool: + """True for the marker meaning "no current error of our own".""" + return message == _NO_ERROR_MARKER_TEXT + + def _read_native_error() -> Optional[str]: """Read the last error from the native library, or None if unset. - Peeks: the error stays in the native slot, - until the next error overwrites it. - - With no error set the native side still returns an owned pointer to an - empty string, so the pointer alone does not tell us whether there is an - error. Only a non-empty message counts as one; the empty string still - has to be freed. + The slot is marked as carrying no error before returning, so a + given error is reported once, by the caller that observes it. The native + slot is thread-local and sticky, so a message left in place stays readable + indefinitely and is available to be reported again by a later, + unrelated call that failed without setting an error of its own + (or a missing clear of an error slot). """ error = _lib.c2pa_error() if not error: + # NULL means the message could not be rendered, not that the slot + # is empty, so it still has to be marked. + _write_no_error_marker() return None try: message = ctypes.string_at(error).decode('utf-8') finally: _lib.c2pa_string_free(error) - return message or None + + _write_no_error_marker() + if not message or _is_no_error_marker(message): + return None + return message + + +_native_section_state = threading.local() + + +def _in_native_section() -> bool: + """True while this thread is between an FFI call and reading back the + native error it may have set (see _native_section()).""" + return getattr(_native_section_state, 'depth', 0) > 0 + + +def _register_for_section_flush(resource): + """Record that `resource`'s teardown was deferred only because this + thread's native-error section was open.""" + pending = getattr(_native_section_state, 'pending_resources', None) + if pending is not None: + pending.append(resource) + + +@contextlib.contextmanager +def _native_section(): + """Mark this thread as inside a section where a native call's result is + about to be read back: an error-slot check, or a consuming call's + success/failure classification. + + Reentrant: a nested native call on the same thread nests. + """ + state = _native_section_state + depth = getattr(state, 'depth', 0) + state.depth = depth + 1 + if depth == 0: + state.pending_resources = [] + + def _drain(): + """Flush every deferred resource. Returns the first error raised.""" + pending, state.pending_resources = state.pending_resources, [] + first_error = None + for resource in pending: + try: + resource._maybe_flush_pending() + except BaseException as e: # noqa: BLE001 + if first_error is None: + first_error = e + return first_error + + try: + yield + except BaseException: + state.depth -= 1 + if state.depth == 0: + drain_error = _drain() + if drain_error is not None: + logger.error( + "Deferred teardown failed while unwinding: %s", + drain_error) + raise + else: + state.depth -= 1 + if state.depth == 0: + drain_error = _drain() + if drain_error is not None: + logger.error( + "Deferred teardown failed: %s", drain_error) class C2paSignerInfo(ctypes.Structure): @@ -751,6 +1219,7 @@ def __init__(self, alg, sign_cert, private_key, ta_url): alg = alg_str elif isinstance(alg, str): # String to bytes, as requested by native lib + _check_cstr_arg("alg", alg) alg = alg.encode('utf-8') elif isinstance(alg, bytes): # In bytes already @@ -768,6 +1237,7 @@ def __init__(self, alg, sign_cert, private_key, ta_url): pass elif isinstance(ta_url, str): # String to bytes, as requested by native lib + _check_cstr_arg("ta_url", ta_url) ta_url = ta_url.encode('utf-8') elif isinstance(ta_url, bytes): # In bytes already @@ -1036,6 +1506,41 @@ def _setup_function(func, argtypes, restype=None): ) _setup_function(_lib.c2pa_free, [ctypes.c_void_p], ctypes.c_int) + +def _learn_no_error_marker_text(): + """Plant the marker once and read back the exact text the native lib + produces for it, so equality checks match this build of the lib. + + Runs on the importing thread; the text is a format constant, so the + learned value holds for every thread. + + No-op/None if the marker couldn't be learned. + """ + _lib.c2pa_free(_NO_ERROR_MARKER_ADDR) + raw = _lib.c2pa_error() + if not raw: + logger.warning( + "c2pa: could not find out error marker") + return None + try: + text = ctypes.string_at(raw).decode('utf-8') + finally: + _lib.c2pa_string_free(raw) + if not text: + logger.warning( + "c2pa: error-slot marker not set, some errors may be stale") + return None + marker_hex = hex(_NO_ERROR_MARKER_ADDR) + if marker_hex not in text: + logger.warning( + "c2pa: error-slot marker %s unclear, some errors may be stale", + marker_hex) + return None + return text + + +_NO_ERROR_MARKER_TEXT = _learn_no_error_marker_text() + _setup_function( _lib.c2pa_context_builder_set_signer, [ctypes.POINTER(C2paContextBuilder), ctypes.POINTER(C2paSigner)], @@ -1249,12 +1754,63 @@ def _convert_to_py_string(value) -> str: # Ignore clean up issues pass except (ctypes.ArgumentError, TypeError, ValueError, OSError): - # Invalid pointer type or value + # Invalid pointer type or value, gracefully handled by native lib. + try: + _lib.c2pa_string_free(value) + except Exception: + pass return "" return py_string +def _check_cstr_arg(name: str, value) -> None: + """Reject a string argument the native layer would refuse. + Checking here keeps the rejection on this side of the boundary, + where the handle is known to be untouched. + + Raises: + C2paError: With same message the native layer would have produced. + """ + if value is None: + raise C2paError(f"NullParameter: {name}") + + embedded_nul = ( + '\x00' in value if isinstance(value, str) else b'\x00' in value) + if embedded_nul: + # ctypes truncates at the first NUL. + raise C2paError(f"NullParameter: {name} contains a null byte") + + +def _check_handle_arg(name: str, handle) -> None: + """Reject a null handle argument. + Note: registry membership is not observable from Python, + so a tracked-but-invalid pointer still reaches native. + + Raises: + C2paError: With same message the native layer would have produced. + """ + if not handle: + raise C2paError(f"NullParameter: {name}") + + +def _check_bytes_arg(name: str, buffer) -> None: + """Reject a byte buffer the native layer would refuse. + + Native rejects a null pointer and any size outside 1..=isize::MAX. + An empty buffer reaches it as size 0. + Checking here keeps the rejection on this side of the boundary, + where the handle is known to be untouched. + + Raises: + C2paError: With same message the native layer would have produced. + """ + if buffer is None: + raise C2paError(f"NullParameter: {name}") + if len(buffer) == 0: + raise C2paError(f"InvalidBufferSize: 0 for '{name}'") + + def _raise_typed_c2pa_error(error_str: str) -> None: """Parse an error string and raise the appropriate typed C2paError. @@ -1452,16 +2008,35 @@ def load_settings(settings: Union[str, dict], format: str = "json") -> None: raise C2paError(f"Failed to serialize settings to JSON: {e}") try: + _check_cstr_arg("settings", settings_str) + _check_cstr_arg("format", format) settings_bytes = settings_str.encode('utf-8') format_bytes = format.encode('utf-8') except (AttributeError, UnicodeEncodeError) as e: raise C2paError(f"Failed to encode settings to UTF-8: {e}") - result = _lib.c2pa_load_settings(settings_bytes, format_bytes) - _check_ffi_operation_result( - result, - "Error loading settings", - check=lambda r: r != 0) + with _native_section(): + result = _lib.c2pa_load_settings(settings_bytes, format_bytes) + _check_ffi_operation_result( + result, + "Error loading settings", + check=lambda r: r != 0) + + +@contextlib.contextmanager +def _context_guard(context): + """Hold a caller-supplied context valid across a native call. + + ContextProvider requires only is_valid and execution_context. + _native_call may also be implemented on other handlers, and + will leverage managed resources capabilities accordingly. + """ + native_call = getattr(context, "_native_call", None) + if native_call is None: + yield + return + with native_call(): + yield class ContextProvider(ABC): @@ -1469,6 +2044,11 @@ class ContextProvider(ABC): Subclass to implement a custom context provider. The built-in Context class is the standard implementation. + + A provider that does not derive from ManagedResource is used without + in-flight teardown protection: closing it on another thread while a Reader + or Builder is being constructed from it can free the native context while + that construction is still using it. """ @property @@ -1548,16 +2128,17 @@ def set(self, path: str, value: str) -> 'Settings': Returns: self, for method chaining. """ - self._ensure_valid_state() - path_bytes = _to_utf8_bytes(path, "settings path") value_bytes = _to_utf8_bytes(value, "settings value") - _check_ffi_operation_result( - _lib.c2pa_settings_set_value( - self._handle, path_bytes, value_bytes), - "Failed to set settings value", - check=lambda r: r != 0) + with self._guarded_op(): + self._ensure_valid_state() + + _check_ffi_operation_result( + _lib.c2pa_settings_set_value( + self._handle, path_bytes, value_bytes), + "Failed to set settings value", + check=lambda r: r != 0) return self @@ -1574,15 +2155,16 @@ def update( Returns: self, for method chaining. """ - self._ensure_valid_state() - data_bytes = _to_utf8_bytes(data, "settings data") - _check_ffi_operation_result( - _lib.c2pa_settings_update_from_string( - self._handle, data_bytes, b"json"), - "Failed to update settings", - check=lambda r: r != 0) + with self._guarded_op(): + self._ensure_valid_state() + + _check_ffi_operation_result( + _lib.c2pa_settings_update_from_string( + self._handle, data_bytes, b"json"), + "Failed to update settings", + check=lambda r: r != 0) return self @@ -1692,16 +2274,25 @@ def __init__( # a successful build consumes it, so close() is then a no-op. with self._NativeBuilder() as nb: if settings is not None: - _check_ffi_operation_result( - _lib.c2pa_context_builder_set_settings( - nb._handle, settings._c_settings), - "Failed to set settings on Context", - check=lambda r: r != 0) + # Count in-progress reads. + with nb._guarded_op(), settings._native_call(): + _check_ffi_operation_result( + _lib.c2pa_context_builder_set_settings( + nb._handle, settings._c_settings), + "Failed to set settings on Context", + check=lambda r: r != 0) if signer is not None: - signer._ensure_valid_state() - # A rejected signer is retained, not closed and leaked. + # No in-flight guard around the hand-off: the consume + # marks the signer CLOSED under its lock before calling + # native, which is what stops a signer.close() on another + # thread from freeing the handle mid-transfer. That mark + # also makes the consume refuse to start while another + # thread is borrowing the handle to sign with. + # + # Retain a rejected signer for later teardown. self._signer_callback_cb = signer._callback_cb + _check_handle_arg('builder', nb._handle) signer._consume_no_replacement( lambda h: _lib.c2pa_context_builder_set_signer( nb._handle, h), @@ -1813,6 +2404,8 @@ def __init__(self, file_like_stream): self._closed = False self._initialized = False self._stream = None + # Serializes close() and __del__ against a concurrent double-free. + self._close_lock = threading.RLock() # Generate unique stream ID using object ID and counter stream_counter = next(Stream._stream_id_counter) @@ -2034,22 +2627,27 @@ def __del__(self): try: if is_foreign_process(self): return - # Only cleanup if not already closed and we have a valid stream - if hasattr(self, '_closed') and not self._closed: - stream = self._stream - if hasattr(self, '_stream') and stream: - # Use internal cleanup to avoid calling close() which could - # cause issues - try: - _lib.c2pa_release_stream(stream) - except Exception: - # Destructors shouldn't raise exceptions - logger.error("Failed to release Stream") - pass - finally: - self._stream = None - self._closed = True - self._initialized = False + lock = getattr(self, '_close_lock', None) + if lock is not None and not lock.acquire(blocking=False): + return + try: + # Only cleanup if not already closed and we have a valid stream + if hasattr(self, '_closed') and not self._closed: + stream = self._stream + if hasattr(self, '_stream') and stream: + try: + _lib.c2pa_release_stream(stream) + except Exception: + # Destructors shouldn't raise exceptions + logger.error("Failed to release Stream") + pass + finally: + self._stream = None + self._closed = True + self._initialized = False + finally: + if lock is not None: + lock.release() except Exception: # Destructors must not raise exceptions pass @@ -2062,45 +2660,52 @@ def close(self): Errors during cleanup are logged but not raised to ensure cleanup. Multiple calls to close() are handled gracefully. """ - if self._closed: - return + # Checked before the lock, as _live_op_lock() and __del__ do: + # a child inherits _close_lock in whatever state it had at fork(), + # and the thread holding it does not exist there to release it. if is_foreign_process(self): self._closed = True self._initialized = False return - try: - # Clean up stream first as it depends on callbacks - # Note: We don't close self._file_like_stream as we don't own it, - # the opener owns it. - stream = self._stream - if stream: - try: - _lib.c2pa_release_stream(stream) - except Exception as e: - logger.error( - Stream._ERROR_MESSAGES['stream_error'].format( - str(e))) - finally: - self._stream = None - - # Clean up callbacks - for attr in ['_read_cb', '_seek_cb', '_write_cb', '_flush_cb']: - if hasattr(self, attr): + # Serializes against __del__ / a concurrent close(). + with self._close_lock: + if self._closed: + return + + try: + # Clean up stream first as it depends on callbacks + # Note: We don't close self._file_like_stream as we don't + # own it, the opener owns it. + stream = self._stream + if stream: try: - setattr(self, attr, None) + _lib.c2pa_release_stream(stream) except Exception as e: logger.error( - Stream._ERROR_MESSAGES['callback_error'].format( - attr, str(e))) + Stream._ERROR_MESSAGES['stream_error'].format( + str(e))) + finally: + self._stream = None - except Exception as e: - logger.error( - Stream._ERROR_MESSAGES['cleanup_error'].format( - str(e))) - finally: - self._closed = True - self._initialized = False + # Clean up callbacks + for attr in [ + '_read_cb', '_seek_cb', '_write_cb', '_flush_cb']: + if hasattr(self, attr): + try: + setattr(self, attr, None) + except Exception as e: + logger.error( + Stream._ERROR_MESSAGES['callback_error'] + .format(attr, str(e))) + + except Exception as e: + logger.error( + Stream._ERROR_MESSAGES['cleanup_error'].format( + str(e))) + finally: + self._closed = True + self._initialized = False def write_to_target(self, dest_stream): self._file_like_stream.seek(0) @@ -2507,9 +3112,7 @@ def __init__( else: # format_or_path is a format string, stream is a stream object - with Stream(stream) as stream_obj: - self._create_reader( - format_bytes, stream_obj, manifest_data) + self._init_from_stream(stream, format_bytes, manifest_data) @staticmethod def _resolve_format_bytes(format_or_path, stream) -> Optional[bytes]: @@ -2580,6 +3183,24 @@ def _init_from_file(self, path, format_bytes, raise C2paError.Io( Reader._ERROR_MESSAGES['io_error'].format(str(e))) + def _init_from_stream(self, stream, format_bytes, + manifest_data=None): + """Create a reader from a caller-supplied stream object. + The native reader reads through this stream as long as it's alive, + so the wrapper is stored on the instance and released by _release(). + + Args: + stream: A stream-like object owned by the caller + format_bytes: UTF-8 encoded format/MIME type + manifest_data: Optional manifest bytes + """ + try: + self._own_stream = Stream(stream) + self._create_reader(format_bytes, self._own_stream, manifest_data) + except Exception: + self._close_streams() + raise + def _init_from_context(self, context, format_or_path, stream, manifest_data=None): """Initialize Reader from a Context object implementing @@ -2605,20 +3226,27 @@ def _init_from_context(self, context, format_or_path, self._own_stream = Stream(stream) try: - # Adopt before the consuming call: _consume_and_swap needs an - # active resource, and cleanup then owns the pointer either way. - self._create_and_activate( - lambda: _lib.c2pa_reader_from_context( - context.execution_context), - Reader._ERROR_MESSAGES['reader_error']) + # The Context is caller-supplied and may be shared, so its handle + # needs its own in-flight guard across the native call. + with _context_guard(context): + # Adopt before the consuming call: _consume_and_swap needs an + # active resource, and cleanup then owns the pointer either + # way. + self._create_and_activate( + lambda: _lib.c2pa_reader_from_context( + context.execution_context), + Reader._ERROR_MESSAGES['reader_error']) + _check_cstr_arg('format', format_arg) + _check_handle_arg('stream', self._own_stream._stream) if manifest_data is not None: + _check_bytes_arg('manifest_data', manifest_data) manifest_array = ( ctypes.c_ubyte * len(manifest_data)).from_buffer_copy(manifest_data) # Consume current reader, # with manifest data and stream (C FFI pattern), - # to create a new one (switch out) + # to switch it out using _consume_and_swap. self._consume_and_swap( lambda handle: ( _lib.c2pa_reader_with_manifest_data_and_stream( @@ -2649,6 +3277,13 @@ def _init_attrs(self): # Tracks a file we opened ourselves and must close later. self._backing_file = None + # Fragment streams handed to the native reader by with_fragment, + # which it keeps reading from for the rest of its lifecycle. + self._fragment_streams = [] + + # Serializes with_fragment against itself. + self._fragment_lock = threading.RLock() + # Caches for manifest JSON string and parsed data. # These are invalidated when with_fragment() is called. self._manifest_json_str_cache = None @@ -2672,6 +3307,12 @@ def _close_streams(self): logger.warning("Failed to close Reader backing file") finally: self._backing_file = None + for fragment in getattr(self, '_fragment_streams', []): + try: + fragment.close() + except Exception: + logger.warning("Failed to close Reader fragment stream") + self._fragment_streams = [] def _release(self): """Release Reader-specific resources (caches, stream, backing file). @@ -2693,22 +3334,24 @@ def _get_cached_manifest_data(self) -> Optional[dict]: Raises: C2paError: If there was an error getting the JSON """ - if self._manifest_data_cache is None: - if self._manifest_json_str_cache is None: - self._manifest_json_str_cache = self.json() + # Locked so the cache fields can't be read and written + # across concurrent handle swaps. + with self._guarded_op(): + if self._manifest_data_cache is None: + if self._manifest_json_str_cache is None: + self._manifest_json_str_cache = self.json() - try: - self._manifest_data_cache = json.loads( - self._manifest_json_str_cache - ) - except json.JSONDecodeError: - # Reset cache to reattempt read, possibly - self._manifest_data_cache = None - self._manifest_json_str_cache = None - # Failed to parse manifest JSON - return None + try: + self._manifest_data_cache = json.loads( + self._manifest_json_str_cache + ) + except json.JSONDecodeError: + # Next call should retry the read. + self._manifest_data_cache = None + self._manifest_json_str_cache = None + return None - return self._manifest_data_cache + return self._manifest_data_cache def with_fragment(self, format: Optional[str], stream, fragment_stream) -> "Reader": @@ -2731,29 +3374,81 @@ def with_fragment(self, format: Optional[str], stream, C2paError: If there was an error processing the fragment. On failure the native call may already have consumed the underlying object, in which case this Reader is closed and - cannot be retried: create a new one instead of reusing this - instance. + cannot be retried: create a new one. + C2paError: If another thread is inside this method on the same + Reader, or another native call is in flight on it. """ - self._ensure_valid_state() - format_arg = _format_ffi_arg(_encode_format(format, "Reader")) - with Stream(stream) as main_obj, Stream(fragment_stream) as frag_obj: - self._consume_and_swap( - lambda handle: _lib.c2pa_reader_with_fragment( - handle, - format_arg, - main_obj._stream, - frag_obj._stream, - ), - Reader._ERROR_MESSAGES['fragment_error']) - - # Invalidate caches: processing a new BMFF fragment updates the native - # reader's state, which can change the manifest data it returns. - # The cached JSON string and parsed dict may now be stale, so clear - # them to force a fresh read from the native layer on next access. - self._manifest_json_str_cache = None - self._manifest_data_cache = None + # A forked child cannot wait on a lock no surviving thread will + # release, so it reports the same error _live_op_lock() does. + if is_foreign_process(self): + raise C2paError(f"{type(self).__name__} is closed") + + # The native call and the ownership transfer are one unit. + # Reentrant so a thread already here can continue. + if not self._fragment_lock.acquire(blocking=False): + raise C2paError( + f"{type(self).__name__} is already processing a fragment " + f"on another thread") + try: + # The native reader keeps reading through both streams. + main_obj = Stream(stream) + frag_obj = None + try: + frag_obj = Stream(fragment_stream) + _check_cstr_arg('format', format_arg) + _check_handle_arg('stream', main_obj._stream) + _check_handle_arg('fragment', frag_obj._stream) + self._consume_and_swap( + lambda handle: _lib.c2pa_reader_with_fragment( + handle, + format_arg, + main_obj._stream, + frag_obj._stream, + ), + Reader._ERROR_MESSAGES['fragment_error']) + except Exception: + main_obj.close() + if frag_obj is not None: + frag_obj.close() + raise + + with self._guarded_op(refuse_mut=False): + try: + self._ensure_valid_state() + except Exception: + main_obj.close() + frag_obj.close() + raise + + # Replace the streams this reader owned, closing the previous + # ones (only the current fragment is retained). + previous = self._own_stream + previous_fragments = self._fragment_streams + self._own_stream = main_obj + self._fragment_streams = [frag_obj] + if previous is not None and previous is not main_obj: + try: + previous.close() + except Exception: + logger.warning( + "Failed to close previous Reader stream") + for fragment in previous_fragments: + if fragment is frag_obj: + continue + try: + fragment.close() + except Exception: + logger.warning( + "Failed to close Reader fragment stream") + + # Cleared here because these describe the replaced handle, + # and a reader must never be served them. + self._manifest_json_str_cache = None + self._manifest_data_cache = None + finally: + self._fragment_lock.release() return self @@ -2767,19 +3462,18 @@ def json(self) -> str: C2paError: If there was an error getting the JSON """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - # Return cached result if available - if self._manifest_json_str_cache is not None: - return self._manifest_json_str_cache + if self._manifest_json_str_cache is not None: + return self._manifest_json_str_cache - result = _lib.c2pa_reader_json(self._handle) - _check_ffi_operation_result(result, - "Error during manifest parsing in Reader") + result = _lib.c2pa_reader_json(self._handle) + _check_ffi_operation_result( + result, "Error during manifest parsing in Reader") - # Cache the result and return it - self._manifest_json_str_cache = _convert_to_py_string(result) - return self._manifest_json_str_cache + self._manifest_json_str_cache = _convert_to_py_string(result) + return self._manifest_json_str_cache def detailed_json(self) -> str: """Get the detailed JSON representation of the C2PA manifest store. @@ -2797,13 +3491,14 @@ def detailed_json(self) -> str: the Reader has been closed. """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - result = _lib.c2pa_reader_detailed_json(self._handle) - _check_ffi_operation_result( - result, "Error during detailed manifest parsing in Reader") + result = _lib.c2pa_reader_detailed_json(self._handle) + _check_ffi_operation_result( + result, "Error during detailed manifest parsing in Reader") - return _convert_to_py_string(result) + return _convert_to_py_string(result) def crjson(self) -> str: """Get the manifest store as a crJSON string. @@ -2819,12 +3514,13 @@ def crjson(self) -> str: call returns null. """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - result = _lib.c2pa_reader_crjson(self._handle) - _check_ffi_operation_result(result, "Error parsing crJSON") + result = _lib.c2pa_reader_crjson(self._handle) + _check_ffi_operation_result(result, "Error parsing crJSON") - return _convert_to_py_string(result) + return _convert_to_py_string(result) def _get_manifest_field(self, extractor): """Extract a field from (cached) manifest data, or None if unavailable. @@ -2940,10 +3636,9 @@ def resource_to_stream(self, uri: str, stream: Any) -> int: Raises: C2paError: If there was an error writing the resource to stream """ - self._ensure_valid_state() - + _check_cstr_arg("uri", uri) uri_str = uri.encode('utf-8') - with Stream(stream) as stream_obj: + with self._exclusive_native_call(), Stream(stream) as stream_obj: result = _lib.c2pa_reader_resource_to_stream( self._handle, uri_str, stream_obj._stream) @@ -2964,11 +3659,12 @@ def is_embedded(self) -> bool: Raises: C2paError: If there was an error checking the embedded status """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - result = _lib.c2pa_reader_is_embedded(self._handle) + result = _lib.c2pa_reader_is_embedded(self._handle) - return bool(result) + return bool(result) def get_remote_url(self) -> Optional[str]: """Get the remote URL of the manifest if it was obtained remotely. @@ -2981,17 +3677,16 @@ def get_remote_url(self) -> Optional[str]: Raises: C2paError: If there was an error getting the remote URL """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - result = _lib.c2pa_reader_remote_url(self._handle) + result = _lib.c2pa_reader_remote_url(self._handle) - if result is None: - # No remote URL set (manifest is embedded) - return None + if result is None: + # No remote URL set (manifest is embedded). + return None - # Convert the C string to Python string - url_str = _convert_to_py_string(result) - return url_str + return _convert_to_py_string(result) class Signer(ManagedResource): @@ -3019,10 +3714,12 @@ def from_info(cls, signer_info: C2paSignerInfo) -> 'Signer': Raises: C2paError: If there was an error creating the signer """ - signer_ptr = _lib.c2pa_signer_from_info(ctypes.byref(signer_info)) + with _native_section(): + signer_ptr = _lib.c2pa_signer_from_info(ctypes.byref(signer_info)) - _check_ffi_operation_result( - signer_ptr, "Failed to create signer from configured signer_info") + _check_ffi_operation_result( + signer_ptr, + "Failed to create signer from configured signer_info") try: return cls(signer_ptr) @@ -3145,16 +3842,17 @@ def wrapped_callback( callback_cb = SignerCallback(wrapped_callback) # Create the signer with the wrapped callback - signer_ptr = _lib.c2pa_signer_create( - None, - callback_cb, - alg, - certs_bytes, - tsa_url_bytes - ) + with _native_section(): + signer_ptr = _lib.c2pa_signer_create( + None, + callback_cb, + alg, + certs_bytes, + tsa_url_bytes + ) - _check_ffi_operation_result(signer_ptr, - "Failed to create signer") + _check_ffi_operation_result(signer_ptr, + "Failed to create signer") try: # Create and return the signer instance with the callback @@ -3209,16 +3907,17 @@ def reserve_size(self) -> int: Raises: C2paError: If there was an error getting the size """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - result = _lib.c2pa_signer_reserve_size(self._handle) + result = _lib.c2pa_signer_reserve_size(self._handle) - _check_ffi_operation_result( - result, - "Failed to get reserve size", - check=lambda r: r < 0) + _check_ffi_operation_result( + result, + "Failed to get reserve size", + check=lambda r: r < 0) - return result + return result class Builder(ManagedResource): @@ -3315,11 +4014,11 @@ def from_archive( stream_obj = Stream(stream) try: - handle = _lib.c2pa_builder_from_archive(stream_obj._stream) + with _native_section(): + handle = _lib.c2pa_builder_from_archive(stream_obj._stream) - _check_ffi_operation_result(handle, - "Failed to create builder from archive" - ) + _check_ffi_operation_result( + handle, "Failed to create builder from archive") try: # A builder from an archive here carries no context. @@ -3388,12 +4087,18 @@ def _init_from_context(self, context, json_str): if not context.is_valid: raise C2paError("Context is not valid") - # Adopt before the consuming call: _consume_and_swap needs an - # active resource, and cleanup then owns the pointer either way. - self._create_and_activate( - lambda: _lib.c2pa_builder_from_context(context.execution_context), - Builder._ERROR_MESSAGES['builder_error']) + # The Context is caller-supplied and may be shared, + # so its handle needs its own in-flight guard across + # the native call, especially for state checks. + with _context_guard(context): + # Adopt before the consuming call. + self._create_and_activate( + lambda: _lib.c2pa_builder_from_context( + context.execution_context), + Builder._ERROR_MESSAGES['builder_error']) + _check_cstr_arg('manifest_json', json_str) + # _consume_and_swap reserves the handle. self._consume_and_swap( lambda handle: _lib.c2pa_builder_with_definition( handle, json_str), @@ -3416,8 +4121,9 @@ def set_no_embed(self): into the asset when signing. This is useful when creating cloud or sidecar manifests. """ - self._ensure_valid_state() - _lib.c2pa_builder_set_no_embed(self._handle) + with self._guarded_op(): + self._ensure_valid_state() + _lib.c2pa_builder_set_no_embed(self._handle) def set_remote_url(self, remote_url: str): """Set the remote URL. @@ -3431,15 +4137,17 @@ def set_remote_url(self, remote_url: str): Raises: C2paError: If there was an error setting the remote URL """ - self._ensure_valid_state() - url_bytes = _to_utf8_bytes(remote_url, "remote URL") - result = _lib.c2pa_builder_set_remote_url(self._handle, url_bytes) - _check_ffi_operation_result( - result, - Builder._ERROR_MESSAGES['url_error'], - check=lambda r: r != 0) + with self._guarded_op(): + self._ensure_valid_state() + + result = _lib.c2pa_builder_set_remote_url(self._handle, url_bytes) + + _check_ffi_operation_result( + result, + Builder._ERROR_MESSAGES['url_error'], + check=lambda r: r != 0) def set_intent( self, @@ -3467,18 +4175,19 @@ def set_intent( Raises: C2paError: If there was an error setting the intent """ - self._ensure_valid_state() + with self._guarded_op(): + self._ensure_valid_state() - result = _lib.c2pa_builder_set_intent( - self._handle, - ctypes.c_uint(intent), - ctypes.c_uint(digital_source_type), - ) + result = _lib.c2pa_builder_set_intent( + self._handle, + ctypes.c_uint(intent), + ctypes.c_uint(digital_source_type), + ) - _check_ffi_operation_result( - result, - Builder._ERROR_MESSAGES['intent_error'], - check=lambda r: r != 0) + _check_ffi_operation_result( + result, + Builder._ERROR_MESSAGES['intent_error'], + check=lambda r: r != 0) def add_resource(self, uri: str, stream: Any): """Add a resource to the builder. @@ -3491,10 +4200,8 @@ def add_resource(self, uri: str, stream: Any): Raises: C2paError: If there was an error adding the resource """ - self._ensure_valid_state() - uri_bytes = _to_utf8_bytes(uri, "resource URI") - with Stream(stream) as stream_obj: + with self._exclusive_native_call(), Stream(stream) as stream_obj: result = _lib.c2pa_builder_add_resource( self._handle, uri_bytes, stream_obj._stream) @@ -3555,7 +4262,7 @@ def add_ingredient_from_stream( ingredient_str = _to_utf8_bytes(ingredient_json, "ingredient JSON") format_str = _to_utf8_bytes(format, "ingredient format") - with Stream(source) as source_stream: + with self._exclusive_native_call(), Stream(source) as source_stream: result = ( _lib.c2pa_builder_add_ingredient_from_stream( self._handle, @@ -3582,15 +4289,17 @@ def add_action(self, action_json: Union[str, dict]) -> None: C2paError: If there was an error adding the action C2paError.Encoding: If the action JSON contains invalid UTF-8 chars """ - self._ensure_valid_state() - action_str = _to_utf8_bytes(action_json, "action JSON") - result = _lib.c2pa_builder_add_action(self._handle, action_str) - _check_ffi_operation_result( - result, - Builder._ERROR_MESSAGES['action_error'], - check=lambda r: r != 0) + with self._guarded_op(): + self._ensure_valid_state() + + result = _lib.c2pa_builder_add_action(self._handle, action_str) + + _check_ffi_operation_result( + result, + Builder._ERROR_MESSAGES['action_error'], + check=lambda r: r != 0) def to_archive(self, stream: Any) -> None: """Write an archive of the builder to a stream. @@ -3602,9 +4311,7 @@ def to_archive(self, stream: Any) -> None: Raises: C2paError: If there was an error writing the archive """ - self._ensure_valid_state() - - with Stream(stream) as stream_obj: + with self._exclusive_native_call(), Stream(stream) as stream_obj: result = _lib.c2pa_builder_to_archive( self._handle, stream_obj._stream) @@ -3629,7 +4336,7 @@ def write_ingredient_archive(self, ingredient_id: str, stream: Any) -> None: ingredient_id_str = _to_utf8_bytes(ingredient_id, "ingredient_id") - with Stream(stream) as stream_obj: + with self._exclusive_native_call(), Stream(stream) as stream_obj: result = _lib.c2pa_builder_write_ingredient_archive( self._handle, ingredient_id_str, stream_obj._stream) @@ -3649,9 +4356,7 @@ def add_ingredient_from_archive(self, stream: Any) -> None: Raises: C2paError: If there was an error reading the archive """ - self._ensure_valid_state() - - with Stream(stream) as stream_obj: + with self._exclusive_native_call(), Stream(stream) as stream_obj: result = _lib.c2pa_builder_add_ingredient_from_archive( self._handle, stream_obj._stream) @@ -3676,11 +4381,13 @@ def with_archive(self, stream: Any) -> 'Builder': C2paError: If there was an error loading the archive. On failure the native call may already have consumed the underlying object, in which case this Builder is closed and cannot be - retried: create a new one instead of reusing this instance. + retried: create a new one. + C2paError: If another native call is in flight on this Builder. """ self._ensure_valid_state() with Stream(stream) as stream_obj: + _check_handle_arg('stream', stream_obj._stream) self._consume_and_swap( lambda handle: _lib.c2pa_builder_with_archive( handle, stream_obj._stream), @@ -3728,35 +4435,43 @@ def _sign_internal( manifest_bytes_ptr = ctypes.POINTER(ctypes.c_ubyte)() try: - if signer is not None: - result = _lib.c2pa_builder_sign( - self._handle, - format_arg, - source_stream._stream, - dest_stream._stream, - signer._handle, - ctypes.byref(manifest_bytes_ptr) - ) - else: - result = _lib.c2pa_builder_sign_context( - self._handle, - format_arg, - source_stream._stream, - dest_stream._stream, - ctypes.byref(manifest_bytes_ptr), - ) - # Sign borrows the Builder without taking ownership. - # Closing here ensures resources clean up, - # and single use/single sign done by a Builder. - self.close() + # Signing needs short guard sections (a Signer can be used in parallel). + with self._exclusive_native_call(): + if signer is not None: + with signer._native_call(): + result = _lib.c2pa_builder_sign( + self._handle, + format_arg, + source_stream._stream, + dest_stream._stream, + signer._handle, + ctypes.byref(manifest_bytes_ptr) + ) + else: + # The Context pins the consumed signer's callback, which + # native invokes during this call + with _context_guard(self._context): + result = _lib.c2pa_builder_sign_context( + self._handle, + format_arg, + source_stream._stream, + dest_stream._stream, + ctypes.byref(manifest_bytes_ptr), + ) except Exception as e: self.close() raise C2paError(f"Error during signing: {e}") from e - _check_ffi_operation_result( - result, - "Error during signing", - check=lambda r: r < 0) + try: + # _native_call already closed, so close() can free. + with _native_section(): + _check_ffi_operation_result( + result, + "Error during signing", + check=lambda r: r < 0) + finally: + # Single use for a Builder, once signed, close. + self.close() # Capture the manifest bytes if available manifest_bytes = b"" @@ -3982,25 +4697,30 @@ def format_embeddable(format: str, manifest_bytes: bytes) -> tuple[int, bytes]: Raises: C2paError: If there was an error converting the manifest """ + _check_cstr_arg("format", format) format_str = format.encode('utf-8') manifest_array = (ctypes.c_ubyte * len(manifest_bytes)).from_buffer_copy( manifest_bytes ) result_bytes_ptr = ctypes.POINTER(ctypes.c_ubyte)() - result = _lib.c2pa_format_embeddable( - format_str, - manifest_array, - len(manifest_bytes), - ctypes.byref(result_bytes_ptr) - ) + with _native_section(): + result = _lib.c2pa_format_embeddable( + format_str, + manifest_array, + len(manifest_bytes), + ctypes.byref(result_bytes_ptr) + ) - _check_ffi_operation_result( - result, - "Failed to format embeddable manifest", - check=lambda r: r < 0) + _check_ffi_operation_result( + result, + "Failed to format embeddable manifest", + check=lambda r: r < 0) size = result + if not result_bytes_ptr: + raise C2paError( + "Failed to format embeddable manifest: no data returned") try: result_bytes = ctypes.string_at(result_bytes_ptr, size) except Exception as e: @@ -4114,20 +4834,22 @@ def ed25519_sign(data: bytes, private_key: str) -> bytes: # Encode private key to bytes try: + _check_cstr_arg("private_key", private_key) key_bytes = private_key.encode('utf-8') except UnicodeError as e: raise C2paError.Encoding( f"Invalid UTF-8 characters in private key: {str(e)}") # Perform the signing operation - signature_ptr = _lib.c2pa_ed25519_sign( - data_array, - data_size, - key_bytes - ) + with _native_section(): + signature_ptr = _lib.c2pa_ed25519_sign( + data_array, + data_size, + key_bytes + ) - _check_ffi_operation_result(signature_ptr, - "Failed to sign data with Ed25519") + _check_ffi_operation_result(signature_ptr, + "Failed to sign data with Ed25519") try: # Ed25519 signatures are always 64 bytes diff --git a/tests/perf/baseline.json b/tests/perf/baseline.json index c151efe5..5d7017ee 100644 --- a/tests/perf/baseline.json +++ b/tests/perf/baseline.json @@ -2,299 +2,304 @@ "_meta": { "memray_version": "1.19.3", "python_version": "3.12.13", - "c2pa_native_version": "c2pa-v0.90.0", + "c2pa_native_version": "c2pa-v0.90.16", "iterations": 200, "perf_env": "python-3.12-slim", "arch": "aarch64" }, "reader_jpeg_legacy": { - "peak_bytes": 3851610, - "leaked_bytes": 3351823, - "total_allocations": 1362322 + "peak_bytes": 3912724, + "leaked_bytes": 3414162, + "total_allocations": 1324293 }, "reader_jpeg_with_context": { - "peak_bytes": 3845367, - "leaked_bytes": 3345097, - "total_allocations": 1349879 + "peak_bytes": 3907256, + "leaked_bytes": 3407478, + "total_allocations": 1333897 }, "reader_manifest_data_context": { - "peak_bytes": 7636730, - "leaked_bytes": 3468040, - "total_allocations": 1147359 + "peak_bytes": 7692972, + "leaked_bytes": 3524955, + "total_allocations": 1132827 }, "reader_mp4": { - "peak_bytes": 4222601, - "leaked_bytes": 3345724, - "total_allocations": 4095915 + "peak_bytes": 4272788, + "leaked_bytes": 3406355, + "total_allocations": 4018933 }, "reader_wav": { - "peak_bytes": 4523095, - "leaked_bytes": 3355666, - "total_allocations": 742391 + "peak_bytes": 4573253, + "leaked_bytes": 3416313, + "total_allocations": 773409 }, "builder_sign_jpeg_legacy": { - "peak_bytes": 7785129, - "leaked_bytes": 3468507, - "total_allocations": 1041412 + "peak_bytes": 7844930, + "leaked_bytes": 3530986, + "total_allocations": 1046607 }, "builder_sign_jpeg_with_context": { - "peak_bytes": 7779538, - "leaked_bytes": 3463042, - "total_allocations": 1027485 + "peak_bytes": 7839456, + "leaked_bytes": 3524460, + "total_allocations": 1058202 }, "builder_sign_png_legacy": { - "peak_bytes": 8023081, - "leaked_bytes": 3468300, - "total_allocations": 3883115 + "peak_bytes": 8082891, + "leaked_bytes": 3530892, + "total_allocations": 3888499 }, "builder_sign_png_with_context": { - "peak_bytes": 8017008, - "leaked_bytes": 3462829, - "total_allocations": 3869515 + "peak_bytes": 8077349, + "leaked_bytes": 3524774, + "total_allocations": 3900456 }, "builder_sign_jpeg_parallel_split_pool": { - "peak_bytes": 45854797, - "leaked_bytes": 3840928, - "total_allocations": 1035646 + "peak_bytes": 45936892, + "leaked_bytes": 3893837, + "total_allocations": 1062520 }, "builder_sign_jpeg_parallel_split_barrier": { - "peak_bytes": 45844809, - "leaked_bytes": 3861014, - "total_allocations": 1037741 + "peak_bytes": 45905378, + "leaked_bytes": 3892593, + "total_allocations": 1061149 }, "builder_sign_png_parallel_split_pool": { - "peak_bytes": 46586728, - "leaked_bytes": 3868054, - "total_allocations": 3877696 + "peak_bytes": 46673225, + "leaked_bytes": 3929128, + "total_allocations": 3904507 }, "builder_sign_png_parallel_split_barrier": { - "peak_bytes": 46082548, - "leaked_bytes": 3879161, - "total_allocations": 3879780 + "peak_bytes": 46143013, + "leaked_bytes": 3910964, + "total_allocations": 3903125 }, "builder_sign_gif": { - "peak_bytes": 14635465, - "leaked_bytes": 3461270, - "total_allocations": 17017654 + "peak_bytes": 14696646, + "leaked_bytes": 3524513, + "total_allocations": 17048351 }, "builder_sign_heic": { - "peak_bytes": 4698434, - "leaked_bytes": 3469086, - "total_allocations": 1563419 + "peak_bytes": 4759642, + "leaked_bytes": 3532315, + "total_allocations": 1582361 }, "builder_sign_m4a": { - "peak_bytes": 18833496, - "leaked_bytes": 3469085, - "total_allocations": 5194205 + "peak_bytes": 18895243, + "leaked_bytes": 3532373, + "total_allocations": 5213365 }, "builder_sign_webp": { - "peak_bytes": 8991237, - "leaked_bytes": 3461271, - "total_allocations": 916145 + "peak_bytes": 9052463, + "leaked_bytes": 3524559, + "total_allocations": 950737 }, "builder_sign_avi": { - "peak_bytes": 7130933, - "leaked_bytes": 3461270, - "total_allocations": 89982012 + "peak_bytes": 7192106, + "leaked_bytes": 3524502, + "total_allocations": 90011891 }, "builder_sign_mp4": { - "peak_bytes": 6245379, - "leaked_bytes": 3469085, - "total_allocations": 3788717 + "peak_bytes": 6306630, + "leaked_bytes": 3532325, + "total_allocations": 3805992 }, "builder_sign_tiff": { - "peak_bytes": 13213169, - "leaked_bytes": 3461271, - "total_allocations": 10862700 + "peak_bytes": 13274395, + "leaked_bytes": 3524559, + "total_allocations": 10898003 }, "builder_sign_jpeg_parent_of": { - "peak_bytes": 14265295, - "leaked_bytes": 3461665, - "total_allocations": 2506107 + "peak_bytes": 14324563, + "leaked_bytes": 3525074, + "total_allocations": 2495210 }, "builder_sign_jpeg_component_of": { - "peak_bytes": 14266996, - "leaked_bytes": 3462012, - "total_allocations": 2551180 + "peak_bytes": 14325966, + "leaked_bytes": 3524803, + "total_allocations": 2538825 }, "builder_sign_jpeg_parent_and_component": { - "peak_bytes": 14665241, - "leaked_bytes": 3614613, - "total_allocations": 4523960 + "peak_bytes": 14605703, + "leaked_bytes": 3610907, + "total_allocations": 4464450 }, "builder_sign_jpeg_parent_and_component_mixed_mime": { - "peak_bytes": 14568780, - "leaked_bytes": 3462718, - "total_allocations": 5517180 + "peak_bytes": 14627596, + "leaked_bytes": 3525478, + "total_allocations": 5516963 }, "builder_sign_jpeg_two_components_same_mime": { - "peak_bytes": 14559274, - "leaked_bytes": 3564233, - "total_allocations": 4497379 + "peak_bytes": 14602589, + "leaked_bytes": 3610897, + "total_allocations": 4436718 }, "builder_sign_jpeg_two_components_mixed_mime": { - "peak_bytes": 14564839, - "leaked_bytes": 3461873, - "total_allocations": 5490592 + "peak_bytes": 14624276, + "leaked_bytes": 3525287, + "total_allocations": 5489138 }, "builder_sign_jpeg_archive_roundtrip": { - "peak_bytes": 14297571, - "leaked_bytes": 3481212, - "total_allocations": 3467149 + "peak_bytes": 14356429, + "leaked_bytes": 3545507, + "total_allocations": 3432412 }, "builder_from_archive_roundtrip": { - "peak_bytes": 14297349, - "leaked_bytes": 3480475, - "total_allocations": 3101030 + "peak_bytes": 14353258, + "leaked_bytes": 3542427, + "total_allocations": 3024501 }, "builder_with_archive_swap": { - "peak_bytes": 3681081, - "leaked_bytes": 3350198, - "total_allocations": 704373 + "peak_bytes": 3753525, + "leaked_bytes": 3421374, + "total_allocations": 744039 }, "reader_with_fragment_swap": { - "peak_bytes": 3778159, - "leaked_bytes": 3353205, - "total_allocations": 3787587 + "peak_bytes": 3839453, + "leaked_bytes": 3414104, + "total_allocations": 3806570 }, "with_fragment_pre_consume_rejection": { - "peak_bytes": 3778057, - "leaked_bytes": 3354795, - "total_allocations": 2094004 + "peak_bytes": 3841126, + "leaked_bytes": 3417826, + "total_allocations": 2128201 }, "with_archive_post_consume_failure": { - "peak_bytes": 3350600, - "leaked_bytes": 3308056, - "total_allocations": 175290 + "peak_bytes": 3423615, + "leaked_bytes": 3380332, + "total_allocations": 208578 }, "with_fragment_marshalling_error": { - "peak_bytes": 3708068, - "leaked_bytes": 3352335, - "total_allocations": 2077090 + "peak_bytes": 3767911, + "leaked_bytes": 3413267, + "total_allocations": 2094661 }, "with_fragment_mixed_outcomes": { - "peak_bytes": 3779175, - "leaked_bytes": 3356294, - "total_allocations": 2656787 + "peak_bytes": 3840297, + "leaked_bytes": 3417238, + "total_allocations": 2686269 }, "builder_to_archive_with_ingredient": { - "peak_bytes": 14069232, - "leaked_bytes": 3337316, - "total_allocations": 1830896 + "peak_bytes": 14142488, + "leaked_bytes": 3409388, + "total_allocations": 1790801 }, "builder_sign_jpeg_archive_roundtrip_ingredient_in_archive": { - "peak_bytes": 14287046, - "leaked_bytes": 3481977, - "total_allocations": 5879957 + "peak_bytes": 14345054, + "leaked_bytes": 3543673, + "total_allocations": 5769615 }, "builder_write_ingredient_archive": { - "peak_bytes": 14069289, - "leaked_bytes": 3337377, - "total_allocations": 1805304 + "peak_bytes": 14142437, + "leaked_bytes": 3409341, + "total_allocations": 1767419 }, "builder_sign_jpeg_add_ingredient_from_archive": { - "peak_bytes": 14133742, - "leaked_bytes": 3480831, - "total_allocations": 3415920 + "peak_bytes": 14207929, + "leaked_bytes": 3544945, + "total_allocations": 3383511 }, "builder_ingredient_archive_roundtrip": { - "peak_bytes": 14284443, - "leaked_bytes": 3480809, - "total_allocations": 5132060 + "peak_bytes": 14345163, + "leaked_bytes": 3545508, + "total_allocations": 5061203 }, "builder_sign_jpeg_two_ingredient_archives": { - "peak_bytes": 14134560, - "leaked_bytes": 3481604, - "total_allocations": 4215728 + "peak_bytes": 14208534, + "leaked_bytes": 3545923, + "total_allocations": 4185124 }, "reader_error_no_manifest": { - "peak_bytes": 3564471, - "leaked_bytes": 3323629, - "total_allocations": 276175 + "peak_bytes": 3622191, + "leaked_bytes": 3383889, + "total_allocations": 291735 }, "builder_error_invalid_manifest": { - "peak_bytes": 3352053, - "leaked_bytes": 3297079, - "total_allocations": 113926 + "peak_bytes": 3421406, + "leaked_bytes": 3365544, + "total_allocations": 126199 }, "reader_string_apis": { - "peak_bytes": 3978113, - "leaked_bytes": 3346111, - "total_allocations": 2287335 + "peak_bytes": 4039136, + "leaked_bytes": 3407512, + "total_allocations": 2238705 }, "signer_construction": { - "peak_bytes": 3350893, - "leaked_bytes": 3288137, - "total_allocations": 153245 + "peak_bytes": 3421644, + "leaked_bytes": 3358098, + "total_allocations": 159717 }, "builder_from_context_construction": { - "peak_bytes": 3350600, - "leaked_bytes": 3288582, - "total_allocations": 112688 + "peak_bytes": 3423152, + "leaked_bytes": 3360708, + "total_allocations": 146014 }, "fork_reader_collect": { - "peak_bytes": 3850530, - "leaked_bytes": 3353063, - "total_allocations": 1328122 + "peak_bytes": 3911936, + "leaked_bytes": 3413740, + "total_allocations": 1284294 }, "fork_contended_mutex": { - "peak_bytes": 7679019, - "leaked_bytes": 3482128, - "total_allocations": 67472694 + "peak_bytes": 7700288, + "leaked_bytes": 3510073, + "total_allocations": 66621221 }, "fork_thread_local_orphan": { - "peak_bytes": 3936170, - "leaked_bytes": 3439733, - "total_allocations": 1381055 + "peak_bytes": 4073730, + "leaked_bytes": 3581333, + "total_allocations": 1339630 }, "fork_gc_cycle": { - "peak_bytes": 3850434, - "leaked_bytes": 3353160, - "total_allocations": 1332098 + "peak_bytes": 3913068, + "leaked_bytes": 3414664, + "total_allocations": 1289268 }, "fork_parent_frees_after_fork": { - "peak_bytes": 5447584, - "leaked_bytes": 3350400, - "total_allocations": 24829257 + "peak_bytes": 5602620, + "leaked_bytes": 3423572, + "total_allocations": 23965279 }, "fork_child_closes_then_parent_frees": { - "peak_bytes": 5446620, - "leaked_bytes": 3350407, - "total_allocations": 24829254 + "peak_bytes": 5603711, + "leaked_bytes": 3424393, + "total_allocations": 23965271 }, "fork_child_sys_exit": { - "peak_bytes": 3850546, - "leaked_bytes": 3353234, - "total_allocations": 1335925 + "peak_bytes": 3911952, + "leaked_bytes": 3413956, + "total_allocations": 1301497 }, "fork_stream_cleanup": { - "peak_bytes": 3464063, - "leaked_bytes": 3291969, - "total_allocations": 105340 + "peak_bytes": 3532824, + "leaked_bytes": 3361106, + "total_allocations": 110397 }, "fork_swap_cleanup": { - "peak_bytes": 3681171, - "leaked_bytes": 3350696, - "total_allocations": 714376 + "peak_bytes": 3753679, + "leaked_bytes": 3421936, + "total_allocations": 754042 }, "fork_contended_mutex_swap": { - "peak_bytes": 7302379, - "leaked_bytes": 3475147, - "total_allocations": 35948516 + "peak_bytes": 7360409, + "leaked_bytes": 3525035, + "total_allocations": 37359891 }, "fork_contended_mutex_wrap": { - "peak_bytes": 7288748, - "leaked_bytes": 3463411, - "total_allocations": 34847186 + "peak_bytes": 7140341, + "leaked_bytes": 3522965, + "total_allocations": 34204380 }, "fork_consumed_signer": { - "peak_bytes": 3350894, - "leaked_bytes": 3288906, - "total_allocations": 175055 + "peak_bytes": 3421645, + "leaked_bytes": 3359803, + "total_allocations": 206540 }, "swap_chain_churn": { - "peak_bytes": 3681161, - "leaked_bytes": 3350287, - "total_allocations": 672537 + "peak_bytes": 3753669, + "leaked_bytes": 3421527, + "total_allocations": 679964 + }, + "deferred_teardown_flush_queue": { + "peak_bytes": 4103247, + "leaked_bytes": 3412690, + "total_allocations": 2448668 } } \ No newline at end of file diff --git a/tests/perf/scenarios.py b/tests/perf/scenarios.py index 23300aed..87dd512e 100644 --- a/tests/perf/scenarios.py +++ b/tests/perf/scenarios.py @@ -587,8 +587,8 @@ def scenario_reader_with_fragment_pre_consume_rejection( # Fail loudly: without these the scenario still runs when the # ownership logic regresses, and a rejection that stops being # recognised looks identical to a pass. - if not any(tag in str(e) for tag in - c2pa_module.ManagedResource._PRE_CONSUME_ERROR_TAGS): + if not c2pa_module.ManagedResource._is_pre_consume_rejection( + str(e)): raise AssertionError( f"expected a pre-consume rejection, got: {e}") from e if reader._handle is None: @@ -1297,6 +1297,40 @@ def scenario_swap_chain_churn(iterations: int = 100) -> None: context.close() +def scenario_deferred_teardown_flush_queue(iterations: int = 100) -> None: + """Close resources from inside an open native-error section, so their + teardowns defer onto one pending list and are drained together when the + section closes. + + Two resources per iteration rather than one: a single-element queue cannot + show a resource stranded behind its predecessor. + """ + signed_bytes = SIGNED_JPEG.read_bytes() + real_free = c2pa_module.ManagedResource._free_native_ptr + for _ in _iterate(iterations): + first = Reader("image/jpeg", io.BytesIO(signed_bytes)) + second = Reader("image/jpeg", io.BytesIO(signed_bytes)) + + freed = [] + c2pa_module.ManagedResource._free_native_ptr = staticmethod( + lambda ptr: (freed.append(ptr), real_free(ptr))[1]) + try: + with c2pa_module._native_section(): + first.close() + second.close() + # Fail loudly: a free here means the teardown was not deferred. + if freed: + raise AssertionError( + "teardown inside a section freed immediately " + "instead of deferring") + if len(freed) != 2: + raise AssertionError( + f"drain freed {len(freed)} of 2 deferred handles; " + f"the rest leak") + finally: + c2pa_module.ManagedResource._free_native_ptr = real_free + + def scenario_fork_swap_cleanup(iterations: int = 100) -> None: """Fork safety benchmark scenario: the handle a Builder owns at fork time came from with_archive(), which @@ -1403,6 +1437,7 @@ def scenario_fork_stream_cleanup(iterations: int = 100) -> None: "fork_contended_mutex_wrap": scenario_fork_contended_mutex_wrap, "fork_consumed_signer": scenario_fork_consumed_signer, "swap_chain_churn": scenario_swap_chain_churn, + "deferred_teardown_flush_queue": scenario_deferred_teardown_flush_queue, } diff --git a/tests/test_unit_tests.py b/tests/test_unit_tests.py index 4bff6dbb..92ecaa67 100644 --- a/tests/test_unit_tests.py +++ b/tests/test_unit_tests.py @@ -11,6 +11,7 @@ # specific language governing permissions and limitations under # each license. +import ast import gc import inspect import os @@ -30,6 +31,7 @@ import shutil import ctypes import threading +import concurrent.futures # Suppress deprecation warnings warnings.simplefilter("ignore", category=DeprecationWarning) @@ -50,6 +52,15 @@ ALTERNATIVE_INGREDIENT_TEST_FILE = os.path.join(FIXTURES_DIR, "cloud.jpg") +def _fail_with_native_error(tag_bytes): + """Build a mock FFI callable that sets a native error and returns None. + """ + def _mock(*args): + c2pa_module._lib.c2pa_error_set_last(tag_bytes) + return None + return _mock + + def load_test_settings_json(): """ Load default (legacy) trust configuration test settings from a @@ -1361,7 +1372,6 @@ def test_sign_and_read_is_not_embedded(self): # Direct the Builder not to embed the manifest into the asset builder.set_no_embed() - with open(temp_file_path, "wb") as temp_file: manifest_data = builder.sign( signer, "image/jpeg", file, temp_file) @@ -7820,7 +7830,7 @@ def test_callbacks_return_minus_one_after_stream_collected(self): class TestManagedResourceLifecycle(unittest.TestCase): - """Lifecycle primitives (_activate, _swap_handle, _wrap_native_handle), + """Lifecycle primitives (_activate, _consume_and_swap, _wrap_native_handle), the _owner_pid stamp that governs which process may free a handle, and the ownership hand-offs between Python and the native library. @@ -7965,41 +7975,47 @@ def test_activate_does_not_mutate_on_rejection(self): "rejected activation replaced the handle") self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE) - def test_swap_handle_does_not_free_consumed_handle(self): + def test_consume_and_swap_does_not_free_consumed_handle(self): res = self._FakeHandleResource() res._activate(0xAAA1) - res._swap_handle(0xAAA2) + res._consume_and_swap(lambda h: 0xAAA2, "swap: {}") # The FFI already owns and frees the old pointer. self.assertEqual(self.freed, []) self.assertEqual(res._handle, 0xAAA2) + self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE) res.close() self.assertEqual(self.freed, [0xAAA2]) - def test_swap_handle_requires_active_resource(self): + def test_consume_and_swap_requires_active_resource(self): uninitialized = self._FakeHandleResource() with self.assertRaises(Error) as ctx: - uninitialized._swap_handle(0x1) - self.assertIn("not active", str(ctx.exception)) + uninitialized._consume_and_swap(lambda h: 0x1, "swap: {}") + self.assertIn("not properly initialized", str(ctx.exception)) closed = self._FakeHandleResource() closed._activate(0x2) closed.close() - with self.assertRaises(Error): - closed._swap_handle(0x3) + self.freed.clear() + with self.assertRaises(Error) as ctx: + closed._consume_and_swap(lambda h: 0x3, "swap: {}") + self.assertIn("closed", str(ctx.exception)) + self.assertEqual(self.freed, []) - def test_swap_handle_rejects_null_replacement(self): + def test_null_replacement_is_a_failure_that_frees_the_handle(self): + """A null return with no native error leaves ownership unknown, + so the handle is freed defensively and the resource closed.""" res = self._FakeHandleResource() res._activate(0x7777) - with self.assertRaises(Error) as ctx: - res._swap_handle(None) + with self.assertRaises(Error): + res._consume_and_swap(lambda h: None, "swap: {}") - self.assertIn("null handle", str(ctx.exception)) - self.assertEqual(res._handle, 0x7777) - self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE) + self.assertEqual(self.freed, [0x7777]) + self.assertIsNone(res._handle) + self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED) def test_wrap_native_handle_bypasses_init(self): seen = [] @@ -8054,7 +8070,7 @@ def test_every_construction_path_records_owner_pid(self): # A swap keeps the original stamp: # the replacement handle was allocated by the same process # that created the object. - wrapped._swap_handle(0xA3) + wrapped._consume_and_swap(lambda h: 0xA3, "swap: {}") self.assertEqual(wrapped._owner_pid, pid) def test_foreign_child_skips_free_for_wrapped_and_swapped(self): @@ -8064,7 +8080,7 @@ def test_foreign_child_skips_free_for_wrapped_and_swapped(self): swapped = self._FakeHandleResource() swapped._activate(0xC2) - swapped._swap_handle(0xC3) + swapped._consume_and_swap(lambda h: 0xC3, "swap: {}") swapped._owner_pid = os.getpid() + 1 swapped.close() @@ -8090,7 +8106,7 @@ def test_owning_process_frees_wrapped_and_swapped_exactly_once(self): swapped = self._FakeHandleResource() swapped._activate(0xC5) - swapped._swap_handle(0xC6) + swapped._consume_and_swap(lambda h: 0xC6, "swap: {}") swapped.close() # 0xC5 was consumed by the test FFI swap. @@ -8276,12 +8292,11 @@ def test_construction_failure_leaves_nothing_to_free(self): c2pa_module._lib.c2pa_builder_from_json = real_json def test_context_build_null_return_frees_builder(self): - # Set a pre-consume tag in the error slot to mock a pointer rejection. + # Mock a pointer rejection. settings = Settings() - c2pa_module._lib.c2pa_error_set_last( - b"UntrackedPointer: mocked pre-consume rejection") real_build = c2pa_module._lib.c2pa_context_builder_build - c2pa_module._lib.c2pa_context_builder_build = lambda ptr: None + c2pa_module._lib.c2pa_context_builder_build = _fail_with_native_error( + b"UntrackedPointer: mocked pre-consume rejection") try: with self.assertRaises(Error): Context(settings=settings) @@ -8340,6 +8355,180 @@ def test_consume_no_replacement_marks_consumed_on_other_error(self): self.assertIsNone(res._handle) self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED) + def test_invoke_consume_success_does_not_consult_error_slot(self): + """A successful consuming call must not read the error slot at all: + only a failure inspects it.""" + res = self._FakeHandleResource() + res._activate(0xCAFE) + + res._consume_no_replacement(lambda h: 0, "set failed: {}") + + self.assertIsNone(c2pa_module._read_native_error()) + + def test_consume_no_replacement_retains_on_tag_set_by_the_call_itself(self): + """Only a *stale* tag left over from before the call is the + thing being defended against.""" + res = self._FakeHandleResource() + res._activate(0xCAFE) + + def fake_call(handle): + c2pa_module._lib.c2pa_error_set_last( + b"UntrackedPointer: rejected by the call itself") + return -1 + + with self.assertRaises(Error): + res._consume_no_replacement(fake_call, "set failed: {}") + + # Rejected before ownership transferred: handle retained. + self.assertEqual(res._handle, 0xCAFE) + self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE) + self.assertEqual(self.freed, []) + res.close() + self.assertEqual(self.freed, [0xCAFE]) + + def test_native_section_defers_unrelated_finalizer_free(self): + """A finalizer for a completely unrelated resource firing mid + native-call must not free immediately. + """ + victim = self._FakeHandleResource() + victim._activate(0xCAFE) + bystander = self._FakeHandleResource() + bystander._activate(0xB00B) + + def polluting_free(ptr): + self.freed.append(ptr) + # Freeing and untracked/ pointer writes its own error into the + # same thread-local slot. + c2pa_module._lib.c2pa_error_set_last( + "Other: UntrackedPointer: {:#x}".format(ptr).encode()) + return -1 + ManagedResource._free_native_ptr = staticmethod(polluting_free) + + def ffi_call(handle): + nonlocal bystander + del bystander # last reference dropped: __del__ fires right here + return None # the real call failed but set no error of its own + + # A bare section: the consume needs the error section, but not a + # borrow on its own handle. _ensure_not_borrowed + # refuses a consume nested in a _native_call() on the same resource. + with c2pa_module._native_section(): + with self.assertRaises(Error): + victim._consume_no_replacement(ffi_call, "op failed: {}") + + self.assertIsNone( + victim._handle, + "victim was wrongly retained") + self.assertEqual(victim._lifecycle_state, LifecycleState.CLOSED) + # The bystander's free is deferred to the section close, so it + # runs after the consuming call, before the victim's free. + self.assertEqual(self.freed, [0xB00B, 0xCAFE], + "deferred free did not run once, before victim's") + + def test_teardown_deferred_by_own_inflight_and_section_together(self): + """A resource blocked by its own handle being in-flight, + and a wholly separate native-error section is also open on this thread + must not free until both clear, and must free exactly once.""" + res = self._FakeHandleResource() + res._activate(0xCAFE) + + call_cm = res._native_call() + call_cm.__enter__() + try: + section_cm = c2pa_module._native_section() + section_cm.__enter__() + try: + res.close() + self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED) + self.assertEqual(self.freed, [], + "freed while still in flight") + finally: + section_cm.__exit__(None, None, None) + # The independent section closed, but res's own in-flight + # guard is still up: still not freed. + self.assertEqual(self.freed, [], + "flushed while the in-flight guard still held") + finally: + call_cm.__exit__(None, None, None) + # Both gates clear only once native_call's own exit drops inflight + # to 0, which is what should trigger the free. + self.assertEqual(self.freed, [0xCAFE]) + + def test_nested_native_sections_flush_only_at_outermost_close(self): + """A native-error section opened inside another, already-open one + on the same thread must not flush anything until the outermost + one closes.""" + res = self._FakeHandleResource() + res._activate(0xCAFE) + + outer = c2pa_module._native_section() + outer.__enter__() + try: + inner = c2pa_module._native_section() + inner.__enter__() + try: + res.close() + self.assertEqual(self.freed, []) + finally: + inner.__exit__(None, None, None) + # Inner closed, outer is still open: still deferred. + self.assertEqual(self.freed, [], + "inner section flushed before the outer closed") + finally: + outer.__exit__(None, None, None) + self.assertEqual(self.freed, [0xCAFE]) + + def test_native_section_flush_isolates_exceptions(self): + """One deferred free raising during a section's flush must not + stop the rest of that flush from running.""" + good = self._FakeHandleResource() + good._activate(0xC0FFEE) + bad = self._FakeHandleResource() + bad._activate(0xBAD) + + def flaky_free(ptr): + if ptr == 0xBAD: + raise RuntimeError("simulated free failure") + self.freed.append(ptr) + return 0 + ManagedResource._free_native_ptr = staticmethod(flaky_free) + + with self.assertLogs('c2pa', level='ERROR') as captured: + with c2pa_module._native_section(): + bad.close() + good.close() + + self.assertEqual(self.freed, [0xC0FFEE], + "a failing deferred free stopped the rest") + self.assertTrue( + any('Failed to free native' in line + for line in captured.output), + "the failing deferred free was not logged: " + "{}".format(captured.output)) + + def test_stale_error_not_misattributed_after_preset_error(self): + """A stale tag left by an earlier, unrelated call on this thread + must not be read as this call's own error.""" + # A stale tag from an earlier, unrelated call. + c2pa_module._lib.c2pa_error_set_last( + b"Other: UntrackedPointer: 0xdeadbeef") + + res = self._FakeHandleResource() + res._activate(0xCAFE) + + # Fails without setting any error of its own. + # The marker written inside _invoke_consume must have cleared + # the stale tag, so this routes to the "no error of our own" branch. + with self.assertRaises(Error): + res._consume_no_replacement(lambda h: -1, "op failed: {}") + + # A misattributed stale tag would have matched + # _PRE_CONSUME_ERROR_TAGS and left the resource ACTIVE. + self.assertIsNone(res._handle) + self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED) + self.assertEqual(self.freed, [0xCAFE], + "unknown ownership must free, not drop the handle") + class TestManagedResourceObjects(TestContextAPIs): """Tests native resource handling management when managed manually. @@ -8601,9 +8790,9 @@ def test_builder_with_archive_null_return_marks_consumed(self): # Mimic a non-tag error: native took ownership then failed and dropped # the value itself, so the handle is marked consumed, not freed. - c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error") real_call = c2pa_module._lib.c2pa_builder_with_archive - c2pa_module._lib.c2pa_builder_with_archive = lambda b, s: None + c2pa_module._lib.c2pa_builder_with_archive = _fail_with_native_error( + b"Other: mocked test error") # Instrument before the failure... freed = self._instrument_frees() @@ -8637,11 +8826,9 @@ def test_reader_with_fragment_null_return_marks_consumed(self): # Mimic a non-tag error: native took ownership then failed and dropped # the value itself, so the handle is marked consumed, not freed. - c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error") - real_call = c2pa_module._lib.c2pa_reader_with_fragment - c2pa_module._lib.c2pa_reader_with_fragment = ( - lambda r, f, s, frag: None) + c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error( + b"Other: mocked test error") # Instrument before failure so any free would be counted. freed = self._instrument_frees() @@ -8709,11 +8896,11 @@ def _raise(*_args): @staticmethod def _is_pre_consume_rejection(error_message): - """True if this native error means ownership never transferred.""" + """True if this native error means ownership never transferred. + """ if not error_message: return False - return any(tag in error_message - for tag in ManagedResource._PRE_CONSUME_ERROR_TAGS) + return ManagedResource._is_pre_consume_rejection(error_message) def _stale_reader_handle(self): """A freed, untracked pointer, captured before close() nulls it. @@ -8739,31 +8926,6 @@ def _untracked_reader_handle(): return (ctypes.cast(buf, ctypes.POINTER(c2pa_module.C2paReader)), buf) - def test_with_fragment_pre_consume_rejection_keeps_handle(self): - # Rejected before native lib took ownership, - # so nothing was consumed and the handle is still ours. - init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4") - fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s") - with open(init_path, "rb") as init: - reader = Reader("video/mp4", init) - real_handle = reader._handle - - reader._handle = self._stale_reader_handle() - try: - with open(init_path, "rb") as init, \ - open(fragment_path, "rb") as frag: - with self.assertRaises(Error) as caught: - reader.with_fragment("video/mp4", init, frag) - finally: - reader._handle = real_handle - - self.assertIn("UntrackedPointer", str(caught.exception)) - # Ownership never transferred, so the resource stays usable. - self.assertIsNotNone(reader._handle) - self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE) - self.assertTrue(reader.json()) - reader.close() - def test_with_fragment_pre_consume_rejection_does_not_leak(self): # A handle dropped on this path leaks one reader per call. init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4") @@ -8786,6 +8948,161 @@ def test_with_fragment_pre_consume_rejection_does_not_leak(self): self.assertTrue(reader.json()) reader.close() + def _reader_from_context(self): + """A Reader holding a fresh native handle and nothing else. + + Built through the FFI so the consuming call can be + set up with one deliberately invalid argument. + """ + context = Context() + self.addCleanup(context.close) + reader = Reader.__new__(Reader) + ManagedResource.__init__(reader) + reader._init_attrs() + with context._native_call(): + reader._create_and_activate( + lambda: c2pa_module._lib.c2pa_reader_from_context( + context.execution_context), + "Failed to create reader: {}") + return reader + + def test_preflight_rejects_before_the_consuming_call(self): + """A bad argument must be refused before the handle reaches native. + + Native validates arguments and takes ownership in an order that + differs between versions, so a rejection that reaches native leaves + ownership ambiguous. Refusing here keeps the handle unambiguously + ours. + """ + reader = self._reader_from_context() + called = [] + + with self.assertRaises(Error) as caught: + reader._consume_and_swap( + lambda h: (called.append(h), + c2pa_module._check_bytes_arg( + 'manifest_data', b''))[1], + "Failed: {}") + + self.assertIn("InvalidBufferSize", str(caught.exception)) + self.assertEqual( + len(called), 1, + "the guard should raise inside the call, before native runs") + + def test_preflight_rejection_frees_the_handle_exactly_once(self): + """The handle is still ours after a preflight rejection, so it is + freed rather than abandoned.""" + freed = self._instrument_frees() + reader = self._reader_from_context() + handle = reader._handle + + with self.assertRaises(Error): + reader._consume_and_swap( + lambda h: c2pa_module._check_bytes_arg( + 'manifest_data', b''), + "Failed: {}") + + reader.close() + self.assertEqual( + self._free_count(freed, handle), 1, + "a preflight-rejected handle must be freed exactly once") + + def test_reader_with_empty_manifest_data_never_calls_native(self): + """End-to-end: the guard is wired into the public path, not just + available as a helper.""" + context = Context() + self.addCleanup(context.close) + with open(os.path.join(FIXTURES_DIR, + DEFAULT_TEST_FILE_NAME), "rb") as image: + image_bytes = image.read() + + freed = self._instrument_frees() + + with self.assertRaises(Error) as caught: + Reader("image/jpeg", io.BytesIO(image_bytes), + manifest_data=b"", context=context) + + # The guard raises before the FFI call, so the reader handle is still + # the binding's to free: exactly one free, and no abandoned handle. + self.assertIn("InvalidBufferSize", str(caught.exception)) + self.assertEqual( + len(freed), 1, + "a preflight-rejected reader handle must be reclaimed, not leaked") + + def test_check_cstr_arg_rejects_none_and_embedded_nul(self): + """Both cases would reach native as something other than the caller + passed: None as a null pointer, an embedded NUL as a short string.""" + with self.assertRaises(Error) as none_case: + c2pa_module._check_cstr_arg('format', None) + self.assertIn("NullParameter", str(none_case.exception)) + + with self.assertRaises(Error) as nul_case: + c2pa_module._check_cstr_arg('format', "image/\x00jpeg") + self.assertIn("null byte", str(nul_case.exception)) + + c2pa_module._check_cstr_arg('format', "image/jpeg") + c2pa_module._check_cstr_arg('format', b"") + + def test_load_settings_rejects_embedded_nul(self): + with self.assertRaises(Error) as caught: + load_settings('{"a": 1}', format="json\x00") + self.assertIn("null byte", str(caught.exception)) + + def test_format_embeddable_null_out_pointer_raises_not_crashes(self): + real = c2pa_module._lib.c2pa_format_embeddable + c2pa_module._lib.c2pa_format_embeddable = ( + lambda fmt, data, size, out: 128) + try: + with self.assertRaises(Error) as caught: + format_embeddable("image/jpeg", b"junk") + finally: + c2pa_module._lib.c2pa_format_embeddable = real + self.assertIn("no data returned", str(caught.exception)) + + def test_check_bytes_arg_rejects_none_and_empty(self): + for bad in (None, b""): + with self.assertRaises(Error): + c2pa_module._check_bytes_arg('manifest_data', bad) + + c2pa_module._check_bytes_arg('manifest_data', b"x") + + def test_check_handle_arg_rejects_null(self): + """A null handle is a NullParameter on both native versions.""" + with self.assertRaises(Error): + c2pa_module._check_handle_arg('stream', None) + + c2pa_module._check_handle_arg( + 'stream', ctypes.cast(1, ctypes.c_void_p)) + + def test_repeated_with_fragment_does_not_accumulate_streams(self): + """Repeated with_fragment Reader calls should not accumulate streams. + """ + init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4") + fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s") + + with open(init_path, "rb") as init: + reader = Reader("video/mp4", init) + self.addCleanup(reader.close) + + superseded = [] + for _ in range(25): + with open(init_path, "rb") as init, \ + open(fragment_path, "rb") as frag: + reader.with_fragment("video/mp4", init, frag) + self.assertLessEqual( + len(reader._fragment_streams), 1, + "fragment streams accumulated across repeated calls") + superseded.append(reader._fragment_streams[-1]) + + # Dropping the reference is not enough: the native stream is only + # released by close(), so every superseded wrapper must be closed. + self.assertTrue( + all(s.closed for s in superseded[:-1]), + "a superseded fragment stream was dropped without being closed") + + # The reader still works on the fragment it holds. + self.assertTrue(reader.json()) + def test_with_archive_post_consume_failure_consumes_handle(self): # Ownership taken, then the operation failed: # The handle is gone, so close() must not free it again. @@ -8843,10 +9160,9 @@ def test_unknown_failure_drops_handle_without_freeing(self): consumed_handle = reader._handle # Simulate an error being set - c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error") real_call = c2pa_module._lib.c2pa_reader_with_fragment - c2pa_module._lib.c2pa_reader_with_fragment = ( - lambda r, f, s, frag: None) + c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error( + b"Other: mocked test error") try: with open(init_path, "rb") as init, \ open(fragment_path, "rb") as frag: @@ -8896,9 +9212,8 @@ def test_pre_consume_tags_still_match_the_native_wording(self): message = str(caught.exception) self.assertTrue( self._is_pre_consume_rejection(message), - f"the native rejection wording changed and no longer matches " - f"_PRE_CONSUME_ERROR_TAGS; ownership will be misjudged: " - f"{message!r}") + f"rejection wording does not match _PRE_CONSUME_ERROR_TAGS, " + f"so ownership will be misjudged: {message!r}") reader.close() def test_stale_handle_is_actually_rejected_every_time(self): @@ -8949,7 +9264,7 @@ def test_perf_scenario_bogus_handle_is_rejected(self): self.assertTrue( self._is_pre_consume_rejection(str(caught.exception)), - "the perf scenarios' bogus handle is no longer rejected, so " + "the perf bogus handle was not rejected, so " "with_fragment_pre_consume_rejection measures nothing") # Handle kept, so the reader still works and frees normally. self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE) @@ -8957,17 +9272,15 @@ def test_perf_scenario_bogus_handle_is_rejected(self): reader.close() def test_every_null_return_sets_its_own_error(self): - # Reading the slot without clearing it is only sound because every - # null return sets an error. Check each path reports its own. + # Each null-returning path must report the error it set itself, never + # one left behind by an earlier call. init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4") fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s") - # Leave a recognisable error behind, so anything stale shows up. - try: - Reader("image/jpeg", io.BytesIO(b"not an image")).json() - except Error: - pass - self.assertIn("NotSupported", c2pa_module._read_native_error() or "") + # Set a recognizable error, so anything stale is caught by the + # assertNotIn checks. + c2pa_module._lib.c2pa_error_set_last( + b"NotSupported: planted by the test") # Pre-consume rejection: reports UntrackedPointer, not NotSupported. with open(init_path, "rb") as init: @@ -9037,21 +9350,19 @@ def worker(): self.assertEqual(problems, [], "ownership was misjudged under concurrency") - def test_reading_the_native_error_does_not_empty_the_slot(self): - # c2pa_error() peeks, so nothing Python can call empties the slot. - # _consume_and_swap depends on this. - try: - Reader("image/jpeg", io.BytesIO(b"not an image")).json() - except Error: - pass + def test_reading_the_native_error_consumes_it(self): + # c2pa_error() itself peeks, so _read_native_error marks the slot as + # carrying no error once it has read one. + # An error belongs to the caller that observes it; + # leaving it readable lets a later, unrelated failure report it as its own. + c2pa_module._lib.c2pa_error_set_last(b"Io: read me exactly once") first = c2pa_module._read_native_error() self.assertTrue(first, "expected a native error to have been set") - self.assertEqual( - c2pa_module._read_native_error(), first, - "reading emptied the native slot; the comments in " - "_consume_and_swap about a persistent error are now wrong") + self.assertIsNone( + c2pa_module._read_native_error(), + "the native error stayed readable after being reported once") def test_read_native_error_returns_none_for_an_empty_message(self): # c2pa_error() returns an owned pointer to "" when no error is set, @@ -9069,22 +9380,30 @@ def test_read_native_error_returns_none_for_an_empty_message(self): finally: c2pa_module._lib.c2pa_error = original - def test_mocked_null_without_error_is_a_known_limitation(self): - # A null with no error of its own is the case that breaks: the slot - # still holds whatever came before. No native path does this, so it - # is pinned here rather than defended in _consume_and_swap. + def test_null_return_with_no_native_error_is_treated_as_consumed(self): + # A null with no error of its own is the case that breaks without + # the marker: + # the slot still held whatever an unrelated, earlier call on this same + # (pooled) thread left behind, and a stale UntrackedPointer/ + # WrongPointerType tag would make this call believe it still owned a + # handle the native side already dropped. init_path = os.path.join(FIXTURES_DIR, "dashinit.mp4") fragment_path = os.path.join(FIXTURES_DIR, "dash1.m4s") + # A stale, unrelated tag left by a prior call on this thread. c2pa_module._lib.c2pa_error_set_last( b"UntrackedPointer: 0xdeadbeef") with open(init_path, "rb") as init: reader = Reader("video/mp4", init) + consumed_handle = reader._handle real_call = c2pa_module._lib.c2pa_reader_with_fragment + # The fake native call sets no error of its own, + # the marker planted by _invoke_consume is left in the slot. c2pa_module._lib.c2pa_reader_with_fragment = ( lambda r, f, s, frag: None) + freed = self._instrument_frees() try: with open(init_path, "rb") as init, \ open(fragment_path, "rb") as frag: @@ -9092,16 +9411,14 @@ def test_mocked_null_without_error_is_a_known_limitation(self): reader.with_fragment("video/mp4", init, frag) finally: c2pa_module._lib.c2pa_reader_with_fragment = real_call - # Nothing clears the slot, so a planted tag would follow other - # tests around and change how their failures are classified. - c2pa_module._lib.c2pa_error_set_last( - b"Other: cleared by test teardown") - # The stale tag wins, so the handle is kept. Safe here (the mock - # consumed nothing), and the reader is still usable. - self.assertIsNotNone(reader._handle) - self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE) - reader.close() + # The marker survived, not the stale tag. + self.assertIsNone(reader._handle) + self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED) + # Ownership is unknown, so the handle is freed once. c2pa_free + # returns -1 if native had already taken the value. + self.assertEqual(self._free_count(freed, consumed_handle), 1, + "unknown-ownership handle was not freed once") # Backfilling a pointer minted by a direct FFI call. Builder.from_archive # is the only production caller of _wrap_native_handle, so these are the @@ -9250,10 +9567,9 @@ def test_consumed_reader_closes_backing_file(self): self.assertFalse(backing_file.closed) # Simulate an error being set - c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error") real_call = c2pa_module._lib.c2pa_reader_with_fragment - c2pa_module._lib.c2pa_reader_with_fragment = ( - lambda r, f, s, frag: None) + c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error( + b"Other: mocked test error") try: with open(DEFAULT_TEST_FILE, "rb") as main, \ open(DEFAULT_TEST_FILE, "rb") as frag: @@ -9272,9 +9588,9 @@ def test_consumed_builder_releases_context(self): archive = self._make_archive() # Simulate an error being set - c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error") real_call = c2pa_module._lib.c2pa_builder_with_archive - c2pa_module._lib.c2pa_builder_with_archive = lambda b, s: None + c2pa_module._lib.c2pa_builder_with_archive = _fail_with_native_error( + b"Other: mocked test error") try: with self.assertRaises(Error): builder.with_archive(archive) @@ -9321,10 +9637,9 @@ def test_consumed_reader_clears_caches(self): self.assertIsNotNone(reader._manifest_json_str_cache) # Simulate an error being set - c2pa_module._lib.c2pa_error_set_last(b"Other: mocked test error") real_call = c2pa_module._lib.c2pa_reader_with_fragment - c2pa_module._lib.c2pa_reader_with_fragment = ( - lambda r, f, s, frag: None) + c2pa_module._lib.c2pa_reader_with_fragment = _fail_with_native_error( + b"Other: mocked test error") try: with open(DEFAULT_TEST_FILE, "rb") as main, \ open(DEFAULT_TEST_FILE, "rb") as frag: @@ -9396,6 +9711,36 @@ def _boom(*args): self.assertIs(ctx.exception.__cause__, sentinel, "signing error dropped the original exception") + def test_sign_reports_the_native_error_it_set(self): + """sign() reads its error in a later section than the call itself. + The signing call runs inside one _native_call() block and the result + check runs in a separate _native_section() afterwards, so anything + that marks the slot as carrying no error on section exit would discard + the real message between the two. + """ + builder = Builder(self.test_manifest) + signer = self._ctx_make_signer() + self.addCleanup(signer.close) + + real_sign = c2pa_module._lib.c2pa_builder_sign + + def _fail(*args): + c2pa_module._lib.c2pa_error_set_last( + b"Signature: native signing refused") + return -1 + + c2pa_module._lib.c2pa_builder_sign = _fail + try: + with self.assertRaises(Error) as ctx: + builder.sign(signer, "image/jpeg", + io.BytesIO(b"x"), io.BytesIO()) + finally: + c2pa_module._lib.c2pa_builder_sign = real_sign + + self.assertIn("native signing refused", str(ctx.exception), + "the native signing error was lost before it was read") + self.assertIsInstance(ctx.exception, Error.Signature) + class TestErrorPlumbing(unittest.TestCase): """Covers the error helpers themselves, which had no direct tests.""" @@ -9424,18 +9769,37 @@ def test_unmapped_tag_falls_back_to_base_error(self): # Base class only: no subclass should claim an unknown tag. self.assertIs(type(ctx.exception), Error) - def test_pre_consume_tag_match_is_substring_not_prefix(self): - """The tags arrive mid-string, so the match must stay a substring one. + def test_pre_consume_tag_match_skips_the_one_wrapper(self): + """A tag reaches the classifier behind at most one "Other: " wrapper. + The match is anchored after that wrapper, not a substring search. + """ + classify = ManagedResource._is_pre_consume_rejection - Guards the triage in _raise_consume_failure against being "cleaned up" - into error.startswith(tag), which would match nothing and silently - turn every retained handle into a consumed one. + self.assertTrue(classify("Other: UntrackedPointer: 0xdeadb000")) + self.assertTrue(classify("UntrackedPointer: 0xdeadb000")) + self.assertTrue(classify("Other: WrongPointerType: 0xdeadb000")) + + def test_stream_release_preserves_a_pending_error(self): + """Releasing a Stream must not clear an error set by another call. + + __del__ runs at any bytecode boundary, including between an FFI call + and its error read, so anything that clears the slot here reports the + caller's failure as "Unknown error". """ - wire_error = "Other: UntrackedPointer: 0xdeadb000" - tags = ManagedResource._PRE_CONSUME_ERROR_TAGS + for label, dispose in ( + ("close", lambda st: st.close()), + ("__del__", lambda st: st.__del__()), + ): + with self.subTest(dispose=label): + stream = c2pa_module.Stream(io.BytesIO(b"payload")) + self._set_native_error("Io: the failure the caller wants") - self.assertTrue(any(tag in wire_error for tag in tags)) - self.assertFalse(any(wire_error.startswith(tag) for tag in tags)) + dispose(stream) + + self.assertEqual( + c2pa_module._read_native_error(), + "Io: the failure the caller wants", + "releasing a Stream swallowed a pending native error") def test_check_ffi_operation_result_raises_with_native_message(self): self._set_native_error("Io: disk exploded") @@ -9527,6 +9891,340 @@ def test_supported_mime_types_reports_the_native_message(self): c2pa_module._get_supported_mime_types(lambda count: None, None) self.assertIn("mime lookup failed", str(ctx.exception)) + def test_reading_an_error_does_not_leave_it_readable(self): + """An error is reportable once, by the reader that observes it. + """ + self._set_native_error("Io: read me once") + + self.assertEqual( + c2pa_module._read_native_error(), "Io: read me once") + self.assertIsNone( + c2pa_module._read_native_error(), + "the same native error was reported a second time") + + def test_handled_error_does_not_survive_later_operations(self): + """A caught failure must not leave its error in-place + (tests the slot is cleaned up). + """ + with self.assertRaises(Error): + Reader("image/jpeg", io.BytesIO(b"not an image")) + + for _ in range(20): + c2pa_module.Stream(io.BytesIO(b"x")) + + self.assertIsNone( + c2pa_module._read_native_error(), + "a handled error was still resident after 20 successful calls") + + def test_later_failure_does_not_inherit_a_handled_errors_type(self): + """A failure with no error of its own must not see an older one. + """ + with self.assertRaises(Error) as first: + Reader("image/jpeg", io.BytesIO(b"not an image")) + self.assertIsInstance(first.exception, Error.NotSupported) + + with self.assertRaises(Error) as second: + c2pa_module._check_ffi_operation_result( + None, "Later unrelated failure: {}") + + self.assertNotIsInstance( + second.exception, Error.NotSupported, + "the later failure inherited the handled error's type") + self.assertIn("Unknown error", str(second.exception)) + self.assertNotIn( + "type is unsupported", str(second.exception), + "the later failure reported the handled error's message") + + def test_the_no_error_marker_never_reaches_a_caller(self): + """The marker is internal, not a message for users.""" + marker = c2pa_module._NO_ERROR_MARKER_TEXT + + c2pa_module._write_no_error_marker() + self.assertIsNone( + c2pa_module._read_native_error(), + "the marker was reported as if it were a native error") + + c2pa_module._write_no_error_marker() + with self.assertRaises(Error) as ctx: + c2pa_module._check_ffi_operation_result(None, "fallback: {}") + self.assertNotIn(marker, str(ctx.exception)) + self.assertIn("Unknown error", str(ctx.exception)) + + def test_write_no_error_marker_writes_the_learned_text(self): + c2pa_module._write_no_error_marker() + raw = c2pa_module._lib.c2pa_error() + try: + text = ctypes.string_at(raw).decode('utf-8') + finally: + c2pa_module._lib.c2pa_string_free(raw) + self.assertEqual(text, c2pa_module._NO_ERROR_MARKER_TEXT) + + def test_read_native_error_maps_the_marker_to_none(self): + c2pa_module._write_no_error_marker() + self.assertIsNone(c2pa_module._read_native_error()) + + def test_read_native_error_marks_the_slot_when_the_pointer_is_null(self): + """A NULL from c2pa_error must still leave the slot marked. + + c2pa_error returns NULL when the stored message cannot be rendered as + a C string. The message stays in the thread-local slot, which is + sticky, so returning without planting the marker leaves that message + readable by the next call that fails without setting an error of its + own, which then reports it as its own failure. + """ + c2pa_module._lib.c2pa_error_set_last(b"Io: unreadable original") + + original = c2pa_module._lib.c2pa_error + try: + c2pa_module._lib.c2pa_error = lambda: None + self.assertIsNone( + c2pa_module._read_native_error(), + "a NULL pointer must read as no error") + finally: + c2pa_module._lib.c2pa_error = original + + self.assertIsNone( + c2pa_module._read_native_error(), + "the NULL branch left the message in the slot instead of " + "planting the marker") + + def test_a_failure_after_a_null_read_does_not_inherit_the_old_message(self): + """The message surviving a NULL read must not become someone's error.""" + c2pa_module._lib.c2pa_error_set_last(b"Io: belongs to an earlier call") + + original = c2pa_module._lib.c2pa_error + try: + c2pa_module._lib.c2pa_error = lambda: None + c2pa_module._read_native_error() + finally: + c2pa_module._lib.c2pa_error = original + + with self.assertRaises(Error) as ctx: + c2pa_module._check_ffi_operation_result( + None, "Later unrelated failure: {}") + + self.assertNotIn( + "belongs to an earlier call", str(ctx.exception), + "a later failure reported a message left by an earlier call") + self.assertIn("Unknown error", str(ctx.exception)) + + def test_every_real_rejection_wording_is_classified_as_pre_consume(self): + """Every tag arrives bare or behind the "Other: " wrapper.""" + wrapper = c2pa_module.ManagedResource._NATIVE_ERROR_WRAPPER + classify = c2pa_module.ManagedResource._is_pre_consume_rejection + + for tag in c2pa_module.ManagedResource._PRE_CONSUME_ERROR_TAGS: + bare = f"{tag} some detail" + wrapped = f"{wrapper}{tag} some detail" + self.assertTrue( + classify(bare), + f"a bare {tag} rejection was read as a consumed handle") + self.assertTrue( + classify(wrapped), + f"a wrapped {tag} rejection was read as a consumed handle") + + def test_caller_text_quoting_a_tag_is_not_a_rejection(self): + """A tag inside the message body describes the caller's input. + + Native errors quote caller-supplied strings verbatim: a JSON parse + failure repeats the offending value, an Io failure names the path. + Reading one of those as a pre-consume rejection hands the resource back + as usable after native may already own and have dropped its handle. + """ + classify = c2pa_module.ManagedResource._is_pre_consume_rejection + + forged = ( + 'Json: invalid type: string "NullParameter: x", expected a ' + 'sequence at line 1 column 43', + 'Json: invalid type: string "WrongPointerType: y", expected a ' + 'sequence at line 1 column 46', + "Io: cannot open /tmp/UntrackedPointer: 0xdead.jpg", + "Other: manifest text mentions InvalidBufferSize: in passing", + ) + for message in forged: + self.assertFalse( + classify(message), + f"caller text was read as a pointer rejection: {message!r}") + + def test_caller_text_quoting_a_tag_reaches_the_error_slot(self): + """test_caller_text_quoting_a_tag_is_not_a_rejection forges this + wording; the library really produces it. + """ + c2pa_module._lib.c2pa_builder_from_json( + b'{"claim_generator_info": "NullParameter: injected"}') + message = c2pa_module._read_native_error() + + self.assertIn( + "NullParameter:", message, + "caller text did not reach the error slot verbatim: the " + "forged wording is stale") + self.assertFalse( + c2pa_module.ManagedResource._is_pre_consume_rejection(message), + f"a caller-supplied string forged a pointer rejection: {message!r}") + + def test_a_failing_flush_does_not_strand_the_rest_of_the_queue(self): + """One resource raising must not skip the resources queued behind it. + """ + flushed = [] + + class Recorder: + def __init__(self, name, raises=None): + self.name = name + self.raises = raises + + def _maybe_flush_pending(self): + if self.raises is not None: + raise self.raises + flushed.append(self.name) + + first = Recorder("first") + middle = Recorder("middle", raises=KeyboardInterrupt()) + last = Recorder("last") + + with self.assertLogs("c2pa", level="ERROR"): + with c2pa_module._native_section(): + for resource in (first, middle, last): + c2pa_module._register_for_section_flush(resource) + + self.assertEqual( + flushed, ["first", "last"], + "a resource queued behind a failing one was never flushed, " + "so its handle leaks") + + def test_a_failing_flush_logs_the_first_exception(self): + """Failures on drain should be logged.""" + flushed = [] + + class Recorder: + def __init__(self, name, raises=None): + self.name = name + self.raises = raises + + def _maybe_flush_pending(self): + if self.raises is not None: + raise self.raises + flushed.append(self.name) + + with self.assertLogs("c2pa", level="ERROR") as captured: + with c2pa_module._native_section(): + for resource in ( + Recorder("boom", raises=RuntimeError("first failure")), + Recorder("survivor"), + Recorder("later", raises=RuntimeError("second failure"))): + c2pa_module._register_for_section_flush(resource) + + self.assertTrue( + any("first failure" in message for message in captured.output)) + self.assertEqual( + flushed, ["survivor"], + "a resource between two failing ones was never flushed") + + def test_runtime_does_not_call_error_set_last(self): + """The marker mechanism must not depend on c2pa_error_set_last, + so this module loads against native builds that lack it.""" + for fn in (c2pa_module.ManagedResource._invoke_consume, + c2pa_module._read_native_error, + c2pa_module._write_no_error_marker): + self.assertNotIn( + 'c2pa_error_set_last', inspect.getsource(fn)) + + +class TestMarkerOutlivesPointerConsumptionSemantics(unittest.TestCase): + """The marker is needed for reasons independent of pointer ownership. + + The native error slot is sticky and thread-local, so failure paths + that carry no still need to tell an error this call set from an + earlier, unrelated call left behind. + """ + + def setUp(self): + # Leave no message from an earlier test in this thread's slot. + c2pa_module._write_no_error_marker() + + def test_non_consuming_failure_does_not_inherit_a_read_error(self): + c2pa_module._lib.c2pa_error_set_last(b"Signature: earlier task") + # The rightful owner reports it, which re-marks the slot. + self.assertEqual( + c2pa_module._read_native_error(), "Signature: earlier task") + + # A later, unrelated failure that sets no error of its own must + # report its own fallback, not the planted Signature message. + with self.assertRaises(Error) as ctx: + c2pa_module._check_ffi_operation_result( + 0, "later op failed: {}", check=lambda r: r == 0) + + self.assertNotIn("earlier task", str(ctx.exception)) + self.assertIn("Unknown error", str(ctx.exception)) + self.assertNotIsInstance(ctx.exception, Error.Signature) + + def test_settings_set_failure_reports_its_own_error(self): + settings = Settings() + self.addCleanup(settings.close) + + c2pa_module._lib.c2pa_error_set_last(b"Signature: earlier task") + self.assertEqual( + c2pa_module._read_native_error(), "Signature: earlier task") + + with self.assertRaises(Error) as ctx: + settings.set("builder.thumbnail.enabled", "not-a-json-value") + + self.assertNotIn("earlier task", str(ctx.exception)) + + def test_marker_is_per_thread_across_pooled_reuse(self): + """The slot is thread-local, so a pooled worker must not hand one + task's error to the next task that runs on it.""" + def failing_task(): + c2pa_module._lib.c2pa_error_set_last(b"Io: first task") + return c2pa_module._read_native_error() + + def quiet_task(): + # Sets no error; must not see the previous task's message. + return c2pa_module._read_native_error() + + # One worker guarantees both tasks run on the same OS thread. + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + self.assertEqual(pool.submit(failing_task).result(), + "Io: first task") + self.assertIsNone( + pool.submit(quiet_task).result(), + "a pooled thread carried an error across unrelated tasks") + + def test_one_thread_marker_does_not_clear_another_threads_error(self): + """Marking on one thread must leave another thread's pending error + readable: the slot is per thread, and so is the marker.""" + set_on_worker = threading.Event() + marked_on_main = threading.Event() + seen = {} + + def worker(): + c2pa_module._lib.c2pa_error_set_last(b"Io: worker error") + set_on_worker.set() + self.assertTrue(marked_on_main.wait(5)) + seen["worker"] = c2pa_module._read_native_error() + + thread = threading.Thread(target=worker, daemon=True) + thread.start() + self.assertTrue(set_on_worker.wait(5)) + + c2pa_module._write_no_error_marker() + marked_on_main.set() + thread.join(5) + + self.assertEqual(seen.get("worker"), "Io: worker error") + + def test_marker_path_is_reached_without_any_consuming_call(self): + """The non-consuming path reaches the marker through _read_native_error, + never through _invoke_consume.""" + self.assertIn("_read_native_error", + inspect.getsource( + c2pa_module._check_ffi_operation_result)) + self.assertNotIn("_invoke_consume", + inspect.getsource( + c2pa_module._check_ffi_operation_result)) + # _read_native_error is what re-marks the slot after every read. + self.assertIn("_write_no_error_marker", + inspect.getsource(c2pa_module._read_native_error)) + class TestErrorsStillRaiseAfterCleanup(unittest.TestCase): """Each surface that lost a _clear_error_state() call still reports.""" @@ -9549,5 +10247,342 @@ def test_ed25519_sign_with_empty_data_raises(self): c2pa_module.ed25519_sign(b"", "not a key") +class TestConsumeOwnership(unittest.TestCase): + """Ownership of the native handle across the consuming call paths.""" + + def setUp(self): + self.freed = [] + self._real_free = ManagedResource._free_native_ptr + + def counting_free(ptr): + self.freed.append(ptr) + return self._real_free(ptr) + + ManagedResource._free_native_ptr = staticmethod(counting_free) + + def tearDown(self): + ManagedResource._free_native_ptr = staticmethod(self._real_free) + + def test_generic_exception_frees_the_reserved_handle(self): + """A reserved consume that raises must free, not drop, the handle. + + _begin_consume() leaves the resource CLOSED with the handle still set, + which _release_handle() reads as "not ours" and nulls without freeing, + while _abort_consume() can no longer restore it. + """ + def boom(handle): + raise RuntimeError("callback failed after the reservation") + + for name in ("_consume_no_replacement", "_consume_into"): + with self.subTest(helper=name): + resource = Settings() + self.freed.clear() + + with self.assertRaises(Error): + getattr(resource, name)(boom, "consume failed: {}") + + self.assertEqual( + len(self.freed), 1, + "{} dropped the handle without freeing it".format(name)) + self.assertIsNone(resource._handle) + + def test_marshalling_error_retains_the_handle(self): + """Positive control for the free counter. + + An ArgumentError means the call never reached native, so the handle is + untouched and must NOT be freed. Without this, a zero-free assertion + could pass because the counter never fires. + """ + def bad_marshal(handle): + raise ctypes.ArgumentError("marshalling failed") + + resource = Settings() + self.freed.clear() + + with self.assertRaises(ctypes.ArgumentError): + resource._consume_no_replacement(bad_marshal, "consume: {}") + + self.assertEqual(self.freed, []) + self.assertIsNotNone(resource._handle) + self.assertEqual(resource._lifecycle_state, LifecycleState.ACTIVE) + + def test_post_consume_failure_keeps_the_resource_closed(self): + """An error without a pre-consume tag means native took ownership. + + The value is native's to drop, so the resource stays closed and frees + nothing. + """ + resource = Settings() + self.freed.clear() + real_read = c2pa_module._read_native_error + c2pa_module._read_native_error = lambda: "Other: operation failed" + try: + with self.assertRaises(Error): + resource._consume_no_replacement(lambda h: 1, "consume: {}") + finally: + c2pa_module._read_native_error = real_read + + self.assertEqual(resource._lifecycle_state, LifecycleState.CLOSED) + self.assertEqual(self.freed, []) + + def test_failure_without_a_native_error_frees_the_handle(self): + """An empty error slot leaves ownership unknown, so the handle is + freed defensively rather than dropped. + """ + resource = Settings() + self.freed.clear() + real_read = c2pa_module._read_native_error + c2pa_module._read_native_error = lambda: None + try: + with self.assertRaises(Error): + resource._consume_no_replacement(lambda h: 1, "consume: {}") + finally: + c2pa_module._read_native_error = real_read + + self.assertEqual( + len(self.freed), 1, + "an unknown-ownership failure dropped the handle without freeing") + self.assertIsNone(resource._handle) + + def test_close_called_during_parallel_call(self): + """Parallel closes handling. + """ + resource = Settings() + spare = Settings() + replacement = spare._handle + # Only test should be able to free. + spare._handle = None + spare._lifecycle_state = LifecycleState.CLOSED + self.freed.clear() + + def close_then_swap(handle): + resource.close() + return replacement + + resource._consume_and_swap(close_then_swap, "swap: {}") + + self.assertIn(replacement, self.freed) + self.assertIsNone(resource._handle) + self.assertEqual(resource._lifecycle_state, LifecycleState.CLOSED) + + +class TestContextProviderContract(unittest.TestCase): + """The published ContextProvider contract is is_valid plus + execution_context, and nothing more. + """ + + class _MinimalProvider(ContextProvider): + """Implements exactly what the abstract base class declares.""" + + def __init__(self): + self._inner = Context(Settings()) + + @property + def is_valid(self): + return self._inner.is_valid + + @property + def execution_context(self): + return self._inner.execution_context + + def test_reader_accepts_a_minimal_provider(self): + provider = self._MinimalProvider() + try: + Reader("image/jpeg", io.BytesIO(b"not a real jpeg"), + context=provider) + except AttributeError as e: + self.fail("Reader requires more than the documented " + "ContextProvider contract: {}".format(e)) + except Error: + # Rejecting the bytes is the native library doing its job. + pass + + def test_builder_accepts_a_minimal_provider(self): + provider = self._MinimalProvider() + try: + Builder({"claim_generator": "test"}, context=provider) + except AttributeError as e: + self.fail("Builder requires more than the documented " + "ContextProvider contract: {}".format(e)) + + def test_built_in_context_still_gets_in_flight_protection(self): + """The compatibility shim must not silently drop the guard for the + provider that does implement it. + """ + context = Context(Settings()) + self.assertEqual(context._inflight, 0) + with c2pa_module._context_guard(context): + self.assertGreater( + context._inflight, 0, + "built-in Context lost its in-flight guard") + self.assertEqual(context._inflight, 0) + + +class TestLockOrderStaticAnalysis(unittest.TestCase): + """Static analysis over the source: no threads are spawned here. + """ + + def test_no_conflicting_lock_acquisition_order(self): + """No two locks may be nested in opposite orders by different methods. + + Two methods nesting the same pair of locks in opposite order is a + AB/BA deadlock shape: thread 1 holds A and waits for B while + thread 2 holds B and waits for A. + """ + tree = ast.parse(inspect.getsource(c2pa_module)) + + # Every self._X = threading.Lock()/RLock()/Condition() assignment, + # grouped by the class that owns it. + lock_attrs_by_class = {} + for cls in ast.walk(tree): + if not isinstance(cls, ast.ClassDef): + continue + found = set() + for node in ast.walk(cls): + if not (isinstance(node, ast.Assign) + and len(node.targets) == 1): + continue + target = node.targets[0] + if not (isinstance(target, ast.Attribute) + and isinstance(target.value, ast.Name) + and target.value.id == "self"): + continue + value = node.value + if (isinstance(value, ast.Call) + and isinstance(value.func, ast.Attribute) + and value.func.attr in + ("Lock", "RLock", "Condition")): + found.add(target.attr) + if found: + lock_attrs_by_class[cls.name] = found + + def lock_name_for_with(item): + """The lock attribute a `with` item enters, or None.""" + ctx = item.context_expr + # with self._fragment_lock: + if (isinstance(ctx, ast.Attribute) + and isinstance(ctx.value, ast.Name) + and ctx.value.id == "self" + and any(ctx.attr in attrs + for attrs in lock_attrs_by_class.values())): + return ctx.attr + # with self._guarded_op(): returns _op_lock itself, and the + # accessors return the lock they are named for. + lock_by_method = { + "_guarded_op": "_op_lock", + "_live_op_lock": "_op_lock", + "_live_teardown_lock": "_teardown_lock", + } + if (isinstance(ctx, ast.Call) + and isinstance(ctx.func, ast.Attribute) + and ctx.func.attr in lock_by_method + and isinstance(ctx.func.value, ast.Name) + and ctx.func.value.id == "self"): + return lock_by_method[ctx.func.attr] + return None + + def lock_name_for_acquire(node): + """A lock taken with acquire() and released in a finally nests just + as a `with` does, so the scan has to follow it or it silently stops + seeing whole regions. + """ + call = node.value if isinstance(node, ast.Expr) else node + if isinstance(call, ast.UnaryOp) and isinstance(call.op, ast.Not): + call = call.operand + if not (isinstance(call, ast.Call) + and isinstance(call.func, ast.Attribute) + and call.func.attr == "acquire"): + return None + owner = call.func.value + if (isinstance(owner, ast.Attribute) + and isinstance(owner.value, ast.Name) + and owner.value.id == "self" + and any(owner.attr in attrs + for attrs in lock_attrs_by_class.values())): + return owner.attr + return None + + def acquires_in_test(node): + """Lock taken by `if not self._X.acquire(...)`-style guards.""" + if isinstance(node, ast.If): + return lock_name_for_acquire(node.test) + return None + + def orders_in(node, stack, pairs): + """Record (outer, inner) for every nesting this node contains.""" + if isinstance(node, (ast.With, ast.AsyncWith)): + names = [n for n in (lock_name_for_with(i) + for i in node.items) if n] + for name in names: + if stack: + pairs.add((stack[-1], name)) + stack.append(name) + for child in node.body: + orders_in(child, stack, pairs) + for _ in names: + stack.pop() + return + # A statement list can open a lock partway through via acquire(); + # everything after it in that list is nested inside. + for field, value in ast.iter_fields(node): + if not isinstance(value, list): + continue + held = [] + for child in value: + if not isinstance(child, ast.stmt): + continue + name = (lock_name_for_acquire(child) + or acquires_in_test(child)) + if name: + if stack: + pairs.add((stack[-1], name)) + stack.append(name) + held.append(name) + continue + orders_in(child, stack, pairs) + for _ in held: + stack.pop() + for child in ast.iter_child_nodes(node): + if not isinstance(child, ast.stmt): + orders_in(child, stack, pairs) + + pairs_by_method = {} + for cls in ast.walk(tree): + if not isinstance(cls, ast.ClassDef): + continue + for method in cls.body: + if not isinstance(method, (ast.FunctionDef, + ast.AsyncFunctionDef)): + continue + pairs = set() + orders_in(method, [], pairs) + if pairs: + pairs_by_method[(cls.name, method.name)] = pairs + + all_pairs = set().union(*pairs_by_method.values()) \ + if pairs_by_method else set() + conflicts = [] + for (outer, inner) in all_pairs: + # Each conflicting pair appears twice in all_pairs, once per direction. + if outer >= inner or (inner, outer) not in all_pairs: + continue + forward = [k for k, v in pairs_by_method.items() + if (outer, inner) in v] + backward = [k for k, v in pairs_by_method.items() + if (inner, outer) in v] + conflicts.append( + "{} nests {} inside {}, but {} nests {} inside {}".format( + forward[0], inner, outer, backward[0], outer, inner)) + + self.assertGreater( + len(pairs_by_method), 0, + "lock nesting scan found no nested lock acquisitions: " + "the scan is broken") + self.assertEqual( + conflicts, [], + "conflicting lock acquisition order:\n " + + "\n ".join(sorted(set(conflicts)))) + + if __name__ == '__main__': unittest.main(warnings='ignore') diff --git a/tests/test_unit_tests_threaded.py b/tests/test_unit_tests_threaded.py index d0d0b2c1..457c2a55 100644 --- a/tests/test_unit_tests_threaded.py +++ b/tests/test_unit_tests_threaded.py @@ -11,22 +11,31 @@ # specific language governing permissions and limitations under # each license. +import ast +import contextlib import ctypes import gc import os +import re +import inspect import io import json +import subprocess +import sys +import textwrap import unittest import threading import concurrent.futures import time +import signal import asyncio import random from unittest.mock import MagicMock, patch from c2pa import Builder, C2paError as Error, Reader, C2paSigningAlg as SigningAlg, C2paSignerInfo, Signer, sdk_version # noqa: E501 from c2pa import Context, Settings -from c2pa.c2pa import ManagedResource, Stream, LifecycleState +from c2pa.c2pa import ManagedResource, Stream, LifecycleState, _native_section +import c2pa.c2pa as c2pa_module from c2pa.lib import is_foreign_process, record_owner_pid PROJECT_PATH = os.getcwd() @@ -58,6 +67,7 @@ def _make_stream(pid_offset): obj._closed = False obj._initialized = True obj._stream = MagicMock() # non-None stream handle + obj._close_lock = threading.Lock() if pid_offset is not None: obj._owner_pid = os.getpid() + pid_offset return obj @@ -178,6 +188,540 @@ def test_foreign_pid_close_marks_closed(self): self.assertFalse(obj._initialized) +class TestForkedChildDoesNotDeadlock(unittest.TestCase): + """A forked child must never block on a lock the parent held at fork(). + + Locking a resource for the duration of an operation means a child that + forks while some thread holds that lock inherits it locked, with the owner + thread gone. Anything in the child that acquires it waits forever. + + The failure mode is a hang: each operation runs on a worker thread and + is joined with a timeout: a test that called it directly would hang the + runner instead of failing. + """ + + _TIMEOUT = 5.0 + + def _foreign_reader_with_lock_held(self, fragment_lock=False): + """A Reader in the state a forked child inherits: + lock held by another thread, and stamped with a PID other than this process's. + """ + with open(DEFAULT_TEST_FILE, "rb") as asset: + reader = Reader("image/jpeg", asset) + holding = threading.Event() + release = threading.Event() + + def hold_the_lock(): + held = (reader._fragment_lock if fragment_lock + else reader._guarded_op()) + with held: + holding.set() + release.wait(30) + + holder = threading.Thread(target=hold_the_lock, daemon=True) + holder.start() + self.assertTrue(holding.wait(self._TIMEOUT), + "helper thread never acquired the lock") + self.addCleanup(holder.join, self._TIMEOUT) + self.addCleanup(release.set) + + reader._owner_pid = os.getpid() + 1 + return reader + + def _foreign_stream_with_close_lock_held(self): + """A Stream in the state a forked child inherits: _close_lock held by + a thread that does not exist in the child, and a foreign owner PID. + """ + stream = Stream(io.BytesIO(b"payload")) + holding = threading.Event() + release = threading.Event() + + def hold_the_lock(): + with stream._close_lock: + holding.set() + release.wait(30) + + holder = threading.Thread(target=hold_the_lock, daemon=True) + holder.start() + self.assertTrue(holding.wait(self._TIMEOUT), + "helper thread never acquired _close_lock") + # Cleanups run last-registered-first, so this one runs after + # release.set and holder.join. + self.addCleanup(self._reclaim_foreign_stream, stream) + self.addCleanup(holder.join, self._TIMEOUT) + self.addCleanup(release.set) + + stream._owner_pid = os.getpid() + 1 + return stream + + def _reclaim_foreign_stream(self, stream): + """Release a stream the foreign-process path left tracked.""" + stream._owner_pid = os.getpid() + stream._closed = False + stream.close() + + def test_stream_close_completes_with_close_lock_held(self): + """close() must take the foreign-process path without acquiring + _close_lock, which no surviving thread would release.""" + stream = self._foreign_stream_with_close_lock_held() + + outcome = self._run_with_timeout(stream.close) + + self.assertEqual(outcome, "ok", + "close() blocked on the inherited _close_lock") + self.assertTrue(stream._closed, + "close() returned without marking the stream closed") + self.assertFalse(stream._initialized) + + def _run_with_timeout(self, operation): + """Run operation on a worker; return 'ok', the exception, or None if it + was still running when the timeout expired.""" + result = {} + + def run(): + try: + operation() + result["outcome"] = "ok" + except BaseException as e: # noqa: BLE001 - asserted on below + result["outcome"] = e + + worker = threading.Thread(target=run, daemon=True) + worker.start() + worker.join(self._TIMEOUT) + return result.get("outcome") + + def test_locked_read_raises_instead_of_blocking(self): + reader = self._foreign_reader_with_lock_held() + outcome = self._run_with_timeout(reader.json) + self.assertIsNotNone( + outcome, "json() blocked on a lock inherited from the parent") + self.assertIsInstance(outcome, Error) + + def test_native_call_path_raises_instead_of_blocking(self): + reader = self._foreign_reader_with_lock_held() + outcome = self._run_with_timeout( + lambda: reader.resource_to_stream("any-uri", io.BytesIO())) + self.assertIsNotNone( + outcome, + "resource_to_stream() blocked on a lock inherited from the parent") + self.assertIsInstance(outcome, Error) + + def test_fragment_lock_path_raises_instead_of_blocking(self): + reader = self._foreign_reader_with_lock_held(fragment_lock=True) + outcome = self._run_with_timeout( + lambda: reader.with_fragment( + "video/mp4", io.BytesIO(b""), io.BytesIO(b""))) + self.assertIsNotNone( + outcome, + "with_fragment() blocked on a lock inherited from the parent") + self.assertIsInstance(outcome, Error) + + def test_close_still_completes(self): + reader = self._foreign_reader_with_lock_held() + self.assertEqual(self._run_with_timeout(reader.close), "ok", + "close() must neither block nor raise") + + def test_teardown_still_completes(self): + # Cleanup has to finish, not report an error. + reader = self._foreign_reader_with_lock_held() + self.assertEqual( + self._run_with_timeout( + lambda: reader._teardown(free_handle=True)), "ok", + "_teardown() must neither block nor raise") + self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED) + self.assertIsNone(reader._handle) + + def test_parent_copy_unaffected(self): + """The child closing its copy must leave the parent's usable. + + Runs in a subprocess so that the fork happens in a single-threaded + process. Operations that reach the network, such as reading an asset + with a remote manifest, start background native threads that outlive + the object that triggered them, and forking a multi-threaded process + can lead to issues. + """ + source = textwrap.dedent(""" + import os, sys + from c2pa import Reader + from c2pa.c2pa import LifecycleState + + asset_path = sys.argv[1] + with open(asset_path, "rb") as asset: + reader = Reader("image/jpeg", asset) + before = reader.json() + + pid = os.fork() + if pid == 0: + try: + reader.close() + os._exit(0) + except BaseException: + os._exit(1) + _, status = os.waitpid(pid, 0) + + assert status >> 8 == 0, "child could not close its own copy" + assert reader._lifecycle_state == LifecycleState.ACTIVE + assert reader.json() == before + reader.close() + print("OK") + """) + + result = subprocess.run( + [sys.executable, "-c", source, DEFAULT_TEST_FILE], + capture_output=True, text=True, timeout=120) + + self.assertEqual( + result.returncode, 0, + "parent copy was affected by the child (rc={}):\n{}".format( + result.returncode, result.stderr[-2000:])) + self.assertIn("OK", result.stdout) + self.assertNotIn("DeprecationWarning", result.stderr) + + +class TestReaderWithFragmentConcurrency(unittest.TestCase): + """with_fragment's native call and its stream-ownership transfer + must not interleave with another with_fragment on the same Reader. + """ + + def setUp(self): + self.init_path = os.path.join(FIXTURES_FOLDER, "dashinit.mp4") + self.fragment_path = os.path.join(FIXTURES_FOLDER, "dash1.m4s") + with open(self.init_path, "rb") as f: + self.init_bytes = f.read() + with open(self.fragment_path, "rb") as f: + self.fragment_bytes = f.read() + + def _advance(self, reader): + reader.with_fragment( + "video/mp4", + io.BytesIO(self.init_bytes), + io.BytesIO(self.fragment_bytes)) + + def test_close_during_with_fragment_does_not_double_close_stream(self): + with open(self.init_path, "rb") as init: + reader = Reader("video/mp4", init) + + entered_gap = threading.Event() + release_gap = threading.Event() + + real_consume_and_swap = reader._consume_and_swap + + def gated_consume_and_swap(ffi_call, error_message): + real_consume_and_swap(ffi_call, error_message) + # Pauses in with_fragment's window before it reassigns _own_stream/_fragment_streams. + entered_gap.set() + release_gap.wait(5) + + reader._consume_and_swap = gated_consume_and_swap + + result = {} + + def run_with_fragment(): + try: + with open(self.init_path, "rb") as init, \ + open(self.fragment_path, "rb") as frag: + reader.with_fragment("video/mp4", init, frag) + result["outcome"] = "ok" + except BaseException as e: # noqa: BLE001 - asserted below + result["outcome"] = e + + worker = threading.Thread(target=run_with_fragment, daemon=True) + worker.start() + self.assertTrue( + entered_gap.wait(5), + "with_fragment never reached the post-native-call gap") + + # close() must win the race, and with_fragment must not hang, + # crash, or succeed without signalling. + reader.close() + release_gap.set() + worker.join(5) + self.assertFalse(worker.is_alive(), "with_fragment hung") + self.assertIsInstance( + result.get("outcome"), Error, + "with_fragment must raise C2paError when it loses the race, " + "not hang, crash, or silently succeed") + + self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED) + # with_fragment must not resurrect these fields on a reader close() already tore down. + self.assertIsNone(reader._own_stream) + self.assertEqual(reader._fragment_streams, []) + + def _manifest_before_and_after_fragment(self): + """Tests the manifest a fresh Reader reports, + and the one it reports once a fragment has been processed. + """ + reader = Reader("video/mp4", io.BytesIO(self.init_bytes)) + try: + before = reader.json() + finally: + reader.close() + + reader = Reader("video/mp4", io.BytesIO(self.init_bytes)) + try: + self._advance(reader) + after = reader.json() + finally: + reader.close() + return before, after + + def test_read_during_swap_never_serves_the_previous_handles_manifest(self): + before, after = self._manifest_before_and_after_fragment() + self.assertNotEqual( + before, after, + "fixtures must differ before and after the fragment for this " + "test to mean anything") + + reader = Reader("video/mp4", io.BytesIO(self.init_bytes)) + # Populates the cache with the soon to be replaced handle. + self.assertEqual(reader.json(), before) + + real_lock = reader._guarded_op + at_gap = threading.Event() + leave_gap = threading.Event() + # _native_call takes this lock before the swap does, + # so park on the acquisition that performed the swap. + swapped = [] + + class GatedLock: + """Parks once after the swap's locked region releases.""" + + def __init__(self, inner): + self._inner = inner + + def __enter__(self): + return self._inner.__enter__() + + def __exit__(self, exc_type, exc_val, exc_tb): + performed_swap = reader._own_stream is not None and ( + reader._own_stream not in swapped) + result = self._inner.__exit__(exc_type, exc_val, exc_tb) + if performed_swap and not at_gap.is_set(): + at_gap.set() + leave_gap.wait(10) + return result + + swapped.append(reader._own_stream) + reader._guarded_op = lambda **kw: GatedLock(real_lock(**kw)) + + served = {} + + def advance(): + try: + self._advance(reader) + except Error as e: + served["advance"] = e + + def read_in_gap(): + try: + served["json"] = reader.json() + except Error as e: + served["json"] = e + + advancer = threading.Thread(target=advance, daemon=True) + advancer.start() + self.assertTrue(at_gap.wait(10), "never reached the post-swap gap") + + gap_reader = threading.Thread(target=read_in_gap, daemon=True) + gap_reader.start() + gap_reader.join(10) + + leave_gap.set() + advancer.join(10) + + try: + self.assertFalse(gap_reader.is_alive(), "json() hung in the gap") + # Smoke test comparison. + names = {before: "the replaced handle's manifest", + after: "the current handle's manifest"} + self.assertEqual( + names.get(served.get("json"), "something else"), + "the current handle's manifest", + "json() must not be served a manifest cached from the " + "handle with_fragment already replaced") + finally: + reader._guarded_op = real_lock + reader.close() + + def test_manifest_accessors_stay_consistent_while_fragments_advance(self): + """get_active_manifest() parses the cached JSON, + so its read and write of the cache must not prevent a clean handle swap. + """ + before, after = self._manifest_before_and_after_fragment() + valid = {before, after} + + reader = Reader("video/mp4", io.BytesIO(self.init_bytes)) + stop = threading.Event() + unexpected = [] + served = [] + swaps = [] + + def read_manifest(): + while not stop.is_set(): + try: + if reader.get_active_manifest() is not None: + served.append(reader.json()) + except Error: + pass + except BaseException as e: # noqa: BLE001 - asserted below + unexpected.append(repr(e)) + + def advance(): + while not stop.is_set(): + try: + self._advance(reader) + swaps.append(None) + except Error: + pass + except BaseException as e: # noqa: BLE001 - asserted below + unexpected.append(repr(e)) + time.sleep(0.001) + + workers = ([threading.Thread(target=read_manifest, daemon=True) + for _ in range(3)] + + [threading.Thread(target=advance, daemon=True) + for _ in range(2)]) + for t in workers: + t.start() + time.sleep(0.3) + stop.set() + for t in workers: + t.join(10) + + try: + self.assertFalse( + [t for t in workers if t.is_alive()], + "a manifest accessor or fragment advance hung") + self.assertEqual(unexpected, []) + self.assertTrue(served, "no manifest was ever read") + self.assertGreater( + len(swaps), 1, + "fragments did not advance during the run") + self.assertTrue( + set(served) <= valid, + "a manifest was served that matches neither the pre- nor the " + "post-fragment state") + finally: + reader.close() + + def test_interleaved_with_fragment_leaves_reader_consistent(self): + reader = Reader("video/mp4", io.BytesIO(self.init_bytes)) + + # Parks one call between its native call + # and its stream bookkeeping. + real_consume_and_swap = reader._consume_and_swap + in_gap = threading.Event() + contended = threading.Event() + leave_gap = threading.Event() + + def gated_consume_and_swap(ffi_call, error_message): + real_consume_and_swap(ffi_call, error_message) + if not in_gap.is_set(): + in_gap.set() + leave_gap.wait(10) + + reader._consume_and_swap = gated_consume_and_swap + + class ContentionReportingLock: + """Flags when a caller finds the lock it wraps already held. + + with_fragment takes this lock with acquire(blocking=False) and + releases it in a finally, so those are the methods wrapped here. + """ + + def __init__(self, inner): + self._inner = inner + + def acquire(self, blocking=True, timeout=-1): + if not blocking: + acquired = self._inner.acquire(blocking=False) + if not acquired: + # The second caller is refused rather than parked, + # which is the mutual exclusion this test checks for. + contended.set() + return acquired + if not self._inner.acquire(blocking=False): + contended.set() + return self._inner.acquire(blocking, timeout) + return True + + def release(self): + self._inner.release() + + def __enter__(self): + self.acquire() + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.release() + return False + + real_fragment_lock = reader._fragment_lock + reader._fragment_lock = ContentionReportingLock(real_fragment_lock) + + outcomes = {} + installed_by_second = {} + + def first(): + try: + self._advance(reader) + outcomes["first"] = "ok" + except Error as e: + outcomes["first"] = e + + def second(): + try: + self._advance(reader) + outcomes["second"] = "ok" + # The streams matching the handle this call swapped in. + installed_by_second["own"] = reader._own_stream + installed_by_second["fragments"] = list( + reader._fragment_streams) + except Error as e: + outcomes["second"] = e + + t1 = threading.Thread(target=first, daemon=True) + t1.start() + self.assertTrue(in_gap.wait(10), "never reached the bookkeeping gap") + + t2 = threading.Thread(target=second, daemon=True) + t2.start() + # Unset when the lock is bypassed, which is the case this test guards against. + contended.wait(5) + + leave_gap.set() + t1.join(10) + self.assertFalse(t1.is_alive(), "first with_fragment hung") + t2.join(10) + self.assertFalse(t2.is_alive(), "second with_fragment hung") + + try: + if outcomes.get("second") != "ok": + # A refused second call never swapped, + # so the first call's streams are the right ones. + self.assertIsInstance(outcomes["second"], Error) + else: + # Both swapped, so the reader must retain one call's streams. + self.assertIs( + reader._own_stream, installed_by_second["own"], + "reader retains a different call's stream than the one " + "its live native handle reads through") + self.assertEqual( + list(reader._fragment_streams), + installed_by_second["fragments"]) + + retained = [reader._own_stream] + list(reader._fragment_streams) + for wrapper in retained: + self.assertIsNotNone(wrapper) + self.assertFalse( + wrapper._closed, + "reader retained a released stream wrapper") + finally: + reader._consume_and_swap = real_consume_and_swap + reader._fragment_lock = real_fragment_lock + reader.close() + + class TestHelpers(unittest.TestCase): def test_record_and_detect_own_pid(self): @@ -699,12 +1243,12 @@ def setUp(self): with open(os.path.join(self.data_dir, "es256_private.key"), "rb") as key_file: self.key = key_file.read() - # Create a local Es256 signer with certs and a timestamp server + # Create a local Es256 signer with certs and no timestamp server. self.signer_info = C2paSignerInfo( alg=b"es256", sign_cert=self.certs, private_key=self.key, - ta_url=b"http://timestamp.digicert.com" + ta_url=None ) self.signer = Signer.from_info(self.signer_info) @@ -2917,10 +3461,260 @@ def thread_work(thread_id): self.assertNotEqual(current_manifest["active_manifest"], thread_manifest_data[other_thread_id]["active_manifest"]) -class TestManagedResourceCrossThread(unittest.TestCase): - """Tests cross-thread resources handling, especially closing/releasind. +class TestWithFragmentReentrancy(unittest.TestCase): + """with_fragment drives caller-supplied stream callbacks, so it must not + hold a lock a callback-spawned thread would wait on. """ + def test_reentrant_call_is_refused_rather_than_blocked(self): + init_path = os.path.join(FIXTURES_FOLDER, "dashinit.mp4") + fragment_path = os.path.join(FIXTURES_FOLDER, "dash1.m4s") + with open(init_path, "rb") as handle: + init_bytes = handle.read() + with open(fragment_path, "rb") as handle: + fragment_bytes = handle.read() + + reader = Reader("video/mp4", io.BytesIO(init_bytes)) + state = {"fired": False, "result": None, "hung": None} + + class ReentrantStream(io.BytesIO): + """Re-enters the API from another thread, from inside a callback, + and waits for it: the shape that deadlocks a lock held across the + native call. + """ + + def _reenter_once(self): + if state["fired"]: + return + state["fired"] = True + + def second_call(): + try: + reader.with_fragment( + "video/mp4", + io.BytesIO(init_bytes), + io.BytesIO(fragment_bytes)) + state["result"] = "completed" + except Error as e: + state["result"] = e + + thread = threading.Thread(target=second_call, daemon=True) + thread.start() + thread.join(10) + state["hung"] = thread.is_alive() + + def read(self, size=-1): + self._reenter_once() + return super().read(size) + + def seek(self, offset, whence=0): + self._reenter_once() + return super().seek(offset, whence) + + reader.with_fragment("video/mp4", + ReentrantStream(init_bytes), + io.BytesIO(fragment_bytes)) + + self.assertTrue(state["fired"], "the callback never re-entered") + self.assertFalse( + state["hung"], + "a with_fragment call started from a stream callback blocked on " + "the lock the running call holds") + self.assertIsInstance( + state["result"], Error, + "the re-entrant call must be refused, not interleaved") + + def test_same_thread_reentry_does_not_corrupt_the_reader(self): + """_fragment_lock is reentrant, so a callback calling with_fragment + synchronously passes the guard. The native layer rejects the handle it + already consumed, and the Reader survives. + """ + init_path = os.path.join(FIXTURES_FOLDER, "dashinit.mp4") + fragment_path = os.path.join(FIXTURES_FOLDER, "dash1.m4s") + with open(init_path, "rb") as handle: + init_bytes = handle.read() + with open(fragment_path, "rb") as handle: + fragment_bytes = handle.read() + + reader = Reader("video/mp4", io.BytesIO(init_bytes)) + state = {"fired": False, "inner": None} + + class SelfReentrantStream(io.BytesIO): + def _reenter_once(self): + if state["fired"]: + return + state["fired"] = True + try: + reader.with_fragment("video/mp4", + io.BytesIO(init_bytes), + io.BytesIO(fragment_bytes)) + state["inner"] = "completed" + except Error as e: + state["inner"] = e + + def read(self, size=-1): + self._reenter_once() + return super().read(size) + + def seek(self, offset, whence=0): + self._reenter_once() + return super().seek(offset, whence) + + reader.with_fragment("video/mp4", + SelfReentrantStream(init_bytes), + io.BytesIO(fragment_bytes)) + + self.assertTrue(state["fired"], "the callback never re-entered") + self.assertIsInstance( + state["inner"], Error, + "a nested consume on the same handle must be rejected") + # The outer call still owns a live handle. + self.assertTrue(reader.is_valid) + self.assertIsInstance(reader.json(), str) + + def test_refused_call_leaves_the_reader_usable(self): + """The refusal reports contention without touching the Reader, so the + caller can retry once the other thread returns. + """ + init_path = os.path.join(FIXTURES_FOLDER, "dashinit.mp4") + fragment_path = os.path.join(FIXTURES_FOLDER, "dash1.m4s") + with open(init_path, "rb") as handle: + init_bytes = handle.read() + with open(fragment_path, "rb") as handle: + fragment_bytes = handle.read() + + reader = Reader("video/mp4", io.BytesIO(init_bytes)) + + holding = threading.Event() + release = threading.Event() + + def hold_the_guard(): + reader._fragment_lock.acquire() + holding.set() + release.wait(10) + reader._fragment_lock.release() + + holder = threading.Thread(target=hold_the_guard, daemon=True) + holder.start() + self.assertTrue(holding.wait(5), "the guard was never taken") + + with self.assertRaises(Error): + reader.with_fragment("video/mp4", + io.BytesIO(init_bytes), + io.BytesIO(fragment_bytes)) + + # Refused before any stream was built or handle consumed. + self.assertTrue(reader.is_valid) + + release.set() + holder.join(5) + + # The same call succeeds once the other thread is out. + reader.with_fragment("video/mp4", + io.BytesIO(init_bytes), + io.BytesIO(fragment_bytes)) + self.assertTrue(reader.is_valid) + + +class TestStreamCloseReentrancy(unittest.TestCase): + """close() clears the callback references inside _close_lock, which can run + a finalizer at that bytecode boundary, and __del__ takes the same lock. + """ + + def test_close_can_be_reentered_on_the_same_thread(self): + stream = Stream(io.BytesIO(b"payload")) + finished = threading.Event() + + def hold_then_reenter(): + with stream._close_lock: + # A finalizer running here re-takes the lock this thread holds. + stream.close() + finished.set() + + worker = threading.Thread(target=hold_then_reenter, daemon=True) + worker.start() + + self.assertTrue( + finished.wait(10), + "close() blocked re-entering _close_lock from the thread that " + "already holds it") + self.assertTrue(stream._closed) + + +class TestConsumeReservationWindow(unittest.TestCase): + """The consume reservation must outlast ownership classification. + + _read_native_error() is a native call that releases the GIL, so a resource + restored to ACTIVE before the error is classified is visible as usable to + another thread while native may already own its handle. + """ + + def test_no_thread_sees_a_consumed_handle_as_valid(self): + resource = Settings() + + reading = threading.Event() + may_finish = threading.Event() + seen_valid = [] + + real_read = c2pa_module._read_native_error + + def gated_read(): + # Stand in for the GIL release inside the real native call. + reading.set() + may_finish.wait(10) + # No pre-consume tag: native took ownership and then failed. + return "Other: operation failed after taking ownership" + + def observer(): + if not reading.wait(10): + return + # The consuming call is mid-classification at this point. + seen_valid.append(resource.is_valid) + may_finish.set() + + watcher = threading.Thread(target=observer, daemon=True) + watcher.start() + + c2pa_module._read_native_error = gated_read + try: + with self.assertRaises(Error): + resource._consume_no_replacement(lambda h: 1, "consume: {}") + finally: + c2pa_module._read_native_error = real_read + may_finish.set() + watcher.join(10) + + self.assertTrue(seen_valid, "observer never sampled the resource") + self.assertFalse( + seen_valid[0], + "another thread saw a resource whose handle native may already " + "own as valid") + + +class TestLocking(unittest.TestCase): + """Tests for the locks that guard native resources: + - the per-object operation lock that serializes native calls against teardown, + - the fragment lock, + - cross-thread creation/closing/releasing. + + Every join here is bounded: + A deadlock must fail the test when timing out, not hang the suite. + """ + + JOIN_TIMEOUT = 30 + + @classmethod + def setUpClass(cls): + cls.data_dir = FIXTURES_FOLDER + with open(DEFAULT_TEST_FILE, 'rb') as handle: + cls.image_bytes = handle.read() + with open(os.path.join(FIXTURES_FOLDER, + "es256_certs.pem"), 'rb') as handle: + cls.certs = handle.read() + with open(os.path.join(FIXTURES_FOLDER, + "es256_private.key"), 'rb') as handle: + cls.private_key = handle.read() + def setUp(self): # Flush pending finalizers through the real free first. gc.collect() @@ -2931,6 +3725,31 @@ def setUp(self): def tearDown(self): ManagedResource._free_native_ptr = self._real_free + def _join_all(self, threads, what): + for thread in threads: + thread.join(self.JOIN_TIMEOUT) + stuck = [t for t in threads if t.is_alive()] + self.assertEqual( + stuck, [], + "{} did not finish within {}s: deadlock".format( + what, self.JOIN_TIMEOUT)) + + def _run_isolated(self, body, timeout=180): + """Run body in a subprocess and return it, + so that crashes can be caught and do not crash the suite itself. + """ + source = textwrap.dedent(body) + return subprocess.run( + [sys.executable, "-c", source], + cwd=os.path.dirname(os.path.dirname(os.path.abspath(__file__))), + capture_output=True, + timeout=timeout, + ) + + def _make_signer(self): + return Signer.from_info(C2paSignerInfo( + SigningAlg.ES256, self.certs, self.private_key, None)) + def _free_counts(self): counts = {} for handle in self.freed: @@ -2994,6 +3813,251 @@ def make_and_drop(index): self.assertEqual(set(counts.values()), {1}, "a dropped resource was freed more than once") + def test_cross_closing_inside_lock_regions_does_not_deadlock(self): + """Tests cocnurrent closes do not deadlock. + """ + first = _ConcreteResource() + first._activate(0x40001) + second = _ConcreteResource() + second._activate(0x40002) + + holding = threading.Barrier(2, timeout=5) + queued = threading.Barrier(2, timeout=5) + failures = [] + + def worker(mine, theirs): + try: + with mine._guarded_op(): + # Both locks required, + holding.wait() + theirs.close() + # Teardowns queue. + queued.wait() + except BaseException as error: + failures.append(error) + + threads = [ + threading.Thread(target=worker, args=(first, second), daemon=True), + threading.Thread(target=worker, args=(second, first), daemon=True), + ] + for thread in threads: + thread.start() + self._join_all(threads, "cross-closing workers") + + self.assertEqual(failures, [], "workers raised: {}".format(failures)) + + counts = {handle: value + for handle, value in self._free_counts().items() + if handle in (0x40001, 0x40002)} + self.assertEqual(counts, {0x40001: 1, 0x40002: 1}, + "cross-closed handles were not each freed once") + + def test_failed_locked_region_still_flushes_a_queued_teardown(self): + resource = _ConcreteResource() + resource._activate(0x50001) + + holding = threading.Event() + queued = threading.Event() + + def holder(): + try: + with resource._guarded_op(): + holding.set() + queued.wait(self.JOIN_TIMEOUT) + raise RuntimeError("locked region failed") + except RuntimeError: + pass + + def closer(): + holding.wait(self.JOIN_TIMEOUT) + resource.close() + queued.set() + + threads = [ + threading.Thread(target=holder, daemon=True), + threading.Thread(target=closer, daemon=True), + ] + for thread in threads: + thread.start() + self._join_all(threads, "failing locked region") + + self.assertEqual(self._free_counts().get(0x50001), 1, + "a teardown queued during the region was orphaned") + + def test_close_racing_a_consumed_handle_does_not_free_it(self): + resource = _ConcreteResource() + resource._activate(0x50002) + + resource._inflight = 1 + resource._teardown(free_handle=False) + self.assertIs(resource._pending_teardown, False, + "the consume was not recorded") + + resource._inflight = 0 + resource.close() + self.assertIsNone(self._free_counts().get(0x50002), + "a consumed handle was freed by a racing close") + + resource._maybe_flush_pending() + self.assertIsNone(self._free_counts().get(0x50002), + "a later flush freed a consumed handle") + + def test_close_against_a_bare_lock_holder_is_not_orphaned(self): + resource = _ConcreteResource() + resource._activate(0x50003) + + holding = threading.Event() + release = threading.Event() + + def holder(): + with resource._live_op_lock(): + holding.set() + release.wait(self.JOIN_TIMEOUT) + resource._release_handle() + + def closer(): + holding.wait(self.JOIN_TIMEOUT) + resource.close() + release.set() + + threads = [ + threading.Thread(target=holder, daemon=True), + threading.Thread(target=closer, daemon=True), + ] + for thread in threads: + thread.start() + self._join_all(threads, "bare lock holder") + + self.assertEqual(self._free_counts().get(0x50003), 1, + "a teardown queued against the lock was orphaned") + + def test_close_queued_inside_a_flush_hold_is_not_orphaned(self): + resource = _ConcreteResource() + resource._activate(0x60001) + + real_lock = resource._op_lock + closed = threading.Event() + join_timeout = self.JOIN_TIMEOUT + + class GatedLock: + def acquire(self, blocking=True, timeout=-1): + if timeout == -1: + return real_lock.acquire(blocking) + return real_lock.acquire(blocking, timeout) + + def release(self): + return real_lock.release() + + def __enter__(self): + real_lock.acquire() + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + if not closed.is_set(): + worker = threading.Thread( + target=lambda: (resource.close(), closed.set()), + daemon=True) + worker.start() + worker.join(join_timeout) + real_lock.release() + return False + + resource._op_lock = GatedLock() + try: + resource._maybe_flush_pending() + finally: + resource._op_lock = real_lock + + self.assertEqual(self._free_counts().get(0x60001), 1, + "a teardown queued during a flush was orphaned") + + def test_close_recording_after_a_flush_is_not_orphaned(self): + resource = _ConcreteResource() + resource._activate(0x70001) + + real_lock = resource._op_lock + real_record = ManagedResource._record_pending_intent + reached_record = threading.Event() + flusher_done = threading.Event() + closer_done = threading.Event() + join_timeout = self.JOIN_TIMEOUT + + def gated_record(target, free_handle): + if (target is resource + and threading.current_thread().name == "delayed-closer"): + reached_record.set() + flusher_done.wait(join_timeout) + return real_record(target, free_handle) + + class GatedLock: + def acquire(self, blocking=True, timeout=-1): + if timeout == -1: + return real_lock.acquire(blocking) + return real_lock.acquire(blocking, timeout) + + def release(self): + return real_lock.release() + + def __enter__(self): + real_lock.acquire() + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + if not closer_done.is_set() and not reached_record.is_set(): + worker = threading.Thread( + target=lambda: (resource.close(), closer_done.set()), + name="delayed-closer", + daemon=True) + worker.start() + reached_record.wait(join_timeout) + real_lock.release() + return False + + ManagedResource._record_pending_intent = gated_record + resource._op_lock = GatedLock() + try: + resource._maybe_flush_pending() + finally: + resource._op_lock = real_lock + flusher_done.set() + closer_done.wait(join_timeout) + ManagedResource._record_pending_intent = real_record + + self.assertEqual(self._free_counts().get(0x70001), 1, + "a teardown recorded after a flush was orphaned") + + def test_stream_finalizer_does_not_block_on_a_held_close_lock(self): + stream = Stream(io.BytesIO(self.image_bytes)) + self.addCleanup(stream.close) + + holding = threading.Event() + release = threading.Event() + returned = threading.Event() + + def holder(): + with stream._close_lock: + holding.set() + release.wait(self.JOIN_TIMEOUT) + + def finalizer(): + stream.__del__() + returned.set() + + holder_thread = threading.Thread(target=holder, daemon=True) + holder_thread.start() + self.assertTrue(holding.wait(self.JOIN_TIMEOUT), + "holder never took the close lock") + + finalizer_thread = threading.Thread(target=finalizer, daemon=True) + finalizer_thread.start() + finalizer_thread.join(5) + blocked = not returned.is_set() + + release.set() + self._join_all([holder_thread, finalizer_thread], "stream finalizer") + self.assertFalse(blocked, + "__del__ waited for a close lock held elsewhere") + def test_settings_relayed_across_threads_stays_usable(self): ManagedResource._free_native_ptr = self._real_free @@ -3034,6 +4098,1805 @@ def build_context_and_builder(): self.assertTrue(valid) self.assertEqual(settings._owner_pid, pid) + def test_json_racing_finalizer_does_not_crash(self): + """Readers used on one thread while others are collected. + """ + result = self._run_isolated(""" + import sys, io, gc, random, threading, time + sys.path.insert(0, "src") + from c2pa import Reader + + data = open("tests/fixtures/C.jpg", "rb").read() + stop = threading.Event() + pool, lock = [], threading.Lock() + + def worker(): + while not stop.is_set(): + choice = random.random() + try: + if choice < 0.40: + reader = Reader("image/jpeg", io.BytesIO(data)) + with lock: + pool.append(reader) + elif choice < 0.75: + with lock: + snapshot = list(pool) + if snapshot: + reader = random.choice(snapshot) + reader._manifest_json_str_cache = None + reader.json() + elif choice < 0.90: + with lock: + reader = pool.pop(0) if pool else None + if reader: + reader.close() + else: + with lock: + if len(pool) > 20: + del pool[0:5] + gc.collect() + except Exception: + pass + + threads = [threading.Thread(target=worker) for _ in range(12)] + for thread in threads: + thread.start() + deadline = time.time() + 10 + while time.time() < deadline: + time.sleep(0.05) + stop.set() + for thread in threads: + thread.join(30) + """) + self.assertEqual( + result.returncode, 0, + "reader churn crashed with {} " + "(139=SIGSEGV, 134=SIGABRT): {}".format( + result.returncode, result.stderr.decode()[-800:])) + + def test_finalizer_inside_locked_operation(self): + """A finalizer can run at any bytecode boundary, including inside a + region this same thread has locked. + A non-reentrant lock deadlocks here, but RLock does not. + """ + resource = _ConcreteResource() + resource._activate(0x51000) + observed = [] + + class Dropped: + def __del__(self): + # Runs on this thread, inside the locked region body() + # holds. + with resource._guarded_op(): + observed.append(True) + + def body(): + with resource._guarded_op(): + dropped = Dropped() + del dropped + gc.collect() + + thread = threading.Thread(target=body) + thread.start() + self._join_all([thread], "finalizer inside locked region") + self.assertEqual(observed, [True], + "finalizer did not re-enter the lock") + resource.close() + + def test_close_racing_json_does_not_deadlock(self): + """close() on one thread against json() on another.""" + data = self.image_bytes + errors = [] + + def rounds(): + try: + for _ in range(40): + reader = Reader("image/jpeg", io.BytesIO(data)) + closer = threading.Thread(target=reader.close) + closer.start() + try: + reader._manifest_json_str_cache = None + reader.json() + except Error: + pass + closer.join(self.JOIN_TIMEOUT) + if closer.is_alive(): + errors.append("closer stuck") + return + except Exception as exc: + errors.append(repr(exc)) + + threads = [threading.Thread(target=rounds) for _ in range(4)] + for thread in threads: + thread.start() + self._join_all(threads, "close/json race") + self.assertEqual(errors, []) + + def test_context_manager_exit_racing_json_does_not_deadlock(self): + """__exit__ closes while another thread is calling json().""" + data = self.image_bytes + errors = [] + + def body(): + try: + for _ in range(40): + reader = Reader("image/jpeg", io.BytesIO(data)) + + def use(): + for _ in range(5): + try: + reader._manifest_json_str_cache = None + reader.json() + except Error: + pass + + user = threading.Thread(target=use) + user.start() + with reader: + pass + user.join(self.JOIN_TIMEOUT) + if user.is_alive(): + errors.append("user stuck") + return + except Exception as exc: + errors.append(repr(exc)) + + thread = threading.Thread(target=body) + thread.start() + self._join_all([thread], "__exit__/json race") + self.assertEqual(errors, []) + + def test_consume_failure_teardown_does_not_deadlock(self): + """A failing consuming call tears the handle down from inside the + operation, re-entering the lock on the same thread. + + with_fragment on a JPEG returns NotSupported, which routes through + _raise_consume_failure (on purpose). + """ + data = self.image_bytes + errors = [] + + def body(): + try: + for _ in range(20): + reader = Reader("image/jpeg", io.BytesIO(data)) + try: + reader.with_fragment( + "image/jpeg", io.BytesIO(data), io.BytesIO(data)) + except Error: + pass + reader.close() + except Exception as exc: + errors.append(repr(exc)) + + thread = threading.Thread(target=body) + thread.start() + self._join_all([thread], "consume-failure teardown") + self.assertEqual(errors, []) + + def test_close_during_sign_does_not_deadlock(self): + """_sign_internal calls self.close() inside its own try block, + so signing re-enters the lock on the signing thread. + """ + certs = self.certs + key = self.private_key + data = self.image_bytes + signer_info = C2paSignerInfo( + alg=b"es256", + sign_cert=certs, + private_key=key, + ta_url=None, + ) + manifest = { + "claim_generator": "python_test", + "claim_generator_info": [ + {"name": "python_test", "version": "0.0.1"}], + "format": "image/jpeg", + "assertions": [], + } + errors = [] + + def body(): + try: + for _ in range(3): + signer = Signer.from_info(signer_info) + builder = Builder(manifest) + builder.sign(signer, "image/jpeg", + io.BytesIO(data), io.BytesIO()) + except Exception as exc: + errors.append(repr(exc)) + + threads = [threading.Thread(target=body) for _ in range(4)] + for thread in threads: + thread.start() + self._join_all(threads, "sign with internal close") + self.assertEqual(errors, []) + + def test_stream_callback_reentering_api_does_not_deadlock(self): + """Construction drives caller-supplied stream callbacks, + and a caller may call back into the API from one. + + This passes because construction does not hold the lock. + """ + data = self.image_bytes + other = Reader("image/jpeg", io.BytesIO(data)) + errors = [] + + class ReentrantStream(io.BytesIO): + def readinto(self, buffer): + try: + other.json() + except Exception: + pass + return super().readinto(buffer) + + def body(): + try: + for _ in range(10): + Reader("image/jpeg", ReentrantStream(data)) + except Exception as exc: + errors.append(repr(exc)) + + thread = threading.Thread(target=body) + thread.start() + self._join_all([thread], "callback re-entering API") + self.assertEqual(errors, []) + other.close() + + def test_stream_callback_blocking_on_other_thread_does_not_deadlock(self): + """A stream callback that blocks on another thread + which touches the same object. + + A lock held across construction deadlocks here, whether it is global + or per-object. + """ + data = self.image_bytes + target = Reader("image/jpeg", io.BytesIO(data)) + errors = [] + + class BlockingStream(io.BytesIO): + def readinto(self, buffer): + def use(): + try: + target._manifest_json_str_cache = None + target.json() + except Exception: + pass + + helper = threading.Thread(target=use) + helper.start() + helper.join(10) + if helper.is_alive(): + errors.append("helper stuck inside stream callback") + return super().readinto(buffer) + + def body(): + try: + for _ in range(5): + Reader("image/jpeg", BlockingStream(data)) + except Exception as exc: + errors.append(repr(exc)) + + thread = threading.Thread(target=body) + thread.start() + self._join_all([thread], "callback blocking on another thread") + self.assertEqual(errors, []) + target.close() + + def test_no_nested_op_locks(self): + """No code path may hold two resources' operation locks at once. + With only one lock ever held, no cycle can form here. + """ + data = self.image_bytes + held = threading.local() + violations = [] + real_lock = ManagedResource._guarded_op + real_live_op_lock = ManagedResource._live_op_lock + + def make_tracking(real): + def tracking(resource, **kw): + lock = real(resource, **kw) + depth = getattr(held, 'stack', None) + if depth is None: + depth = held.stack = [] + + class Tracked: + def __enter__(self): + others = [r for r in depth if r is not resource] + if others: + violations.append( + "{} while holding {}".format( + type(resource).__name__, + [type(o).__name__ for o in others])) + depth.append(resource) + return lock.__enter__() + + def __exit__(self, *exc): + depth.pop() + return lock.__exit__(*exc) + + return Tracked() + return tracking + + ManagedResource._guarded_op = make_tracking(real_lock) + ManagedResource._live_op_lock = make_tracking(real_live_op_lock) + try: + reader = Reader("image/jpeg", io.BytesIO(data)) + reader.json() + reader.detailed_json() + reader.is_embedded() + reader.get_remote_url() + reader.close() + finally: + ManagedResource._guarded_op = real_lock + ManagedResource._live_op_lock = real_live_op_lock + + self.assertEqual(violations, [], + "a thread held two operation locks at once") + + def test_concurrent_storm_terminates(self): + """Readers, closers and collection running together must all finish.""" + data = self.image_bytes + stop = threading.Event() + shared = [Reader("image/jpeg", io.BytesIO(data))] + errors = [] + + def reader_worker(): + while not stop.is_set(): + try: + current = shared[0] + current._manifest_json_str_cache = None + current.json() + except Exception: + pass + + def closer_worker(): + while not stop.is_set(): + try: + shared[0].close() + shared[0] = Reader("image/jpeg", io.BytesIO(data)) + gc.collect() + except Exception as exc: + errors.append(repr(exc)) + return + + threads = [threading.Thread(target=reader_worker) for _ in range(6)] + threads += [threading.Thread(target=closer_worker) for _ in range(2)] + for thread in threads: + thread.start() + deadline = time.time() + 5 + while time.time() < deadline: + time.sleep(0.05) + stop.set() + self._join_all(threads, "concurrent storm") + self.assertEqual(errors, []) + + def test_native_section_deferred_free_is_thread_local(self): + """Two threads each with their own open native-error section: one + thread's section closing must not flush a free deferred inside + the other thread's still-open section. + """ + freed = self._counted_free() + resource = _ConcreteResource() + resource._activate(0x1001) + + thread_ready = threading.Event() + release_thread = threading.Event() + + def worker(): + with _native_section(): + resource.close() + thread_ready.set() + release_thread.wait(self.JOIN_TIMEOUT) + # Flush happens here, on the worker thread, once its own + # section closes. + + thread = threading.Thread(target=worker) + thread.start() + try: + self.assertTrue( + thread_ready.wait(self.JOIN_TIMEOUT), + "worker thread did not reach its open section in time") + + # A section opened and closed entirely on this (main) thread, + # while the worker's section is still open on its own thread. + with _native_section(): + pass + + self.assertEqual( + freed, [], + "a different thread's section flushed this thread's " + "pending resource") + finally: + release_thread.set() + self._join_all([thread], "native-section worker") + + self.assertEqual(freed, [0x1001], + "worker thread's own section never flushed") + + def _counted_free(self): + """Patch _free_native_ptr to count frees; returns the list.""" + freed = [] + real = ManagedResource._free_native_ptr + + def counting(ptr): + freed.append(ptr) + return real(ptr) + + ManagedResource._free_native_ptr = staticmethod(counting) + self.addCleanup( + lambda: setattr(ManagedResource, '_free_native_ptr', real)) + return freed + + def _thumbnail_uri(self, reader): + manifests = json.loads(reader.json()).get("manifests", {}) + for manifest in manifests.values(): + thumbnail = manifest.get("thumbnail") + if thumbnail and thumbnail.get("identifier"): + return thumbnail["identifier"] + self.skipTest("fixture has no thumbnail resource to stream") + + def test_close_inside_callback_defers_free(self): + """A close() from inside a stream callback must not free the handle + the native call is still using.""" + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + during = [] + + class Closer(io.BytesIO): + def write(self, buffer): + reader.close() + during.append(len(freed)) + return super().write(buffer) + + try: + reader.resource_to_stream(uri, Closer()) + except Error: + pass + + self.assertEqual(during, [0], "handle was freed mid-call") + self.assertEqual(len(freed), 1, "deferred free did not run once") + self.assertEqual(reader._inflight, 0) + self.assertIsNone(reader._pending_teardown) + self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED) + + def test_with_fragment_closes_main_stream_when_second_stream_fails(self): + """Streams in with_fragment on failure must not get into a broken state""" + opened = [] + real_init = Stream.__init__ + + def tracking_init(wrapper, source): + if opened: + raise ValueError("fragment stream could not be built") + real_init(wrapper, source) + opened.append(wrapper) + + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + self.addCleanup(reader.close) + + with patch.object(Stream, '__init__', tracking_init): + with self.assertRaises(ValueError): + reader.with_fragment( + "video/mp4", + io.BytesIO(self.image_bytes), + io.BytesIO(self.image_bytes)) + + self.assertEqual(len(opened), 1, "main stream was never built") + self.assertTrue(opened[0].closed, + "main stream was left open for the collector") + + def test_cross_thread_close_during_callback_defers_free(self): + """A close() from inside a stream callback must not free the handle + the native call is still using.""" + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + during = [] + started = threading.Event() + + class Slow(io.BytesIO): + def write(self, buffer): + started.set() + time.sleep(0.3) + during.append(len(freed)) + return super().write(buffer) + + def closer(): + started.wait(self.JOIN_TIMEOUT) + reader.close() + + thread = threading.Thread(target=closer) + thread.start() + try: + reader.resource_to_stream(uri, Slow()) + except Error: + pass + self._join_all([thread], "cross-thread closer") + + self.assertEqual(during, [0], "handle was freed mid-call") + self.assertEqual(len(freed), 1) + self.assertEqual(reader._inflight, 0) + + def test_deferred_teardown_still_closes(self): + """After a deferred free the resource is closed and a later + close() frees nothing.""" + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + + class Closer(io.BytesIO): + def write(self, buffer): + reader.close() + return super().write(buffer) + + try: + reader.resource_to_stream(uri, Closer()) + except Error: + pass + + self.assertEqual(len(freed), 1) + reader.close() + self.assertEqual(len(freed), 1, "second close() freed again") + self.assertIsNone(reader._handle) + + def test_use_after_deferred_close_is_rejected(self): + """Deferring must not leave the resource usable: + the free is pending, so the handle is about to go away.""" + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + states = [] + + class Closer(io.BytesIO): + def write(self, buffer): + reader.close() + states.append(reader._lifecycle_state) + try: + reader.json() + states.append("json succeeded") + except Error: + states.append("json rejected") + return super().write(buffer) + + try: + reader.resource_to_stream(uri, Closer()) + except Error: + pass + + self.assertEqual(states[0], LifecycleState.CLOSED) + self.assertEqual(states[1], "json rejected") + + def test_exception_from_callback_still_frees(self): + """An exception unwinding through the native call must not + leave the inflight-handler hanging.""" + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + + class Exploding(io.BytesIO): + def write(self, buffer): + reader.close() + raise RuntimeError("callback failure") + + try: + reader.resource_to_stream(uri, Exploding()) + except Exception: + pass + + self.assertEqual(reader._inflight, 0, "in-flight counter stranded") + self.assertEqual(len(freed), 1, "deferred free did not run") + + def test_inflight_cleared_before_deferred_free(self): + """The counter must reach zero before the deferred free runs. + + _teardown defers whenever _inflight is above zero, so performing the + free while the counter is still raised would defer it a second time + and the handle would never be released. + """ + seen = [] + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + real_release = Reader._release + + def probing_release(self): + seen.append(self._inflight) + return real_release(self) + + class Closer(io.BytesIO): + def write(self, buffer): + reader.close() + return super().write(buffer) + + with patch.object(Reader, '_release', probing_release): + try: + reader.resource_to_stream(uri, Closer()) + except Error: + pass + + self.assertEqual(seen, [0], + "deferred free ran while still counted in flight") + self.assertIsNone(reader._handle) + + def test_release_raising_during_deferred_teardown_does_not_leak(self): + """The deferred free survives a failing _release: + the handle must still be freed.""" + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + + def boom(self): + raise RuntimeError("release failure") + + class Closer(io.BytesIO): + def write(self, buffer): + reader.close() + return super().write(buffer) + + with patch.object(Reader, '_release', boom): + try: + reader.resource_to_stream(uri, Closer()) + except Error: + pass + + self.assertEqual(reader._inflight, 0) + self.assertEqual(len(freed), 1, "handle leaked when _release raised") + + def test_concurrent_closes_during_callback_free_once(self): + """Many threads closing while one native call is in flight + must produce exactly one free (avoid double-frees, + or freeing something the object wouldn't own).""" + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + uri = self._thumbnail_uri(reader) + started = threading.Event() + closers = [] + + class Slow(io.BytesIO): + def write(self, buffer): + started.set() + time.sleep(0.3) + return super().write(buffer) + + def closer(): + started.wait(self.JOIN_TIMEOUT) + reader.close() + + for _ in range(8): + thread = threading.Thread(target=closer) + closers.append(thread) + thread.start() + try: + reader.resource_to_stream(uri, Slow()) + except Error: + pass + self._join_all(closers, "concurrent closers") + + self.assertEqual(len(freed), 1, + "racing closers freed {} times".format(len(freed))) + self.assertEqual(reader._inflight, 0) + + def _borrow_resource(self): + """An ACTIVE resource with no native handle behind it.""" + res = _ConcreteResource() + res._lifecycle_state = LifecycleState.ACTIVE + res._handle = ctypes.c_void_p(1) + return res + + def test_consume_during_foreign_borrow_raises(self): + """A consume must refuse to start while another thread borrows. + """ + res = self._borrow_resource() + borrowing = threading.Event() + release = threading.Event() + + def borrower(): + with res._native_call(): + borrowing.set() + release.wait(self.JOIN_TIMEOUT) + + thread = threading.Thread(target=borrower) + thread.start() + try: + self.assertTrue(borrowing.wait(self.JOIN_TIMEOUT), + "borrower never entered the native call") + with self.assertRaises(Error) as caught: + res._consume_no_replacement(lambda h: 0, "unused: {}") + self.assertIn("in use", str(caught.exception)) + self.assertEqual( + res._lifecycle_state, LifecycleState.ACTIVE, + "a refused consume must leave the resource usable") + self.assertIsNotNone(res._handle) + finally: + release.set() + self._join_all([thread], "borrower") + + def test_unborrowed_consume_proceeds(self): + """A consume with nothing in flight runs and closes the resource. + + The guard rejects on any in-flight count, so a consuming call must not + wrap itself in _native_call(): the callers pin the handle by marking + the resource CLOSED under the lock instead. + """ + res = self._borrow_resource() + res._consume_no_replacement(lambda h: 0, "unused: {}") + self.assertEqual(res._lifecycle_state, LifecycleState.CLOSED) + + def test_consume_inside_own_borrow_is_refused(self): + """A consume is refused even when this thread owns the borrow. + + The guard counts frames, not threads. A consuming call nested in a + _native_call() would hand a pointer to native while that same frame + still expects it back, so no such nesting is allowed. + """ + res = self._borrow_resource() + with res._native_call(): + with self.assertRaises(Error): + res._consume_no_replacement(lambda h: 0, "unused: {}") + self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE) + + def test_refused_consume_leaves_borrow_counts_intact(self): + """A refused consume must not disturb the in-flight bookkeeping.""" + res = self._borrow_resource() + borrowing = threading.Event() + release = threading.Event() + + def borrower(): + with res._native_call(): + borrowing.set() + release.wait(self.JOIN_TIMEOUT) + + thread = threading.Thread(target=borrower) + thread.start() + try: + self.assertTrue(borrowing.wait(self.JOIN_TIMEOUT)) + with self.assertRaises(Error): + res._consume_no_replacement(lambda h: 0, "unused: {}") + self.assertEqual(res._inflight, 1, "the real borrow was lost") + finally: + release.set() + self._join_all([thread], "borrower") + self.assertEqual(res._inflight, 0) + + def test_failed_consume_restores_active_state(self): + """A call that did not take the handle must leave it usable. + """ + res = self._borrow_resource() + with patch('c2pa.c2pa._read_native_error', + return_value="Other: UntrackedPointer: 0x1"): + with self.assertRaises(Exception): + res._consume_no_replacement(lambda h: -1, "rejected: {}") + self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE, + "a retained handle was left marked closed") + self.assertIsNotNone(res._handle) + + def test_consume_raising_restores_active_state(self): + """An exception from the native call must not leave a stale mark.""" + res = self._borrow_resource() + + def boom(handle): + raise ctypes.ArgumentError("marshalling failed") + + with self.assertRaises(ctypes.ArgumentError): + res._consume_no_replacement(boom, "unused: {}") + self.assertEqual(res._lifecycle_state, LifecycleState.ACTIVE) + + def test_deferred_consume_is_not_upgraded_to_free(self): + """A deferred consuming teardown must not be overwritten by a later + free intent arriving while the same call is still in flight. + + Scenario: a Signer shared across concurrent signs: sign borrows + the handle (holding the in-flight guard) while Context.__init__ + consumes it. + """ + freed = self._counted_free() + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + releases = [] + orig_release = reader._release + + def counting_release(): + releases.append(1) + orig_release() + + reader._release = counting_release + + with reader._native_call(): + # The consuming call: native took ownership, so nothing here frees. + reader._teardown(free_handle=False) + self.assertFalse( + reader._pending_teardown, + "consuming teardown did not record free_handle=False") + + # A free intent arriving behind it, past a stale state check. + reader._teardown(free_handle=True) + self.assertFalse( + reader._pending_teardown, + "recorded consume was upgraded back to a free") + + self.assertEqual( + freed, [], + "freed a handle the native library already owns") + self.assertEqual( + len(releases), 1, + "_release() ran {} times, expected once".format(len(releases))) + self.assertEqual(reader._inflight, 0) + self.assertIsNone(reader._pending_teardown) + self.assertEqual(reader._lifecycle_state, LifecycleState.CLOSED) + + def test_concurrent_close_runs_release_once(self): + """Two racing close() calls on one instance must run _release() + exactly once. + + The native free is already single (the handle is nulled after the + first teardown), so a free-counting test cannot see this: it is + What must not run twice is _release(), the Python-side + stream/cache cleanup a subclass overrides. _teardown() has to be + idempotent under its own lock. + + Gate _teardown so the first close() pauses on entry, before taking + the lock; the second then runs a full teardown (release + free + + mark closed); the first resumes and must find the resource already + released and do nothing. + """ + join_timeout = self.JOIN_TIMEOUT + orig_teardown = ManagedResource._teardown + + for _ in range(20): + reader = Reader("image/jpeg", io.BytesIO(self.image_bytes)) + release_calls = [] + orig_release = reader._release + + def counting_release(_orig=orig_release, _calls=release_calls): + _calls.append(1) + _orig() + + reader._release = counting_release + + call_count = {"n": 0} + count_lock = threading.Lock() + first_arrived = threading.Event() + release_first = threading.Event() + + def gated_teardown(self, free_handle, _target=reader, + _timeout=join_timeout): + if self is _target: + with count_lock: + call_count["n"] += 1 + is_first = call_count["n"] == 1 + if is_first: + first_arrived.set() + release_first.wait(_timeout) + return orig_teardown(self, free_handle) + + with patch.object(ManagedResource, '_teardown', gated_teardown): + t1 = threading.Thread(target=reader.close) + t1.start() + self.assertTrue( + first_arrived.wait(join_timeout), + "first close() never reached _teardown()") + + t2 = threading.Thread(target=reader.close) + t2.start() + t2.join(join_timeout) + self.assertFalse( + t2.is_alive(), + "second close() should complete unblocked while the " + "first is paused") + + release_first.set() + self._join_all([t1], "paused close() resuming") + + self.assertEqual( + len(release_calls), 1, + "_release() ran {} times for one instance across racing " + "close() calls; _teardown() must be idempotent under its " + "own lock".format(len(release_calls))) + + def test_sign_with_internal_close_frees_once(self): + """_sign_internal closes the Builder inside its own try, + so the close defers and the free happens on the way out.""" + freed = self._counted_free() + signer_info = C2paSignerInfo( + alg=b"es256", + sign_cert=self.certs, + private_key=self.private_key, + ta_url=None, + ) + manifest = { + "claim_generator": "python_test", + "claim_generator_info": [ + {"name": "python_test", "version": "0.0.1"}], + "format": "image/jpeg", + "assertions": [], + } + signer = Signer.from_info(signer_info) + builder = Builder(manifest) + builder.sign(signer, "image/jpeg", + io.BytesIO(self.image_bytes), io.BytesIO()) + + self.assertEqual(builder._lifecycle_state, LifecycleState.CLOSED) + self.assertEqual(builder._inflight, 0) + builder_frees = [f for f in freed if f is not None] + self.assertGreaterEqual(len(builder_frees), 1) + with self.assertRaises(Error): + builder.sign(signer, "image/jpeg", + io.BytesIO(self.image_bytes), io.BytesIO()) + + def test_class_a_construction_is_not_guarded(self): + """Construction is unguarded: no external caller holds a reference yet. + """ + entered = [] + real = ManagedResource._native_call + + def recording(resource): + entered.append(type(resource).__name__) + return real(resource) + + ManagedResource._native_call = recording + try: + Reader("image/jpeg", io.BytesIO(self.image_bytes)) + finally: + ManagedResource._native_call = real + + self.assertEqual(entered, [], + "construction entered _native_call: guarding it " + "reintroduces the callback deadlock") + + def test_every_callback_running_method_is_guarded(self): + """Every method that hands a Stream to the native lib must be guarded, + except the construction paths. + """ + source = inspect.getsource(sys.modules[Reader.__module__]) + lines = source.split("\n") + class_a = { + ("Reader", "_create_reader"), + ("Reader", "_init_from_context"), + ("Builder", "from_archive"), + } + stream_use = re.compile( + r"(_stream|stream_obj|source_stream|dest_stream|main_obj" + r"|frag_obj)\._stream") + + bodies = {} + current_class = current_method = None + start = None + for index, line in enumerate(lines): + if re.match(r"^class ", line): + current_class = line.split("(")[0].replace( + "class ", "").strip(":") + if re.match(r"^def ", line): + current_class = None + match = re.match(r"^ def (\w+)", line) + if match: + if current_class and current_method and start is not None: + bodies[(current_class, current_method)] = "\n".join( + lines[start:index]) + current_method = match.group(1) + start = index + if current_class and current_method and start is not None: + bodies[(current_class, current_method)] = "\n".join(lines[start:]) + + unguarded = [] + checked = 0 + for key, body in bodies.items(): + if not stream_use.search(body): + continue + checked += 1 + if key in class_a: + continue + if ("_native_call()" not in body + and "_exclusive_native_call()" not in body + and "_consume_and_swap(" not in body): + unguarded.append("{}.{}".format(*key)) + + self.assertGreater(checked, 0, "coverage scan found no methods") + self.assertEqual(unguarded, []) + + def test_every_borrowed_handle_is_guarded(self): + """When a method hands a second object's handle to the native library, + that object needs its own _native_call() guard. + + This can happen in callbacks, where you can't express whose handle + is the one needing attention. + """ + module = sys.modules[Reader.__module__] + tree = ast.parse(inspect.getsource(module)) + + # Attributes that carry a native handle out of an object. + handle_attrs = {"_handle", "execution_context"} + + def guarded_names(node): + """Names X guarded at this node, by either form: + `with X._native_call():`, or `with _context_guard(X):` for a + caller-supplied ContextProvider, which enters X._native_call() + when X offers it. + """ + found = set() + for item in getattr(node, "items", []): + call = item.context_expr + if not isinstance(call, ast.Call): + continue + if (isinstance(call.func, ast.Attribute) + and call.func.attr == "_native_call" + and isinstance(call.func.value, ast.Name)): + found.add(call.func.value.id) + elif (isinstance(call.func, ast.Name) + and call.func.id == "_context_guard" + and call.args + and isinstance(call.args[0], ast.Name)): + found.add(call.args[0].id) + return found + + def borrowed_in_call(call): + """Names X whose handle this _lib.* call receives, X not self.""" + if not (isinstance(call.func, ast.Attribute) + and isinstance(call.func.value, ast.Name) + and call.func.value.id == "_lib"): + return set() + names = set() + for arg in ast.walk(call): + if (isinstance(arg, ast.Attribute) + and arg.attr in handle_attrs + and isinstance(arg.value, ast.Name) + and arg.value.id != "self"): + names.add(arg.value.id) + return names + + def locally_owned(method): + """A resource created inside the method never escapes to another + thread, so nothing can close it mid-call and it needs no guard. + """ + owned = set() + for node in ast.walk(method): + # `with self._NativeBuilder() as nb:` / `x = Foo()` + if isinstance(node, (ast.With, ast.AsyncWith)): + for item in node.items: + if (isinstance(item.context_expr, ast.Call) + and isinstance(item.optional_vars, ast.Name)): + owned.add(item.optional_vars.id) + elif isinstance(node, ast.Assign): + if isinstance(node.value, ast.Call): + for target in node.targets: + if isinstance(target, ast.Name): + owned.add(target.id) + return owned + + unguarded = [] + checked = 0 + + for cls in ast.walk(tree): + if not isinstance(cls, ast.ClassDef): + continue + for method in cls.body: + if not isinstance(method, (ast.FunctionDef, + ast.AsyncFunctionDef)): + continue + owned = locally_owned(method) + + # Walk the body tracking which guards are open, so a borrowed + # handle is only accepted when its own guard encloses the use. + def visit(node, active): + nonlocal checked + if isinstance(node, (ast.With, ast.AsyncWith)): + active = active | guarded_names(node) + if isinstance(node, ast.Call): + for name in borrowed_in_call(node) - owned: + checked += 1 + if name not in active: + unguarded.append( + "{}.{} passes {}._handle to native " + "without {}._native_call()".format( + cls.name, method.name, name, name)) + for child in ast.iter_child_nodes(node): + visit(child, active) + + visit(method, frozenset()) + + self.assertGreater( + checked, 0, + "ownership scan found no borrowed handles: the scan is broken") + self.assertEqual( + unguarded, [], + "borrowed handles used without their own guard:\n " + + "\n ".join(unguarded)) + + def test_consume_during_concurrent_sign_does_not_crash(self): + """Consuming a shared Signer must not free it under a live sign. + + Runs in a subprocess: the failure mode is a segfault, which would take + the test runner down with it otherwise. + """ + source = textwrap.dedent(""" + import io, os, sys, threading, time + from c2pa import (Builder, Context, Signer, C2paSignerInfo, + C2paSigningAlg as SigningAlg) + + data_dir = sys.argv[1] + certs_path = os.path.join(data_dir, "es256_certs.pem") + key_path = os.path.join(data_dir, "es256_private.key") + certs = open(certs_path, "rb").read() + key = open(key_path, "rb").read() + img = open(os.path.join(data_dir, "C.jpg"), "rb").read() + manifest = {"claim_generator_info": + [{"name": "test", "version": "0.1"}], + "assertions": []} + + signer = Signer.from_info(C2paSignerInfo( + SigningAlg.ES256, certs, key, None)) + stop = threading.Event() + + def sign(): + while not stop.is_set(): + try: + builder = Builder(manifest) + builder.sign(signer, "image/jpeg", + io.BytesIO(img), io.BytesIO()) + builder.close() + except Exception: + # A consumed signer may be rejected; + # only a crash is a failure here. + pass + + threads = [threading.Thread(target=sign) for _ in range(6)] + for t in threads: + t.start() + time.sleep(0.4) + try: + Context(signer=signer) + except Exception: + # Refusing the consume while borrows are live is the fix. + pass + stop.set() + for t in threads: + t.join() + print("OK") + """) + + result = subprocess.run( + [sys.executable, "-c", source, self.data_dir], + capture_output=True, text=True, timeout=300) + + self.assertNotEqual( + result.returncode, -11, + "SIGSEGV: a signer was consumed while a sign was using its handle") + self.assertEqual( + result.returncode, 0, + "shared-signer consume race failed (rc={}):\n{}".format( + result.returncode, result.stderr[-2000:])) + self.assertIn("OK", result.stdout) + + def _callback_signer_source(self): + """Shared subprocess preamble: an ES256 callback signer.""" + return """ + import io, os, sys, threading, time + from c2pa import (Builder, Context, Signer, + C2paSigningAlg as SigningAlg) + from cryptography.hazmat.primitives import hashes, serialization + from cryptography.hazmat.primitives.asymmetric import ec + + data_dir = sys.argv[1] + certs = open(os.path.join(data_dir, + "es256_certs.pem"), "rb").read().decode() + key_path = os.path.join(data_dir, "es256_private.key") + key = open(key_path, "rb").read() + img = open(os.path.join(data_dir, "C.jpg"), "rb").read() + manifest = {"claim_generator_info": + [{"name": "test", "version": "0.1"}], + "assertions": []} + private_key = serialization.load_pem_private_key( + key, password=None) + + def sign_callback(data): + return private_key.sign(data, ec.ECDSA(hashes.SHA256())) + + def make_context(): + return Context(signer=Signer.from_callback( + sign_callback, SigningAlg.ES256, certs, + "http://timestamp.digicert.com")) +""" + + def test_context_close_during_context_sign_does_not_crash(self): + """Closing a Context must not free the signer callback mid-sign. + + Context.__init__ pins the consumed signer's ctypes callback so it + outlives the Signer object, and Context._release() drops that pin. + Without an in-flight guard on the Context, a close() on another thread + runs _release() while c2pa_builder_sign_context is calling through the + trampoline, and the process dies with SIGSEGV. + + Runs in a subprocess: the failure mode is a segfault, which would take + the test runner down with it otherwise. + """ + source = textwrap.dedent(self._callback_signer_source() + """ + for trial in range(60): + ctx = make_context() + entered = threading.Event() + + def worker(): + try: + builder = Builder(dict(manifest), context=ctx) + entered.set() + builder.sign("image/jpeg", io.BytesIO(img), + io.BytesIO()) + builder.close() + except Exception: + # A closed context may be rejected; + # only a crash is a failure here. + entered.set() + + t = threading.Thread(target=worker) + t.start() + entered.wait(5) + time.sleep(0.002) + ctx.close() + t.join(20) + print("OK") + """) + + result = subprocess.run( + [sys.executable, "-c", source, self.data_dir], + capture_output=True, text=True, timeout=300) + + self.assertNotEqual( + result.returncode, -11, + "SIGSEGV: the signer callback was freed while native was " + "calling it") + self.assertEqual( + result.returncode, 0, + "context-close-during-sign race failed (rc={}):\n{}".format( + result.returncode, result.stderr[-2000:])) + self.assertIn("OK", result.stdout) + + def test_context_close_during_sign_defers_teardown(self): + """A close() arriving mid-sign defers instead of releasing. + + The callback pin and the native handle both have to survive until the + call in flight finishes, so a sign already running is never cut short. + """ + context = Context() + with context._native_call(): + context.close() + self.assertEqual(context._lifecycle_state, LifecycleState.CLOSED, + "close() must mark the context closed at once") + self.assertIsNotNone( + context._pending_teardown, + "the teardown should be recorded, not performed") + self.assertFalse( + context._released, + "_release() ran while a native call was still in flight") + self.assertTrue(context._handle, + "the handle was freed mid-call") + + self.assertTrue(context._released, + "the deferred teardown never ran") + self.assertIsNone(context._pending_teardown) + + def test_deferred_teardown_survives_a_flush_inside_a_section(self): + """A flush blocked by a section must re-register, not drop the free. + + The teardown defers on _inflight, so it is queued for the in-flight + call rather than for a section. When that call finishes inside a + section opened later on this thread, the flush cannot free yet, and + without re-registering nothing would ever free this handle. + """ + context = Context() + freed = [] + real_free = ManagedResource._free_native_ptr + ManagedResource._free_native_ptr = staticmethod( + lambda ptr: (freed.append(ptr), real_free(ptr))[1]) + try: + with context._native_call(): + closer = threading.Thread(target=context.close) + closer.start() + closer.join() + self.assertIsNotNone( + context._pending_teardown, + "close() during a native call should defer") + section = _native_section() + section.__enter__() + + self.assertEqual( + freed, [], + "the flush freed while a native section was still open") + self.assertIsNotNone( + context._pending_teardown, + "the deferral was dropped") + + section.__exit__(None, None, None) + self.assertEqual( + len(freed), 1, + "the deferred teardown was stranded and never freed") + self.assertIsNone(context._pending_teardown) + finally: + ManagedResource._free_native_ptr = real_free + + def test_abort_consume_leaves_a_queued_teardown_closed(self): + """A resource whose free is already queued must not become usable. + + The deferred free still runs when the section drains, so restoring + ACTIVE would hand the caller a resource that closes underneath it. + """ + context = Context() + with _native_section(): + context.close() + self.assertIsNotNone(context._pending_teardown) + + context._abort_consume(LifecycleState.ACTIVE) + self.assertEqual( + context._lifecycle_state, LifecycleState.CLOSED, + "a resource with a queued teardown was revived") + self.assertFalse( + context.is_valid, + "a resource with a queued teardown reported itself usable") + + def test_section_drain_error_does_not_mask_the_body_error(self): + """The body's exception is what the caller asked for, so it wins.""" + + class FlushRaises: + _pending_teardown = True + + def _maybe_flush_pending(self): + raise RuntimeError("flush failed") + + class BodyError(Exception): + pass + + with self.assertLogs('c2pa', level='ERROR') as logs: + with self.assertRaises(BodyError): + with _native_section(): + c2pa_module._register_for_section_flush(FlushRaises()) + raise BodyError("the error the caller cares about") + + self.assertTrue( + any("flush failed" in line for line in logs.output), + "the flush failure was not logged") + + def test_drain_errors_log(self): + """Log flushing failures.""" + + class FlushRaises: + _pending_teardown = True + + def _maybe_flush_pending(self): + raise RuntimeError("flush failed") + + with self.assertLogs("c2pa", level="ERROR") as captured: + with _native_section(): + c2pa_module._register_for_section_flush(FlushRaises()) + self.assertTrue( + any("flush failed" in message for message in captured.output)) + + def test_context_sign_after_close_raises_rather_than_skipping_signer(self): + """Signing through a closed Context must raise, not silently succeed. + + Context._release() has already dropped the pinned callback, so the + native side signs without ever invoking it: the call returns a + manifest of the same size while the caller's signing callback runs + zero times. Refusing the call is what makes that visible. + + Runs in a subprocess because the callback signer needs the + cryptography package, which this module does not otherwise import. + """ + source = textwrap.dedent(self._callback_signer_source() + """ + calls = [] + + def counting_callback(data): + calls.append(1) + return private_key.sign(data, ec.ECDSA(hashes.SHA256())) + + signer = Signer.from_callback( + counting_callback, SigningAlg.ES256, certs, + "http://timestamp.digicert.com") + ctx = Context(signer=signer) + builder = Builder(dict(manifest), context=ctx) + ctx.close() + + try: + builder.sign("image/jpeg", io.BytesIO(img), io.BytesIO()) + print("SIGNED_WITH_CALLS", len(calls)) + except Exception as exc: + print("RAISED", type(exc).__name__, len(calls)) + """) + + result = subprocess.run( + [sys.executable, "-c", source, self.data_dir], + capture_output=True, text=True, timeout=300) + + self.assertEqual(result.returncode, 0, result.stderr[-2000:]) + self.assertIn( + "RAISED", result.stdout, + "signing through a closed context returned a manifest its " + "signer callback never produced: {}".format(result.stdout.strip())) + self.assertIn("0", result.stdout.split()[-1]) + + def test_close_during_concurrent_sign_does_not_crash(self): + """A Signer shared across threads must not be freed mid-sign. + + Builder.sign borrows the signer's handle for the duration of the + native call. Without a guard on the signer itself, a close() on + another thread frees that handle while c2pa_builder_sign is using + it, and the process dies with SIGSEGV instead of raising. + + Rotates a shared signer while other threads sign with it. Runs in a + subprocess: the failure mode is a segfault, which would take the + test runner down with it otherwise. + """ + source = textwrap.dedent(""" + import io, os, sys, threading + from c2pa import (Builder, Signer, C2paSignerInfo, + C2paSigningAlg as SigningAlg) + + data_dir = sys.argv[1] + certs = open(os.path.join(data_dir, "es256_certs.pem"), "rb").read() + key = open(os.path.join(data_dir, "es256_private.key"), "rb").read() + img = open(os.path.join(data_dir, "C.jpg"), "rb").read() + manifest = {"claim_generator_info": + [{"name": "test", "version": "0.1"}], + "assertions": []} + + def make(): + return Signer.from_info(C2paSignerInfo( + SigningAlg.ES256, certs, key, None)) + + box = {"signer": make(), "stop": False} + + def rotate(): + while not box["stop"]: + old = box["signer"] + try: + box["signer"] = make() + old.close() + except Exception: + pass + + def sign(): + for _ in range(120): + if box["stop"]: + return + try: + b = Builder(manifest) + b.sign(box["signer"], "image/jpeg", + io.BytesIO(img), io.BytesIO()) + b.close() + except Exception: + # A closed signer may be rejected; + # only a crash is a failure here. + pass + + rot = threading.Thread(target=rotate, daemon=True) + rot.start() + threads = [threading.Thread(target=sign) for _ in range(8)] + for t in threads: + t.start() + for t in threads: + t.join() + box["stop"] = True + rot.join(timeout=5) + print("OK") + """) + + result = subprocess.run( + [sys.executable, "-c", source, self.data_dir], + capture_output=True, text=True, timeout=300) + + self.assertNotEqual( + result.returncode, -11, + "SIGSEGV: a signer was freed while a sign was using its handle") + self.assertEqual( + result.returncode, 0, + "shared-signer teardown race failed (rc={}):\n{}".format( + result.returncode, result.stderr[-2000:])) + self.assertIn("OK", result.stdout) + + +class TestSwapConsumeExclusion(unittest.TestCase): + """with_archive, with_fragment must be rejected during other in-flight calls. + """ + + _MANIFEST = { + "claim_generator": "c2pa_python_test", + "claim_generator_info": [{ + "name": "c2pa_python_test", + "version": "0.1.0", + }], + "format": "image/jpeg", + "title": "Python Test", + "ingredients": [], + "assertions": [], + } + + def _archive_bytes(self): + builder = Builder(self._MANIFEST) + try: + archive = io.BytesIO() + builder.to_archive(archive) + archive.seek(0) + return archive + finally: + builder.close() + + def test_with_archive_rejected_when_to_archive_in_progress(self): + archive = self._archive_bytes() + builder = Builder(self._MANIFEST) + + inside = threading.Event() + release = threading.Event() + + class BlockingSink(io.BytesIO): + def write(self, data): + inside.set() + release.wait(10) + return super().write(data) + + def seek(self, *args): + inside.set() + release.wait(10) + return super().seek(*args) + + borrow_errors = [] + + def borrow(): + try: + builder.to_archive(BlockingSink()) + except Exception as e: # noqa: BLE001 - asserted below + borrow_errors.append(e) + + worker = threading.Thread(target=borrow, daemon=True) + worker.start() + try: + self.assertTrue( + inside.wait(10), "to_archive never reached its callback") + + with self.assertRaises(Error) as raised: + builder.with_archive(archive) + self.assertIn("in use", str(raised.exception)) + finally: + release.set() + worker.join(10) + + self.assertFalse(worker.is_alive(), "to_archive hung") + self.assertEqual(borrow_errors, []) + + # The refusal must leave the builder untouched and usable. + self.assertEqual(builder._lifecycle_state, LifecycleState.ACTIVE) + builder.add_action('{"action": "c2pa.color_adjustments"}') + builder.close() + + def test_with_fragment_rejected_when_native_in_progress(self): + init_path = os.path.join(FIXTURES_FOLDER, "dashinit.mp4") + fragment_path = os.path.join(FIXTURES_FOLDER, "dash1.m4s") + with open(init_path, "rb") as f: + init_bytes = f.read() + with open(fragment_path, "rb") as f: + fragment_bytes = f.read() + + reader = Reader("video/mp4", io.BytesIO(init_bytes)) + try: + with reader._native_call(): + with self.assertRaises(Error) as raised: + reader.with_fragment( + "video/mp4", + io.BytesIO(init_bytes), + io.BytesIO(fragment_bytes)) + self.assertIn("in use", str(raised.exception)) + + # The refusal must leave the reader untouched: the swap still + # works once the borrow is gone. + self.assertEqual(reader._lifecycle_state, LifecycleState.ACTIVE) + reader.with_fragment( + "video/mp4", + io.BytesIO(init_bytes), + io.BytesIO(fragment_bytes)) + reader.json() + finally: + reader.close() + + def test_close_during_with_archive_defers_and_frees(self): + archive_bytes = self._archive_bytes().getvalue() + builder = Builder(self._MANIFEST) + + inside = threading.Event() + release = threading.Event() + + class BlockingArchive(io.BytesIO): + def read(self, *args): + inside.set() + release.wait(10) + return super().read(*args) + + def seek(self, *args): + inside.set() + release.wait(10) + return super().seek(*args) + + outcome = {} + + def consume(): + try: + builder.with_archive(BlockingArchive(archive_bytes)) + outcome["result"] = "ok" + except Exception as e: # noqa: BLE001 - asserted below + outcome["result"] = e + + worker = threading.Thread(target=consume, daemon=True) + worker.start() + try: + self.assertTrue( + inside.wait(10), "with_archive never reached its callback") + # Defers: the swap is counted in flight. + builder.close() + finally: + release.set() + worker.join(10) + + self.assertFalse(worker.is_alive(), "with_archive hung") + # The deferred teardown freed the replacement handle: closed for + # good, nothing left to free, exactly one release. + self.assertEqual(builder._lifecycle_state, LifecycleState.CLOSED) + self.assertIsNone(builder._handle) + self.assertTrue(builder._released) + self.assertIsNone(builder._pending_teardown) + + def test_calling_close_should_not_corrupt_other_objects(self): + """Other threads asking for close() should not corrupt objects. + """ + real_free = ManagedResource._free_native_ptr + + k = 1 + while True: + archive = self._archive_bytes() + builder = Builder(self._MANIFEST) + + freed = [] + ManagedResource._free_native_ptr = staticmethod( + lambda p, _real=real_free: (freed.append(int( + ctypes.cast(p, ctypes.c_void_p).value or 0)), + _real(p))[1]) + + real_live_op_lock = builder._live_op_lock + enters = [0] + injected = [] + + class LockProxy: + def __init__(self, inner): + self._inner = inner + + def __enter__(self): + enters[0] += 1 + self._n = enters[0] + self._inner.__enter__() + return self + + def __exit__(self, *exc): + result = self._inner.__exit__(*exc) + if self._n == k and not injected: + injected.append(True) + builder._live_op_lock = real_live_op_lock + closer = threading.Thread(target=builder.close) + closer.start() + closer.join(10) + builder._live_op_lock = gated + return result + + def gated(_lock=real_live_op_lock): + return LockProxy(_lock()) + + builder._live_op_lock = gated + try: + try: + builder.with_archive(archive) + except Error: + pass + finally: + builder._live_op_lock = real_live_op_lock + ManagedResource._free_native_ptr = real_free + + with self.subTest(injection_point=k): + self.assertFalse( + builder._released + and builder._lifecycle_state == LifecycleState.ACTIVE, + "resource resurrected to ACTIVE after its close()") + self.assertEqual( + len(freed), len(set(freed)), + f"a pointer was freed twice: {freed}") + builder.close() + self.assertIsNone( + builder._handle, + "a handle survived every close(): it leaks") + + if not injected: + # k exceeded the number of lock releases in the + # operation: the sweep is complete. + self.assertGreater(k, 2, "sweep never covered the " + "historical bug's window") + break + k += 1 + + def test_second_mutating_call_is_rejected(self): + builder = Builder(self._MANIFEST) + + inside = threading.Event() + release = threading.Event() + + class BlockingSink(io.BytesIO): + def write(self, data): + inside.set() + release.wait(10) + return super().write(data) + + def seek(self, *args): + inside.set() + release.wait(10) + return super().seek(*args) + + worker = threading.Thread( + target=lambda: builder.to_archive(BlockingSink()), daemon=True) + worker.start() + try: + self.assertTrue( + inside.wait(10), "to_archive never reached its callback") + + with self.assertRaises(Error) as second_mut: + builder.to_archive(io.BytesIO()) + self.assertIn("mutating operation", str(second_mut.exception)) + + # A _lock-path native call is refused too: the in-flight + # mutating call holds `&mut` on the same native object. + with self.assertRaises(Error) as read_call: + builder.add_action('{"action": "c2pa.color_adjustments"}') + self.assertIn("mutating operation", str(read_call.exception)) + finally: + release.set() + worker.join(10) + + self.assertFalse(worker.is_alive(), "first to_archive hung") + # Both refused calls work once the mutating call has returned. + builder.to_archive(io.BytesIO()) + builder.add_action('{"action": "c2pa.color_adjustments"}') + builder.close() + + def test_read_during_mutation_is_rejected(self): + with open(os.path.join(FIXTURES_FOLDER, "C.jpg"), "rb") as f: + image = f.read() + reader = Reader("image/jpeg", io.BytesIO(image)) + manifest = reader.get_active_manifest() + uri = (manifest or {}).get("thumbnail", {}).get("identifier") + self.assertTrue(uri, "fixture must carry a thumbnail resource") + + inside = threading.Event() + release = threading.Event() + + class BlockingSink(io.BytesIO): + def write(self, data): + inside.set() + release.wait(10) + return super().write(data) + + def seek(self, *args): + inside.set() + release.wait(10) + return super().seek(*args) + + worker = threading.Thread( + target=lambda: reader.resource_to_stream(uri, BlockingSink()), + daemon=True) + worker.start() + try: + self.assertTrue( + inside.wait(10), + "resource_to_stream never reached its callback") + + with self.assertRaises(Error) as raised: + reader.detailed_json() + self.assertIn("mutating operation", str(raised.exception)) + finally: + release.set() + worker.join(10) + + self.assertFalse(worker.is_alive(), "resource_to_stream hung") + # Works again once the mutating call has returned. + self.assertTrue(reader.detailed_json()) + reader.close() + if __name__ == '__main__': unittest.main()