RFC 6330 RaptorQ FEC scheme (not 3GPP-mandated, offered as reference work) - #64
Conversation
7bbb1e1 to
77b7f86
Compare
77b7f86 to
61ae0ae
Compare
5e2e338 to
f43eca6
Compare
f43eca6 to
aa36087
Compare
61ae0ae to
9ae681a
Compare
aa36087 to
2007684
Compare
9ae681a to
3d25c37
Compare
6e55625 to
f9ad94d
Compare
470df62 to
db452b7
Compare
f9ad94d to
bf6b1a9
Compare
db452b7 to
5252e03
Compare
bf6b1a9 to
ee01d44
Compare
5252e03 to
d25ff1d
Compare
ee01d44 to
929741f
Compare
d25ff1d to
59d12c7
Compare
929741f to
3d5abc0
Compare
e8adab8 to
16deddc
Compare
3d5abc0 to
6c63c90
Compare
16deddc to
af63605
Compare
6c63c90 to
3cf4c92
Compare
This pull request: merge it third, after #61Status: ready. 3 commits, 22 files, +1826/-85. 114 cases pass. What to do
Why it is separate from #61 at all. Raptor is required and RaptorQ is not, so they are reviewed and merged on different terms. Merging #61 without #64 leaves the library fully conformant; the reverse is not true. How this set of pull requests came to look like this
Where the nine closed pull requests went
One correction, @rjb1000. #93 is listed above as "already merged in PR #60". #60 merged the FDT growth fix on 17 August and changed no test file; Every open issue, and which pull request closes it
Four open issues are deliberately closed by nothing:
The stack, and the order to merge it inGitHub retargets each child to One trap for whoever merges. The heads live in two repositories: #61, #68 and #98 in |
|
Merge third, after #61, whose branch is its base. The only one not based directly on #62. Closes no issue, deliberately: RaptorQ is mandated by no profile here, and it refuses RaptorQ under the MBMS Download Profile naming the clause. If |
3cf4c92 to
aeeb923
Compare
af63605 to
7a02968
Compare
aeeb923 to
5ca2199
Compare
7a02968 to
59f5ff2
Compare
…ecific OTI layout Change type New feature, not 3GPP. This branch is the RaptorQ delta on top of the Raptor branch and nothing else. Basis RaptorQ is outside the set of FEC schemes 3GPP references, which is why it lives here and not on the compliant branches. TS 26.346 V18.2.0 clause L.4.7: "Regarding Application Layer FEC support, the two FEC schemes referenced in this specification, the Compact No-Code FEC scheme as specified in RFC 3695 [13], and the Raptor FEC scheme as specified in RFC 5053 [91] are optional to implement by the BM-SC and mandatory to support by the UE." RaptorQ and RFC 6330 appear nowhere in TS 26.346: zero occurrences. Not forbidden, just outside the referenced set. RFC 6330: "This document describes a Fully-Specified Forward Error Correction" scheme "corresponding to FEC Encoding ID 6", so its Codepoint value is 6 under the identity mapping the rest of this library uses. Corrected while reinstating it The previous version of this branch was an unexplained revert of the removal commit, and it brought back the three invented FDT attribute names the Raptor branch had already replaced with the schema-defined FEC-OTI-Scheme-Specific-Info. Reinstating it that way would have undone a conformance fix. Worse, it shared one encoder between the two schemes, which is wrong: the same four octets carry DIFFERENT field widths, with Z and N the opposite way round, so encoding one scheme with the other's layout silently corrupts both values. RFC 5053 clause 3.2.3, Raptor: "a 4-octet field consisting of the parameters Z (2 octets), N (1 octet), and Al (1 octet)" RFC 6330 clause 3.3.3, RaptorQ, the Scheme-Specific parameter list: "The number of source blocks (Z): 8-bit unsigned integer." "The number of sub-blocks (N): 16-bit unsigned integer." Change Reinstates RaptorQCodec, RaptorQMath, RaptorQTables and GF256LinearSystem with their headers and test binary, and the FecScheme::RaptorQ dispatch branches. Replaces the single encoder with one that selects the octet layout from the scheme, on both the encode and decode sides and at both the FDT-Instance and File levels, keeping the schema-defined attribute the Raptor branch introduced. Rebuilt on the Raptor branch rather than on the old lineage, so it is exactly one delta above it and the two non-3GPP features, this and FLUTE version 2, are now separate from each other as well as from the compliant core. Verification T1: builds clean, ctest at the build root 35/35, the 25 from the Raptor branch plus the RaptorQ suite and 2 new cases. The layout difference is asserted rather than assumed: identical inputs Z=1, N=0x0203, Al=4 encode to AQIDBA== under RaptorQ and AAEDBA== under Raptor, and each is checked absent from the other's output. A round trip with N=0x0140, a value a shared one-octet layout would truncate, recovers all three parameters. Quotations checked with tools/verify-citations.py in --strict mode. Not in this change No claim that RaptorQ is 3GPP conformant, because it is not in the referenced set at this baseline. The Raptor paths below are untouched. Carries its own repair-symbol coverage rather than leaving it to a follow-up, for the reason given on the Raptor commit this builds on: a fountain code that transmits no repair symbol protects nothing, and the object-level tests that pull every symbol in one call cannot see it. RaptorQRepairTransmissionTest drives one datagram at a time and acknowledges each, and two of its four cases fail if the completion condition is reverted, emitting 79 symbols instead of 90 and no repair symbol instead of 11. T2 for this scheme: an object recovered byte-identically over real multicast at a 7.6% datagram loss rate, where Raptor with its smaller default budget does not.
Problem The example transmitter drove one FDT schema, one FEC scheme and a fixed redundancy level, so the profiled-schema, Raptor and RaptorQ paths could not be exercised against a real receiver over a real socket without editing and rebuilding it. Code-derived, no spec claim. Raised by looking for a way to drive the FEC and FDT schema paths end to end Change Adds -S/--fdt-schema (draft2005, rfc3926, profiled), -F/--fec (compact, raptor, raptorq) and -L/--fec-redundancy-level, each mapping its argument onto the enumeration or value the Transmitter constructor already accepts. Defaults reproduce the previous behaviour, so an existing command line is unaffected. Verification T2: used to capture the FDT off the wire per schema, and to run Raptor and RaptorQ end to end over multicast including under datagram loss. The RaptorQ run's FDT carries FEC-OTI-FEC-Encoding-ID="6" and the RaptorQ scheme-specific layout, captured by a sniffer that links no library code. Not in this change Any library behaviour. This only exposes choices the library already offered.
Problem A Transmitter under either 3GPP profile would send a session using a FEC scheme no conformant receiver is obliged to decode. This branch adds RaptorQ, which neither TS 26.346 nor TS 26.517 references. [source-derived] Basis TS 26.346 V18.2.0 clause L.4.7: "the two FEC schemes referenced in this specification, the Compact No-Code FEC scheme as specified in RFC 3695 [13], and the Raptor FEC scheme as specified in RFC 5053 [91] are optional to implement by the BM-SC and mandatory to support by the UE." The set is closed and TS 26.517 does not widen it: its clause 6.2.4.5 gives the repair byte-range determination for FEC Encoding ID 0 and FEC Encoding ID 1 and for no other value. Raised by reading the authority during this work Change is_3gpp_admissible_fec_scheme() states the admissible set, and the Transmitter constructor refuses anything outside it under a 3GPP profile. Written as a match against the two schemes the clause names rather than as a rejection of RaptorQ by name: a scheme added to the enumeration later is then refused under a profile until someone decides otherwise, instead of being admitted by default. fec_scheme_from_encoding_id() gains FEC Encoding ID 6, without which this branch's own parser would refuse a received FDT declaring the scheme this branch implements. Refused at construction rather than at send: a session no receiver can decode is worse than a refusal at setup. RaptorQ remains available under Profile::Unprofiled, which is what this branch adds it for. Verification T1: 123 cases pass, 2 new (a scheme outside the admissible set refused under both 3GPP profiles, and the RaptorQ encoding identifier mapping to its scheme), 0 failures. 121 passed before. Not in this change Whether RaptorQ should be offered to 3GPP profiles at all. It is not, on the clause above, and this branch does not argue otherwise.
Problem The Raptor branch shortens the encoding symbol so an object too small to fill four symbols can still be encoded, because Raptor has no systematic index below K = 4. That shortening is deliberately Raptor-only, and the case proving RaptorQ is left alone could not live there: the enumerator does not exist on that branch. [code-derived] Basis unverified: could not obtain RFC 6330. The claim that RaptorQ imposes no equivalent floor, because it pads a source block up to the smallest supported K', is inherited from this branch's own codec implementation and its existing comments, not re-derived from the RFC during this work. The assertion itself is code-derived and needs no clause: it pins the observable behaviour, that the symbol length is unchanged for an object RaptorQ can encode as-is. If the inherited reading of RFC 6330 turns out to be wrong, this case is what will fail. Raised by reading the code during this work, splitting a case off the Raptor branch that could only be asserted here Change One case, asserting that a 600-byte object at a 1024-byte session symbol keeps that symbol length under RaptorQ, where the same object under Raptor is shortened. Verification T1: 136 cases pass, 1 new, 0 failures. 135 passed before. Not in this change Nothing about the Raptor path, which is unchanged and asserted on its own branch. No claim about what RFC 6330 requires; see Basis.
5ca2199 to
45ac74c
Compare
59f5ff2 to
8a3671b
Compare
Addresses part of #1.
Dependencies and issues
Depends on: #61, and is based on its branch; #61 in turn depends on #62. Merge order is #62, then
#61, then this one.
Paired with: #65, #68 and #98, independent of this one, either order.
Closes on merge: nothing on its own. Completes #1 together with #61. #1 needs closing by hand once both are in, since neither pull request delivers it alone.
What is in this pull request
Three commits, and the diff shows only those: this pull request is based on #61's branch, not on
development, so #62's and #61's work does not appear here.fec: reinstate RaptorQ as this branch's delta— RaptorQ alongside Raptor, sharing the same codecinterface, with its own scheme-specific OTI layout, which differs from Raptor's for the same values.
Its repair-symbol coverage is in this commit rather than a follow-up, for the reason given on RFC 5053 Raptor FEC scheme: codec, FLUTE integration, and the missing-symbol accessor #61.
examples: select the FDT schema, the FEC scheme and the redundancy level— adds-S,-Fand-Lto the example transmitter, which is how both schemes and the profiled schema were driven against a
real receiver.
Not mandated by 3GPP
TS 26.346 V18.2.0 clause L.4.7 names Compact No-Code and Raptor for the MBMS Download Profile. RaptorQ is
neither required nor forbidden there. It is not the default, and nothing in the profile path selects it.
Verification
T1: 114 cases passing. T2: an object received byte-identical over real multicast with
FEC-OTI-FEC-Encoding-ID="6", and recovered at a 7.6% datagram loss rate where Raptor's smaller defaultbudget does not.
Merge order
After #62, then #61.
Rebased onto #62. That branch gained three commits from a prose read of RFC 5651 and RFC 5775, so
this branch was rebased onto its new head. Nothing in this pull request's own commits changed; the case
count above rose from 110 to 114 because the base branch's own new cases run alongside them.
Issues closed by this pull request
Closes #103, #104.
Profile::Unprofiledonly.fec: admit only the FEC schemes the 3GPP profiles name. Fixed here rather than on FLUTE v1 (RFC 3926) and MBMS Download Profile conformance, plus assorted correctness fixes #62 because this is the branch that introduces a scheme outside the admissible set.