Pure Pascal Firebird wire protocol client: connect without fbclient

[mariuz]

Adds a pure Pascal implementation of the Firebird remote (wire) protocol, so
fbintf can reach a Firebird 3.0, 4.0, 5.0 or 6.0 server over TCP with no
fbclient library installed. It is a third provider alongside the existing
two, so code written against the fbintf interfaces runs unchanged.

Offered for consideration rather than as a finished proposal: it is a large
addition and it touches shared build files, so if the direction is welcome
but the packaging is not, I am happy to rework it. Everything new lives under
client/wire; the existing providers are untouched.

uses IB, FBWireClientAPI;

API := WireFirebirdAPI;      {instead of IB.FirebirdAPI - loads no library}
DPB := API.AllocateDPB;
DPB.Add(isc_dpb_user_name).AsString := 'SYSDBA';
DPB.Add(isc_dpb_password).AsString := 'masterkey';
Attachment := API.OpenDatabase('localhost:employee',DPB);

Everything below that line is the ordinary object model: IAttachment,
ITransaction, IStatement, IResultSet, IBlob.

Why

client/2.5 and client/3.0 both dynamically load fbclient. That is the
right answer on a developer machine and an awkward one for containers,
single file deployments, cross compiled targets, and any machine whose
installed client library is older than the server it has to reach. This
provider implements the protocol those libraries speak, so the dependency
disappears: the compiled program is the whole client.

What works

• handshake with protocol negotiation, versions 13 to 17
• Srp256 and Srp authentication, in both the op_cond_accept flow
  (authentication completes before the attach) and the op_accept_data
  flow (the proof travels in the DPB)
• wire encryption: ChaCha64, ChaCha, Arc4
• attach, create and drop database, detach
• transactions: start, commit, rollback, the retaining variants, 2PC prepare
• DSQL: allocate, prepare with full describe, execute, op_execute2 for
  singleton results, fetch, close, drop, set cursor name
• every data type, including INT128, DECFLOAT(16), DECFLOAT(34),
  BOOLEAN and the time zone types
• blobs: create, open, segmented read and write, close, cancel
• information calls for database, transaction, statement and blob

The password is used for the SRP exchange and stripped from the DPB before
the attach, so it never reaches the network in any form.

What does not, yet

Each raises ibxeNotSupported rather than failing obscurely:

Feature	What it needs
Events	the auxiliary connection from op_connect_request and a listener thread
Services	the packet exchanges exist in FBWireProtocol; the IServiceManager wrapper does not
Arrays	op_get_slice / op_put_slice with SDL descriptions
Batches	the protocol 16 op_batch_* family
Scrollable cursors	op_fetch_scroll, protocol 18
Multi database transactions	a two phase commit coordinator

FPC only for now. The transport is written against the FPC socket units;
everything above it is compiler neutral, so a Delphi transport is the only
missing piece. The Delphi packages are deliberately left untouched.

Layout

Thirteen units under client/wire, in three layers:

Layer	Units
Transport	FBWireStream (buffered TCP, per direction ciphers, XDR), FBWireCrypto, FBWireSRP, FBWireBigInt, FBWireConst
Protocol	FBWireProtocol (handshake, one method per packet), FBWireMessage, FBWireDescribe
Provider	FBWireClientAPI, FBWireAttachment, FBWireTransaction, FBWireStatement, FBWireBlob

FBWireProtocol is usable on its own if you want the protocol without the
fbintf object model.

The provider reuses the existing machinery rather than duplicating it:
FBParamBlock already builds exactly the clumplet buffers the protocol
carries, FBOutputBlock already parses the information responses, and
TSQLDataItem supplies every data conversion — the provider only points it
at the right offset in the message buffer.

Testing

testsuite/WireTest.pas is self contained and needs no fbclient:

fpc -Fuclient -Fuclient/2.5 -Fuclient/3.0 -Fuclient/3.0/firebird \
    -Fuclient/wire -Ficlient/include -FEbuild testsuite/WireTest.pas
./build/WireTest localhost:employee SYSDBA masterkey localhost:/tmp/scratch.fdb

Four offline layers that need no server — big integer arithmetic including a
1024 bit modular exponentiation of the size SRP performs, SHA-1, SHA-256,
RC4 and ChaCha20 against the published FIPS and RFC 8439 vectors, a complete
SRP exchange against a fixed reference exchange, and the message layout and
generated BLR — then the live ones: the handshake, what version is
negotiated when the offer is capped at 14/15/16/17, queries and parameters,
every data type, DML, null handling, accented text, blob round trips,
rollback, and create and drop database.

Every supported server version has been run, all with 0 failures:

Server	WireCrypt	Negotiated	Encryption	Result
6.0 (CI)	Enabled, Required	17	ChaCha64	81 tests
6.0.0 (local, LI-T6.0.0.2076)	Required	17	ChaCha64	81 tests
5.0 (CI)	Enabled, Required	17	ChaCha64	81 tests
5.0.4 (local container)	Enabled, Required	17	ChaCha64	81 tests
4.0 (CI)	Enabled, Required	17	ChaCha64	81 tests
3.0 (CI)	Enabled	15	Arc4	81 tests
no server	—	—	—	36 tests, live sections skipped

Firebird 3 settles on protocol 15 with Arc4 because that is the newest
protocol it knows and it predates the ChaCha plugins. Everything else in
the suite behaves identically there.

Capping the offer, against a modern server: 14 → Arc4, 15 → Arc4,
16 → ChaCha64, 17 → ChaCha64. Protocol 13 is refused with
isc_miss_wirecrypt by a server that requires encryption, which is correct:
it predates wire encryption. It is still offered for Firebird 3 and for
servers set to WireCrypt = Enabled.

CI

.github/workflows/wire-protocol.yml, modelled on the node-firebird matrix.

The server job runs Firebird 3, 4, 5 and 6 containers against both
WireCrypt = Enabled and WireCrypt = Required — seven combinations.
Firebird 3 with Required is excluded because protocol 13 cannot satisfy it
by definition. Each job starts the container, creates a database with isql
inside it, builds with the distribution's FPC and runs the test over TCP.
Nothing Firebird related is installed on the runner, which is the whole
point of the client. The negotiated protocol and the totals go to the step
summary, so the matrix view shows what each version settled on.

The offline job builds and runs the same binary on Linux and Windows
with no server at all, so an arithmetic or cryptography regression cannot be
masked by a server problem and the code keeps compiling on Windows.

All nine jobs are green on this branch.

The matrix earned its keep on the first run. It found two things: the
negotiation test wrongly demanded that capping the offer at protocol N make
the server negotiate exactly N, which is false for a server that does not
know N (Firebird 3 tops out at 15); and, from a compiler warning the matrix
surfaced, that the character set of a text column was never being read.
if SQLType in [SQL_TEXT,SQL_VARYING] compares against a Pascal set, which
holds 0..255, while the type codes are 452 and 448 — so the test matched
truncated values, never fired, and every text column fell back to character
set NONE. That was masked locally because the sample database is character
set NONE. Both are fixed, and a UTF8 column now correctly reports character
set 4.

Roadmap

doc/WireProtocol.md carries a twelve point roadmap saying what each missing
piece needs and what already exists to support it. The order it proposes:

#	Milestone	Needs
1	Run the existing test suite against this provider	a provider switch in TTestApplication
2	Events	op_connect_request, a second socket, a listener thread
3	Services	the IServiceManager wrapper over exchanges that already exist
4	A Delphi transport	TFBWireTransport over Winsock and Posix sockets
5	Array columns	op_get_slice, op_put_slice, SDL generation
6	Statement timeouts and cancellation	op_cancel, the P16 timeout field
7	Scrollable cursors	op_fetch_scroll
8	The batch API	the op_batch_* family
9	Inline blobs	op_inline_blob
10	Firebird 6 protocol 20	schema search path, named arguments
11	Wire compression	zlib either side of the cipher
12	Engine message text	a firebird.msg reader or a generated table

The first is deliberately first: testsuite/ already contains twenty two
programs written against the interfaces rather than against a provider, and
they exercise far more of the API than WireTest does. A switch in
TTestApplication.GetFirebirdAPI would run all of them over the wire, and
that run is then the acceptance criterion for the events, services and array
milestones. I did not add the switch here because it changes shared test
code and seemed better raised than assumed.

Deliberately not planned, with reasons in the document: Legacy_Auth (it
produces no session key, so it cannot satisfy a server that requires
encryption), Win_SSPI, and protocols below 13.

Protocol details worth recording

These are the places where a plausible reading of the packet layouts
produces a client that does not work. Each cost a debugging cycle and each
is documented in doc/WireProtocol.md and commented where it is
implemented.

• The op_crypt key type is the type the server advertised, in practice
  Symmetric — not the authentication plugin name. Sending the plugin name
  gets the connection closed with "unknown key".
• Text columns must be described with blr_text2 / blr_varying2 naming
  the column's character set. With plain blr_text / blr_varying the
  engine assumes the connection character set and reinterprets the byte
  length as length div max bytes per character: a VARCHAR(37) read over a
  UTF8 connection silently becomes 9 characters and the fetch fails with a
  string truncation error.
• XDR sends a boolean as one value byte padded to four, value first. As a
  big endian integer the value lands in the last byte and every boolean reads
  back false.
• A batched op_fetch must be drained completely before anything else is
  sent; the server streams one op_fetch_response per row and ends the batch
  with a message count of zero. Rows are cached as they arrive.
• ISC_QUAD puts the high word first, which is not the memory layout of
  an Int64 on a little endian machine.
• SRP deviates from RFC 5054 in the engine: the exponent is reduced mod
  N, H(N) xor H(g) is a modular exponentiation, the salt is hashed as its
  hexadecimal text, and the session key is always SHA-1 even for Srp256.
• arch_generic must be sent in the version entries: matching the
  server's own architecture makes it transmit messages as raw memory images
  instead of XDR.

Bugs found and fixed while bringing the provider up against a live server,
all of which are now covered by tests: parameters were all being sent as
null (the encoder reads the buffer's null indicator, so IsNull must read it
directly rather than gate on the column's nullable flag); a VARCHAR value
was overwriting its own two byte length prefix; CHAR values were not blank
padded, so a parameter never compared equal to a padded column; closing a
cursor whose transaction had ended was treated as an error when the server
has already closed it.

One behaviour worth flagging

An attachment that has read a table keeps an interest in it that outlives
the transaction, and Firebird 5 then refuses drop table on that same
attachment with isc_no_meta_update. This is not a property of this
client: the stock fbclient provider fails in the same place with the same
code, and a second attachment issuing the drop blocks rather than
succeeding. WireTest therefore treats dropping its test table as best
effort and drops it at the start of the next run.

Error messages

firebird.msg is a client library resource, so it is not available. The
status vector is decoded and rebuilt into a standard ISC vector, so
EIBInterBaseError, GetIBErrorCode and CheckStatusVector behave as
usual, and the interpreted text the server supplies is used where it is
present. Where the stock provider says "unsuccessful metadata update" this
one says Firebird Error Code: 335544351. Closing that gap needs either a
firebird.msg reader or a generated table of the common codes.

Documentation

• doc/WireProtocol.md — design, protocol details, SRP, encryption, message
  encoding, error reporting, testing and CI
• client/wire/README.md — status and the measured results
• README.md — a short section pointing at both

Commits

1. the crypto, XDR and protocol engine
2. the fbintf provider on top of it
3. configurable protocol version capping, and the test for it
4. the CI matrix and the documentation
5. the character set and negotiation fixes the matrix found
6. the CI verified server matrix recorded in the documentation
7. the roadmap

Notes for review

The reference throughout was the Firebird server's own C++ implementation:
src/remote/protocol.h, protocol.cpp, interface.cpp and
src/auth/SecureRemotePassword. Nothing here is derived from another
client; the SRP test vectors were computed independently and cross checked
against a client/server pair rather than taken from an existing
implementation.

Worth a careful look: the message buffer layout in FBWireMessage
(alignment rules and the null bitmap), the lifetime handling in
FBWireStatement where the base class prepares lazily, and the encryption
changeover in TFBWireConnection.StartWireEncryption, which is asymmetric —
the receive cipher is installed before the op_crypt response is read
because the server encrypts from the moment it receives the request.

Changes outside client/wire, all additive:

• fbintf.lpk and Makefile.fpc — client/wire added to the unit search
  path, and the thirteen units added to the package file list
• fbintf.pas — the units added to the uses clause
• README.md — a short section pointing at the new documentation
• testsuite/WireTest.pas and .github/workflows/wire-protocol.yml — new
  files

The Delphi packages (fbintf.dpk, fbintf.dproj) are deliberately
untouched, because the transport is FPC only for now. If you would rather
the wire units were not compiled into the default package at all, dropping
them from fbintf.pas and fbintf.lpk leaves them available on the search
path for anyone who wants them and changes nothing for anyone who does not.

[mariuz]

Update: the roadmap has been implemented — this branch now carries the full feature set

Since this PR was opened, the twelve milestones from the roadmap in doc/WireProtocol.md have been implemented and merged into this branch. Each milestone was developed as its own stacked PR in the fork, design document first, implementation after, so every feature has a written plan, a record of what the implementation actually found, and its own reviewable diff:

#	Milestone	Fork PR
1	Full fbintf test suite over the wire (testsuite -a wire)	mariuz#2
2	Events (auxiliary connection + listener thread)	mariuz#3
3	Services API (IServiceManager)	mariuz#4
4	Delphi transport (written, unverified — see below)	mariuz#5
5	Array columns (op_get_slice/op_put_slice, SDL)	mariuz#6
6	Statement timeouts and cancellation (op_cancel)	mariuz#7
7	Scrollable cursors (protocol 18)	mariuz#8
8	Batch API (protocol 16 op_batch_*)	mariuz#9
9	Inline blobs (protocol 19)	mariuz#10
10	Firebird 6 / protocol 20 (schemas, search path)	mariuz#11
11	zlib wire compression (WireCompression=true)	mariuz#12
12	Engine error message text without firebird.msg	mariuz#13

What that means for the provider

The gaps listed under "not implemented yet" in the original description are closed. The provider now negotiates protocols 13 through 20 and implements events, services, arrays, statement timeouts and cancellation, scrollable cursors, the batch API, inline blobs, schema support on Firebird 6, optional zlib wire compression beneath the wire encryption, and fb_interpret-style error message text from a generated message table (1682 messages) — so status vectors read the same as with fbclient, firebird.msg not required.

Testing

• The entire existing fbintf test suite (all 22 programs) runs over the wire provider via testsuite -a wire and is compared against a committed reference log in CI. Getting there found and fixed real wire bugs (among them an SRP breaker from AnsiUpperCase on trailing #0, swapped execute2 handles, and a missing DECFLOAT DPD codec), which is the strongest compatibility evidence this provider has: the same programs, the same output, fbclient absent.
• The provider's own WireTest grew from 81 to 178 checks.
• The CI matrix runs Firebird 3, 4, 5 and 6 × WireCrypt Enabled/Required, plus a dedicated wire-compression row, plus offline builds on Linux and Windows. All green on the current head.

Documentation

• doc/WireProtocol.md — architecture, the handshake and per-protocol-version packet field differences (13→20) as implemented, and the roadmap with each milestone's outcome.
• doc/roadmap/*.md — one design document per milestone, updated after implementation with what was actually found (these record the protocol subtleties: batch clumplet widths, inline blob cache semantics, compression stream placement relative to the cipher, and so on).
• client/wire/README.md — user-facing summary and the tested-server matrix.

Two things reviewers should know

1. The Delphi transport (milestone 4) is written but unverified. TFBWireTransport has Delphi branches (Winapi.WinSock2 on Windows, the Posix.* units elsewhere) and the wire units are listed in fbintf.dpk/fbintf.dproj, but no Delphi toolchain has compiled them — CI is FPC-only. The docs say so explicitly; expect a first-compile pass. FPC behaviour is untouched.
2. Milestone 6 adds small public interface members across all providers: IAttachment.CancelOperation and IStatement.SetStatementTimeout/GetStatementTimeout, implemented for the 2.5, 3.0 and wire providers. That and a few virtuals in FBClientAPI/FBAttachment are the only touches to shared code beyond the build file listings; everything else lives under client/wire.

As before: offered for consideration, and if the direction is welcome but any of the packaging or the shared-interface additions are not, I am happy to rework them.