Part 2 of 14
Serialization and framing
A sample sitting in memory has a layout chosen by a compiler. The program reading it may have been built by a different compiler, for a different processor, in a different language. So the sample is written out in a form that both sides agree on, byte by byte, with the rules fixed by the specification rather than by either compiler.
Two things make this more than a mechanical translation. The reader is parsing bytes that arrived from the network, so every length it reads is a number chosen by somebody else. And the writer's version of the type may not be the reader's version, so fields can arrive reordered, missing, or unknown.
Layout
Alignment and padding
any value is written to or read from an encoded body
Every field of every sample, in both directions.
Each value is written at an offset that is a multiple of its size: a two-byte value at an even offset, an eight-byte value at a multiple of eight. Where the next value would land at the wrong offset, padding bytes are inserted until it does not. Both sides compute the same padding from the same rule, so the padding itself is never described on the wire.
The offsets are counted from the start of the encoded body, not from the start of the message. That is what lets a body be moved without re-encoding it.
Where the two versions differ
The first aligns every value to its own size, up to eight bytes. The second caps alignment at four, so an eight-byte value needs at most four bytes of padding rather than seven. The cap costs nothing in correctness and makes encoded bodies smaller.
Byte order is not fixed. Each encoded body carries a small header saying which order it used, and the reader follows it rather than converting. One value is treated differently: the 16-byte floating-point type is held in one canonical order internally and reversed only when the stream is the other order, so both stream orders read back the identical bit pattern.
Scalars
Strings of wide characters
a sample contains a wide-string field
Once per such field per sample.
A wide string is written as a count followed by its characters. The count is in bytes, not characters, and each character is two bytes. There is no byte-order mark and no terminating zero.
An odd byte count
If the byte count is odd, it cannot be a whole number of two-byte characters, so the value is malformed no matter what follows it. The check is worth making explicitly rather than letting the division round: an odd count that is halved silently produces a string that ends half a character early, and the error surfaces somewhere else entirely.
Read-Wide-String(r)
r the reader — the input bytes, a cursor into them, and the configured length limit
- B = Read-32(r) // a byte count, chosen by the sender
- if B is odd
- return error
- if B > r.limit or B > Remaining(r)
- return error // refuse before reserving anything
- N = B / 2
- V = Borrow(r, B) // a view into the input, not a copy
- return View(V, N)
Line 7 hands back a view over the input bytes rather than a copy. Walking that view decodes two bytes at a time in the stream's byte order, and nothing is allocated. A caller that wants to keep the string past the lifetime of the input asks for an owned copy, and that one allocation is the only one on the path.
Cost
Fixed-size scalars are O(1). A wide string is O(N) in its characters. Reading a view allocates nothing; taking ownership costs one allocation and one copy.
Collections
Sequences and arrays
a sample contains a collection field
Once per collection per sample. The count check is the first thing an untrusted message meets.
A sequence carries its length on the wire because it varies. A fixed array does not, because both sides already know how many elements there are. The difference matters more than it first appears.
A sequence's length is a number sent by someone else. Reserving storage for it before checking it means a four-byte field decides how much memory this process allocates. A sender can claim four billion elements in a message that is twelve bytes long.
So the declared type carries a maximum, the storage for that maximum lives inside the value itself, and the count is checked against it before a single element is constructed.
Read-Sequence(r, C)
r the reader · C the maximum element count the declared type allows
- n = Read-32(r)
- if n > C
- return error // before constructing any element
- S.length = 0
- for i = 1 to n
- v = Read-Element(r)
- if v == error
- return error // S is discarded, not returned partly filled
- S.slot[i] = v
- S.length = i
- return S
An array, where the risk is the other way round
An array has no count to check, so the risk is a half-built array escaping when an element fails to decode. Every position is filled with the type's default first, then replaced in order, and the array is returned only once every replacement has succeeded.
Read-Array(r, N)
r the reader · N the element count fixed by the declared type
- for i = 1 to N
- A[i] = Default()
- for i = 1 to N
- v = Read-Element(r)
- if v == error
- return error // A never leaves this procedure
- A[i] = v
- return A
Cost
O(K) element operations for K present elements, or O(N) for an array. Neither allocates: the storage is part of the value and was sized when the type was declared. Elements that own storage themselves, such as strings, still allocate their own.
Structures
Reading a structure by member number
a sample of a type declared to allow added or reordered members is read
Once per sample of such a type. Types not declared that way never take this path.
Two programs in the same system are not always built from the same version of a type. One may have gained a field. A structure can be declared to allow this: its members are written each with a small header giving the member's number and its length, so a reader can find the members it knows and step over the ones it does not.
A member the reader does not know
Stepping over an unknown member is not always safe. A field added to say that the coordinates are now relative rather than absolute cannot be ignored by a reader that does not know it — ignoring it means reading the numbers wrongly. So the writer marks each member with whether it may be skipped. A reader that meets an unknown member marked must-understand rejects the whole sample rather than guessing.
A member the reader declares and the writer never sent
That is not an error: an omitted member takes the type's default value.
Read-Structure(r, scope, F)
r the reader · scope the byte extent this structure occupies · F the fields the reader's version of the type declares
- for each field f in F
- slot[f] = nil
- while Remaining(scope) > 0
- (id, len, must-understand) = Read-Member-Header(r)
- f = Find-Field(F, id)
- if f ≠ nil
- slot[f] = Read-Element(r, len)
- Seek(r, member end) // trust the declared length, not the decode
- else if must-understand
- return error // the writer said this cannot be ignored
- else
- Skip(r, len)
- for each field f in F
- if slot[f] == nil and f is not optional
- slot[f] = Default(f)
- return Build(slot)
Worked example
A writer built from version 2 of a type sends three members. A reader still on version 1 declares three fields, two of which the writer sent.
| Member | In the reader's version | What happens |
|---|---|---|
| 2 · velocity | declared | decoded into the velocity field, out of order but found by number |
| 1 · position | declared | decoded into the position field |
| 9 · battery, skippable | unknown | stepped over using its declared length |
| 3 · timestamp | declared, absent | never sent; takes the type's default |
The reader gets a usable sample from a writer a version ahead of it, without either side knowing about the other's schema.
Now suppose version 2 also changed position from absolute to relative and signalled it with member 7, marked must-understand. The version 1 reader has no field for member 7, so it rejects the whole sample rather than returning a position it would interpret as absolute. Losing the sample is the correct outcome: the alternative is a robot acting on coordinates that mean something else.
Line 8 is what makes the skipping reliable. After decoding a known member the cursor moves to where the member's declared length said it ends, rather than to wherever decoding happened to stop. If the writer's version of that member is longer than the reader's, the extra bytes are left behind and the next header is still found.
Line 16 builds the value once, at the end. Until then the decoded fields sit in slots that belong to nothing, so a rejection at line 10 cannot leave a half-populated structure anywhere a caller could see it.
Knowing a member's length before writing it
Each member's header states its length, which means the length has to be known before the member is written. Rather than serializing into a scratch buffer to measure it, the member is run twice: once through a sink that only counts bytes, then again into the real output. Both runs take the same value and make the same decision about whether an optional member is present, so the count matches what is written.
Cost
Finding the field for a member number is a scan of the declared fields, so decoding M members against F fields is O(M · F). Both are small and known when the type is declared, and the scan is generated rather than searched through a table built at run time — which is why no lookup structure is allocated per sample.
Framing
Destination and timestamp records
a message containing one of these records is built or read
A few times per message.
One network message carries several records. Some of them hold data; others say something about the records that follow. Two of those matter here: one names the participant the following records are meant for, and one gives the time the following samples were written.
Both are small and both have exactly one job, which makes their decoders short. The interesting part is what they refuse. The destination record must contain a 12-byte identifier and nothing else — a record declaring any other length is rejected rather than padded or truncated, and it is rejected before the destination is changed.
A record with an empty body
The timestamp record has two forms: one carrying a time, and one saying that there is no longer a time in force. The second has an empty body, and that is where an easy mistake lives. A record with an empty body still occupies its header. A reader that treats a length of zero as meaning "no record here" and moves on by nothing loops; one that assumes every record has a body swallows the record that follows it.
Cost
Both records encode and decode in constant time with no allocation: the destination is a fixed 12-byte copy, the timestamp is at most two scalar loads. Walking a message parses one header per record and keeps only a borrowed slice of what remains.
Framing
Applying context in order
any RTPS message is received
Once per received message, before any of its records reach an endpoint.
Because those two records change how later records are read, the reader has to carry their values forward as it walks the message. The rule is that just before reading record i, the destination and timestamp in force are the most recent ones named among records 1 through i − 1.
The destination check has to happen early. A message can be addressed to a participant that is not this one, and the records inside it name endpoints. Looking up an endpoint, feeding a reliability state machine, or adding a fragment to a partly assembled sample are all changes to local state. So a record whose destination is not this participant is dropped before any of that, not after.
Dispatch-Message(p, msg)
p the local participant, whose identifier the destination is compared against · msg one received message and its records in wire order
- dest = all // no destination record seen yet
- time = nil
- for each record s in msg // wire order; do not stop at the first data record
- if s.kind == destination
- dest = s.prefix
- else if s.kind == timestamp
- if s.invalidating
- time = nil
- else
- time = s.value
- else if s.kind == padding
- continue // changes neither
- else
- if dest ≠ all and dest ≠ p.prefix
- continue // dropped before any local state is touched
- e = Resolve-Endpoint(msg.source, s.entity-id)
- Deliver(e, s, time)
The comment on line 3 is a real trap. It is tempting to return once a sample has been delivered, since that looks like the message's purpose. But records after it may carry their own destination and timestamp changes and their own samples, and stopping early discards them.
The timestamp arrives as a seconds field and a fractional field, where the fraction counts in units of 1/232 of a second. It is converted to whole nanoseconds with integer arithmetic:
To-Nanoseconds(seconds, fraction)
the two fields of a wire timestamp; fraction counts in units of 1/232 of a second
- return seconds · 109 + ⌊fraction · 109 / 232⌋
A sample split across records
For a sample split across several records, the timestamp is part of the metadata the pieces have to agree on. If a later piece arrives under a different timestamp, the assembly is rejected — and rejected before anything is written into the partly assembled sample, so a disagreement cannot leave a half-updated one behind.
Cost
O(k) for k records, with constant additional storage: the context is one optional identifier and one optional time. Each decision is a fixed-size comparison and at most a scalar copy. Nothing on this path allocates or copies a payload.
Next: reliability and fragmentation
Networks drop packets and refuse large messages. Every sample still has to arrive, complete and once.
These descriptions follow the implementations in Tin DDS and Retina. Evaluation access and the full algorithm notes are available from engineering@tinverse.com.