Tinverse logo Tinverse LLC

Part 12 of 14

Teleoperation

Driving a robot from a distance puts every earlier part under a deadline. A command has to be considered before it stops being useful, and no logical command may be accepted twice. Remote commands can cross two network paths, while a command from the same machine can arrive through shared memory. All three routes must lead to one admission decision.

The command data path uses storage prepared at startup. Its buffers and timing limits come from the product configuration so an ordinary command does not have to wait for new heap storage. Configuration loading and other control-plane work have separate allocation and timing costs.

Configuration

Admitting a resource profile

Runs when

an endpoint is built, before any command can be sent

How often

Once at startup. Nothing on the command path repeats it.

The profile states five storage capacities — for serializing, receiving, history, duplicate suppression, and safety events — along with the timing, how many independent network paths there are, and the largest sample allowed. Every one of them has to be positive, and there have to be at least two paths, or the profile is not a valid one.

Rather than checking those values wherever they are used, they are checked once when the profile is built, and a profile that fails cannot be constructed at all. Everything downstream then takes a profile that is already valid and does not re-check it.

A type whose size cannot be known

Creating an endpoint asks the sample type for its largest possible encoded size, and rejects three things in order: a type with no bound at all, one whose bound is larger than the configured sample capacity, and one whose bound is large enough to need splitting across messages.

The third rejection is the one specific to this profile. Splitting a sample is well-defined and Part 3 covers it, but it means a command arrives only when all its pieces do, and one lost piece costs the whole command. A control path declines that trade and requires commands that fit in one message.

Where the serialization buffers come from

Serializing a command needs somewhere to put the bytes. Allocating that per command is what the profile exists to avoid, so the factory allocates exactly as many blocks as the profile's serialization capacity, once, and hands one to each writer as it is created.

Create-Writer(factory)

factory holding the pool of pre-allocated serialization blocks

  1. b = Take-Free-Block(factory.pool)
  2. if b == nil
  3. return error // the pool is unchanged
  4. w = New-Writer(b) // the writer now owns the block
  5. if w == error
  6. Return-Block(factory.pool, b)
  7. return error
  8. return w

The count of blocks in the pool plus the count held by live writers is always the number the profile asked for. A block is never in both places, because taking it moves ownership rather than copying a reference, and dropping a writer moves it back. Line 6 covers the case where the pool gave out a block and endpoint creation then failed for its own reasons.

Cost

Building the profile and admitting a sample type are O(1) and allocate nothing. The factory allocates its S blocks once. Taking and returning a block scans the pool's S slots behind a lock, but that happens when an endpoint is created, not when a command is sent. Publishing is O(P) for a P-byte command into a block the writer already owns, with no lock and no allocation.

Receiving

Servicing UDP and shared memory

Runs when

the receive worker takes a turn, which it does continuously

How often

Every turn: one bounded UDP batch, then one shared-memory opportunity.

UDP datagrams arrive in bursts. Handling one per turn leaves the rest waiting; draining without a limit lets a busy network occupy the worker indefinitely. A turn therefore handles at most the configured number of UDP datagrams, then gives shared memory one nonblocking opportunity. Even a continuous network stream cannot prevent a notified local command from being considered.

Declaring a receive-buffer count is only a resource control if the service actually owns that many buffers and uses no others. Here it does: a turn fills slots from the first, stops at the first poll that finds nothing or when every slot is full, then hands each filled slot to the handler in order. After that finite batch, it checks shared memory once before beginning another turn.

two paths path A path B polled alternately R slots, allocated at startup filled filled filled empty empty poll found nothing — stop filling here handler, in slot order each slot cleared after its call returns A slot is available or pending, never both, so a turn can never admit more than R datagrams.
Figure 12.1 The UDP half of a worker turn owns exactly the slots the profile declared. One shared-memory check follows this bounded batch.

Receive-Turn(batch, R, shm, ingress)

batch the pre-allocated UDP slots · R the declared slot count · shm the secured local receive mechanism · ingress the command state shared by every transport

  1. filled = 0
  2. for i = 1 to R
  3. n = Poll-Next-Path(batch.slot[i]) // nonblocking, alternating
  4. if n == 0
  5. break // nothing waiting; do not spin the rest of the batch
  6. batch.slot[i].extent = n
  7. batch.slot[i].path = current path
  8. filled = i
  9. for i = 1 to filled
  10. r = Handle(batch.slot[i]) // borrows the bytes; keeps no reference
  11. batch.slot[i].extent = 0
  12. if r ≠ continue
  13. Clear-Remaining(batch, i + 1, filled)
  14. return r
  15. s = Try-Shared-Memory(shm) // one nonblocking opportunity
  16. if s contains a command
  17. r = Handle-With-Same-Ingress(ingress, s)
  18. if r ≠ continue
  19. return r
  20. return continue

Line 11 clears a slot after the handler returns rather than before it is called. The handler borrows the bytes for the duration of its call, so the slot cannot be refilled while it is running — which is what stops one datagram overwriting another that is still being examined.

Line 13 matters for shutdown. If a handler asks to stop, the slots after it still hold datagrams that will never be serviced, and leaving them marked as pending would make the accounting wrong for anything that inspects the batch afterwards.

Why both transports share one command state

A local shared-memory copy and a UDP copy can represent the same logical command. They therefore pass through the same reader history, authority state, sequence frontier, and duplicate window. If each transport kept its own state, accepting a command from one would not stop the other from accepting the same command again.

Shared memory still has its own authentication and decoding boundary. Only after that succeeds does it enter the common command policy. An idle check, duplicate, policy rejection, or security rejection leaves the worker running; a command-processing fault stops it through the same owned shutdown path as a UDP fault.

Cost

Startup allocates R × D bytes for R slots of D bytes, plus the configured shared-memory resources. A turn makes at most R nonblocking UDP receive attempts, at most R UDP handler calls, and one nonblocking shared-memory progress call. After startup, scheduling and dispatch add no allocation.

Admission

Suppressing a duplicate command

Runs when

a command has passed authority, ordering, and freshness checks

How often

Once per command that reaches admission, on the control path.

Sending each command over two independent paths is what makes one path failing survivable. When both paths work — which is most of the time — the receiver gets everything twice, and acting twice on "rotate 10 degrees" is not the same as acting once.

Each command carries an identity: which robot, which session, which authority epoch, which stream, and its sequence number. The receiver remembers the identities it has recently accepted and rejects a second arrival.

Remembering all of them would grow without bound, so it remembers a fixed number of the most recent, in a ring. When the ring is full the oldest entry is overwritten.

ring of 4 remembered identities #41 #42 #43 #44 next to overwrite #45 on path A first observation #45 on path B duplicate — rejected After #45 is accepted it replaces #41 and the pointer advances. A duplicate does not advance it, so a repeated command cannot push older identities out of the window and make itself admissible again.
Figure 12.2 A full ring is the normal steady state, not a fault. The oldest entry is overwritten rather than reported as exhaustion.

Observe(window, x, C)

window the ring, its write position, and how many slots are occupied · x the arriving command's identity · C the capacity the profile declared

  1. for i = 1 to window.occupied
  2. if window.slot[i] == x
  3. return duplicate // the write position does not move
  4. window.slot[window.next] = x
  5. window.next = (window.next mod C) + 1
  6. if window.occupied < C
  7. window.occupied = window.occupied + 1
  8. return first-observation

Why a full window is not an error

Most bounded structures in this series reject work when they fill up. This one does not, and the difference is what filling up means. A full assembly table means more samples are in flight than the deployment planned for. A full duplicate window just means enough commands have been accepted to fill it, which happens almost immediately and stays true forever. Reporting that as a resource fault would report a fault continuously during normal operation.

Why rejected commands are not remembered

The identity is recorded only for a command that was actually accepted — after it has passed the authority, ordering, freshness, and clock checks. A command rejected for arriving too early relative to its clock may become admissible when it arrives again on the other path, and recording the rejected copy would suppress the copy that would have worked.

Cost

One allocation of C slots at startup. Testing and inserting are O(C) in the worst case with no extra storage, no lock, and no system call. The scan is linear on purpose: C is small and set by the deployment, and an index with better asymptotic behaviour would cost ownership and allocation to save nothing at this size.

Authority

Holding control authority

Runs when

control is granted, renewed, revoked, or handed to another operator

How often

Rarely, but every command checks the result.

Only one operator may be driving at a time, and handing control over has to survive the messages involved being delayed or repeated. Authority is held as a session, an epoch that increases on every grant, and a flag saying whether it is currently active.

Revoking clears the flag but keeps the session and epoch. That is the part worth stating plainly, because discarding them looks tidier and is wrong: a later grant must carry a strictly newer epoch than anything seen before, and forgetting the epoch is what would let a replayed old grant be accepted after a revoke.

Adopt-Authority(a, grant)

a the held authority — session, epoch, and active flag · grant an authenticated grant naming a session and epoch

  1. if grant.session == a.session and grant.epoch ≤ a.epoch
  2. return reject // not newer than something already seen
  3. a.session = grant.session
  4. a.epoch = grant.epoch
  5. a.active = true
  6. return ok

Revoke-Authority(a)

a the held authority

  1. a.active = false // session and epoch are deliberately retained

The race this closes

An operator is granted control at epoch 7. Control is revoked. The grant message for epoch 7 is delayed in the network and arrives again. If revoking had cleared the epoch, that grant is indistinguishable from a first grant and is accepted, and an operator who was removed is driving again. Because the epoch survives, the replayed grant fails the test on line 1.

What a progress argument does and does not add

Everything above is about what must not happen. A separate obligation covers the other kind of failure: a command copy that is selected for delivery and then simply never looked at. Each copy is either received and put through admission, or classified as lost, and either way it leaves the set of outstanding copies. Since that set only shrinks and the worker keeps servicing it, it eventually empties.

The claim is deliberately weak. It permits every copy to be lost. What it rules out is the receive machinery holding a copy forever without deciding about it, which no amount of checking that bad commands are rejected would catch.

Freshness without comparing host clocks

Robot-owned command deadlines

Runs when

the robot issues a permit or considers a command

How often

One deadline check per admission; permit refresh follows the product configuration.

The robot decides how long a command may remain useful. It issues a permit tied to its current observation and control session, and keeps the expiry locally. The controller echoes that permit with its command. The robot checks the deadline using the same local elapsed-time clock that created it. No controller timestamp participates in this mode.

This is an optional application protocol above DDS, not a change to the DDS wire standard. It is separate from the timestamp-based admission mode described earlier. The owner is robot_control_session in C++ and RobotControlSession in Rust; both keep the operator lease and command windows together.

Issuing a permit

Let L be the configured command-window lifetime, A the maximum age of the robot observation, o its acquisition time, and t the robot's current time. The deadline is min(t + L, o + A). A new permit cannot make an old observation fresh. The owner rejects expired or future observations, invalid configuration, and arithmetic overflow.

Each permit contains the robot, session, authority epoch, observation, serial, and an unpredictable challenge. The robot stores the complete permit in one of W configured slots. Refreshes are separated by R, with W ≥ ceil(L / R); a still-live slot is never overwritten. The controller cannot choose or extend the stored deadline.

Admit-Robot-Command(session, command, now)

session the robot-owned lease and permit windows · command an authenticated, authorized command with its permit and sequence · now the robot's local elapsed time

  1. if session is retired or now regresses then retire and reject
  2. Expire-Operator-Lease(session, now)
  3. if no live operator lease then reject
  4. slot = Locate-By-Serial(session.windows, command.permit.serial)
  5. if slot is absent or its complete permit differs then reject
  6. if now ≥ slot.deadline then reject
  7. if command.sequence ≤ session.last-accepted then reject
  8. session.last-accepted = command.sequence
  9. return admitted

Acceptance requires both a live operator lease and an unexpired permit. Renewing the lease does not extend an existing permit. A successful handover, revocation, or lease expiry discards its grants; a rejected request preserves still-valid grants. Authorization loss retires the owner through its Security collaborator. UDP and SHM use that same owner.

What the deadline guarantees

A rejected command does not advance the accepted sequence. Admission at the exact deadline is rejected. These checks decide whether the robot may consider a command; they do not guarantee timely network delivery or that an actuator has executed it. DDS acknowledgement, robot admission, and actuator completion are three different events.

The product must qualify its robot clock, including suspend and drift, and the source of sensor timestamps and unpredictable challenges. Losing trust in that clock retires the owner. Defaults in a configuration file are tuning values, not validated clinical safety limits.

Cost

Permit lookup, deadline checks, and sequence updates take O(1) work in fixed-capacity storage. W selects how many slots may be used. Session setup initializes that storage; cryptography, encoding, transport, sensor acquisition, and actuator execution have separate costs. No command payload is retained by the deadline owner.

Next: resource limits

Every part above bounds something by configuration. What those bounds are and what happens at each.

Part 13 →

These descriptions follow the implementations in Tin DDS and Retina. Evaluation access and the full algorithm notes are available from engineering@tinverse.com.