Predictive feedback augmentation for manual control of an unmanned aerial vehicle with latency

Introduction

Unmanned aerial vehicles (UAVs) are seeing widespread growth in many existing and new applications. Presently, UAVs are often operated within the line of sight (LoS), where the control delays come primarily from the control radio system in use. Entry level hobby transmitters commonly achieve latencies in the order of 20–30 ms. Advanced hobby radio systems boast ranges of 60 km.^1–6

When operating via a local video link, camera latencies range from 40 ms for the fastest analog systems to hundreds of milliseconds for digital systems. Recently, hobby grade low latency digital video transmission has been introduced, with latencies below 50 ms achievable. Latency of wifi-based transmission systems can exceed 100 ms. Adoption has been low, likely due to cost and fragility.^7–9

Some military UAVs are operated from thousands of kilometers away via encrypted satellite connection, which imposes a round trip delay in the order of seconds. Often the terminal flight phases are controlled from ground crews near the runway. Due to the short time scales on which the state of the vehicle can change during takeoff and landing, the latency presents difficulty in control. Outside of terminal flight phases, the latency is acceptable as maneuvers and disturbances occur on timescales much larger than the latency.^10,11

Proposed system overview

To combat latency, the client-side prediction concept is applied to a UAV. The adaptation to a UAV can be summarized as predicting and displaying the vehicle state at the time when the commands being given currently will arrive at the vehicle. The eventual goal is to implement the scheme on a micro (112 mm size) quadrotor flown in rate mode using a video feed from onboard as feedback. Rate mode means that the operator’s stick position maps to an angular rate command about all three axes, which is the most primitive mode, usually with only a single loop in the controller on each axis. The operator receives information about the vehicle’s state from the video feed, including the angular position from the image of the horizon.

Figure 1 shows the flow of commands and feedbacks around a time t_n at the remote station. 2τ is the roundtrip latency (the one-way latencies have been assumed to be identical and equal to τ). A convenient choice for the origin of the time axis is t n = t n ′ = 0 s at the start of the flight (the time of the first command) so that for any command sent t n = t n ′ .

OPEN IN VIEWER

Figure 1.

Flow of command and feedback.

At the control station at t_n, the remote station has the feedback that is delayed by τ, but it can also see the commands that have already been sent. The commands between t_n – 2τ and t_n will arrive at the quad after the feedback at t_n – 2τ, and prior to the command being given at t_n and arriving at t_n. This command information is then used to predict the change in vehicle state, and hence the feedback that the operator would expect to see if there was no delay.

The mathematics of the problem can be described more simply as:

Given the system state and command inputs to be executed, predict the system state after those command inputs have been applied, which is a familiar problem from control theory with well-developed means of solution.

System block diagram

Figure 2 shows the suggested system, which is similar to the Smith predictor.²² The operator is modeled as determining a desired attitude, comparing that attitude to the feedback they are shown and then applying a controller to determine a command input. The compensator compares two simulated flight states, one of which is delayed by the round trip delay time and then augments the feedback by this amount. The remainder of the system is formed by the actual transmission delay and vehicle dynamics.

OPEN IN VIEWER

Figure 2.

Block diagram with delay and compensator.

Because the implementation later in this paper deals with an angular rate controlled quadrotor, while the attitude feedback is an angular position from a video stream, some integrator blocks are included. These terms are shown for completeness and could be modified if a different control arrangement was used.

The diagram shown is for the one-dimensional case, where θ is any of the roll pitch or yaw axes. A real implementation will require propagation of these axes in three dimensions.

The compensator can be removed by setting G_est = 0, which gives the system with only the delay. The delayed system can be reduced to the typical system by setting τ = 0, effectively removing the delay. Using the fact that the compensator and delay can be removed by substituting terms, the closed loop transfer functions need only be found for the delay and compensator case and the others will be readily found from those results.

Output and reported output response to command derivation

First, the closed-loop transfer functions are found for the block diagram with the compensation system. For convenience, we define

A ( s ) ≡ P ( s ) [ G est ( s ) + e − 2 s τ ( G ( s ) − G est ( s ) ) ]

(1)

≈ P ( s ) G ( s )

(2)

which approximately is equal to the latency-free open-loop transfer function for the pilot gain-quadrotor system if G_est(s) ≈ G(s). The displayed feedback and actual response can then be found

θ display ( s ) θ c ( s ) = A ( s ) s + A ( s )

(3)

θ ( s ) θ c ( s ) = e − s τ P ( s ) G ( s ) s + A ( s )

(4)

The delay term in the numerator is unavoidable, as the input cannot reach the output without being subject to transmission delay (and a negative delay is infeasible without predicting c(s) a short time in advance).

The disturbance response can also be found

θ ( s ) d ( s ) = 1 − e − 2 s τ P ( s ) G ( s ) s + A ( s )

(5)

which makes intuitive sense as the response is the disturbance plus a delayed, negative response to the disturbance. Once again the delay in the numerator is unavoidable as the disturbance must travel once around the loop to pass through the pilot and correct the disturbance.

Interpretation

The transfer functions for the compensated system can be used to find the delayed system and undelayed system for comparison. The control response transfer functions are shown in Table 1 and the disturbance response transfer functions in Table 2. With A(s) highlighted in green and delay terms highlighted in red.

OPEN IN VIEWER

Table 1.

Transfer functions for the control response.

	θ ( s ) θ c ( s )
No delay	P ( s ) G ( s ) s + P ( s ) G ( s )
Delay	e − s τ P ( s ) G ( s ) s + e − 2 s τ P ( s ) G ( s )
Delay and compensator	e − s τ P ( s ) G ( s ) s + A ( s )

OPEN IN VIEWER

Table 2.

Transfer functions for the disturbance response.

	θ ( s ) d ( s )
No delay	1 − P ( s ) G ( s ) s + P ( s ) G ( s )
Delay	1 − e − 2 s τ P ( s ) G ( s ) s + e − 2 s τ P ( s ) G ( s )
Delay and compensator	1 − e − 2 s τ P ( s ) G ( s ) s + A ( s )

The introduction of the delay causes a delay term in the numerator of the command response and the counteraction to the disturbance.

The delay term in the numerator is caused by the forward latency for the command response and by disturbances needing to be observed and then responded to for the disturbance response. Addressing these delay terms is beyond the scope of this paper.

A delay term is also seen in the denominator. This term is caused by the operator being unable to immediately see the feedback from their inputs.

The introduction of the delay compensator modifies the term in the denominator. The term A(s), highlighted in green, still contains the delay term as the feedback from the vehicle is incorporated. If a suitably accurate estimate of the vehicle dynamics G_est(s) is known, then the term in the denominator becomes approximately equal to the undelayed denominator. The compensator acts to provide feedback that shows immediately the effect of the command inputs, removing the effect of the delay from the denominator.

The change that the compensator brings to the denominator makes the transfer functions much more similar to the no delay case. The command response is affected only by the forward latency, which delays the response but otherwise does not affect its behavior. The disturbance response is improved by the compensator, but it will still take at least the roundtrip latency before the pilot begins to correct disturbances.

Estimation error

If the vehicle attitude is estimated by integrating a model, which provides the vehicle attitude rate, then

θ est ( t ) = ∫ 0 t θ ˙ est d t

(6)

= ∫ 0 t θ ˙ + θ ˙ est . error d t

(7)

= θ ( t ) + ∫ 0 t + θ ˙ est . error d t

(8)

where θ represents an angular position, θ ˙ est is an estimate of the angular rate obtained from a model of the dynamics applied to the recorded inputs, and θ ˙ est . error is the error in θ ˙ est . The issue with a dead reckoning approach can be seen here; the size of the error will tend to grow with flight time. If the displayed attitude is based on an out of date attitude plus a prediction of its upcoming changes, then

θ display = θ est ( t n ) − θ est ( t n − 2 τ ) + θ ( t n − 2 τ )

(9)

= θ ( t n ) + ∫ 0 t n + θ ˙ est . error d t − [ θ ( t n − 2 τ ) + ∫ 0 t n − 2 τ + θ ˙ est . error d t ] + θ ( t n − 2 τ )

(10)

= θ ( t n ) + ∫ t n − 2 τ t n θ ˙ est . error d t

(11)

Only errors in the model that manifest between t_n – 2τ and t_n form part of the displayed system state. Importantly, the error does not accumulate with time and large errors (such as from disturbances or rapid motions where model errors are amplified) will only manifest for the delay time. Also, so long as the latency 2τ is small, the error in the displayed information is minimal. If a dead reckoning approach is used, then the error will increase by roughly a factor of t_n/2τ, and will hence grow with time.

System identification

To predict the motion of the vehicle, a dynamic model is needed.

For the quadrotor vehicles considered in this paper, the dynamics of the three rotational axes were assumed to be independent of each other and the flight condition (speed, altitude, throttle setting). This assumption is valid only in rate mode.

The development of high rate flight data recording ²³ allows system identification by measurement rather than modeling. Quadrotor models do exist, but model measurement was chosen due to the importance of accuracy to the specific hardware and software configuration. A large amount of vehicle geometry information and parameters would also be needed for a derived model approach, many of which are difficult to measure and quoted incorrectly by the manufacturer.

Identified system

For the 112 mm quadrotor used, eight flight logs were used to generate a system model. The procedure was validated by generating eight models with a log each excluded and verifying that the models remained valid on the excluded log. Bode plots of the system model generated from all eight logs are shown in Figure 3. The models map angular rate command to the angular rate.

OPEN IN VIEWER

Figure 3.

The models generated for the 112 mm quadrotor.

Augmenting the feedback

With the change in vehicle attitude predicted, the feedback can be augmented to reflect this prediction. The video is first undistorted to remove fisheye effects from the camera lens, and then warped so that the horizon (and other objects far from the camera) appears to move immediately with operator input.

Figure 4 shows an example of image warping for a prediction of rolling. In this example, the feedback shows the quadrotor in a near-inverted attitude, but the command inputs that are yet to arrive will bring the quadrotor to nearly level, so the compensator warps the image such that the horizon appears level. Warping for the yaw and pitch axes is achieved by shifting the image left and right, with the camera field of view determining the rate (pixels per degree) of shifting.

OPEN IN VIEWER

Figure 4.

Example of image warping. The image is warped to reflect the compensator predicting a 135° roll to the right.

Simulated predictive performance

Before implementing the compensator, it is desirable to estimate the predictive performance and to understand how the duration of the latency will affect the performance. The predicted position can be expressed in terms of the actual position and the error in the modeled system as

θ display = θ ( t n ) + ∫ t n − 2 τ t n θ ˙ est . error d t

(12)

∫ t n − 2 τ t n θ ˙ est . error d t = ∫ 0 t n θ ˙ est . error d t − ∫ 0 t n − 2 τ θ ˙ est . error d t

(13)

The accumulated error from the start of the flight is found by integrating the difference of the measured output and the output modeled from the measured input. The difference across a moving window is then taken to find the prediction errors throughout a flight log. The predictive performance was only measured on logs not used to generate the model used to assess performance (as a model derived from the flight log after a flight cannot be used to predict during that flight).

A histogram of the prediction errors was computed to visualize the performance. A histogram of the typical prediction errors on the roll axis with a 1 s delay is shown in Figure 5. The standard deviation of predictive errors with a 1 s delay for this flight is 1.1°, and most flights show similar distributions of errors, with no more than a couple of degrees standard deviation. A more thorough analysis of the simulated performance and variation in performance with latency can be found in Cox. ²⁵ With predictive performance within a few degrees, the concept is now ready for implementation and proof of concept.

OPEN IN VIEWER

Figure 5.

Histogram of predictive errors on the roll axis throughout a flight, for a delay time of 1 s.

Implementation

The latency compensation system was implemented on the 112 mm size quadrotor. The quadrotor was fitted with a fixed camera with analog transmission. The real-time implementation had three key parts:

Introduce an artificial latency

The Node.js environment was chosen for implementation. Node.js is a javascript runtime, designed for internet connected applications and has event-based features. The event-based features are useful for running the simulation in real time, I/O in real time, and introducing delays. There is a large ecosystem of pre-existing libraries for a variety of tasks, including mathematical tasks, image processing and I/O. The Node.js environment will also be ideal for future work implementing on a real source of latency. Node.js is fast, but being an interpreted javascript environment it is not fast enough to run the image warping in real time.

OpenCV was used to perform the image undistortion and warping, which was faster than Node.js because it runs as precompiled C code.

The architecture for testing the compensator is shown in Figure 6. Regular flight uses Node.js as a passthrough to display video, to give a baseline. Flying with latency uses the Node.js environment to introduce latency to the system, and latency compensated flight uses Node.js to introduce latency and augment the feedback. All command and feedback passes through the Node.js environment.

OPEN IN VIEWER

Figure 6.

The flight test architectures.

Figure 7 shows the simulation arrangement used. The simulation is indicated in blue, beginning from the start of the flight. The vehicle states in red are simulated and still in memory at t_n. The simulated vehicle states at t_n and t_n – 2τ are compared to find the expected change in the feedback. A single propagation of the flight was used as it is difficult to guarantee that two simulations running side by side but separated in time will run with identical inputs, and hence may be prone to drift apart. In future work, input to the simulation from telemetry feedback may be necessary to ensure that the simulation does not drift from the true vehicle state. Because the quadrotor’s angular response does not vary with flight condition, this architecture is sufficient.

OPEN IN VIEWER

Figure 7.

Simulation and computation of the predicted feedback.

Flight testing and results

The 112 mm quadrotor was flown without any latency, with latency and with compensated latency. Qualitative results are presented based on pilot remarks.

The test pilot did not have any remarks about abnormal flight behaviors when flown without latency. The size of the quadrotor meant that its position was quite susceptible to gusts.

Latencies of 500 ms and 1000 ms were tested. With a 500 ms latency, the test pilot struggled to maintain control. Early flights resulted in crashes. Later flights the test pilot was able to maintain flight, but only achieving the most basic maneuvers and tasks. Control was achieved by giving brief inputs and then waiting to observe the effect of those inputs before determining the next input. Controlled flight was not achieved with a 1000 ms delay.

Compensated latencies of 500 ms and 1000 ms were tested. The test pilot remarked immediately when flying with 500 ms of compensated latency that the aid allowed smooth control of the vehicle. With both 500 and 1000 ms of compensated latency, tasks such as following a fence line were achieved. Controlled flips in the roll and pitch axis were achieved. Figure 8 shows the pilot’s view at various points through a flip at 1000 ms latency. The flips highlighted that not accounting for the delays in the various command and feedback interfaces had an effect on the prediction. The compensator expected the video to show the vehicle rolling briefly before it did, so the horizon dipped from level as the quad was “catching up” to the pilot. This bug will be addressed in future flight testing.

OPEN IN VIEWER

Figure 8.

Frames displayed by the compensator during a flip flown with 1000 ms latency. The issue introduced by the latency not accounted for can be seen in frames (f) and (g). Frames are labeled (a)–(h) chronologically. Frames on the left are taken from when the flip is commanded (vehicle remains level) while frames on the right are taken from when the vehicle executes the flip (horizon should appear to remain level, but does not because the compensated latency is slightly less than the true latency). The total time between the first and last frame is approximately 2 s.

Conclusion

The problem of teleoperation has been explored, and the client-side prediction model has been borrowed from multiplayer videogames to address the control latency problem in UAVs. The mathematical basis for the concepts viability has been explored and a scheme that does not suffer from drift has been found. The concept has been implemented and proof of concept achieved in flight testing. Future work may refine the system, explore systems where drift of intermediate states like airspeed could affect the system response or explore implementation with a real latency source (such as an internet connection).