ByteToMessageDecoder

Purpose

A ByteToMessageDecoder provides a simplified API for handling streams of incoming data that can be broken up into messages. This API boils down to two methods: decode, and decodeLast. These two methods, when implemented, will be used by a ByteToMessageHandler paired with a ByteToMessageDecoder to decode the incoming byte stream into a sequence of messages.

The reason this helper exists is to smooth away some of the boilerplate and edge case handling code that is often necessary when implementing parsers in a SwiftNIO ChannelPipeline. A ByteToMessageDecoder never needs to worry about how inbound bytes will be buffered, as ByteToMessageHandler deals with that automatically. A ByteToMessageDecoder also never needs to worry about memory exclusivity violations that can occur when re-entrant ChannelPipeline operations occur, as ByteToMessageHandler will deal with those as well.

Implementing ByteToMessageDecoder

A type that implements ByteToMessageDecoder may implement two methods: decode and decodeLast. Implementations must implement decode: if they do not implement decodeLast, a default implementation will be used that simply calls decode.

decode is the main decoding method, and is the one that will be called most often. decode is invoked whenever data is received by the wrapping ByteToMessageHandler. It is invoked with a ByteBuffer containing all the received data (including any data previously buffered), as well as a ChannelHandlerContext that can be used in the decode function.

decode is called in a loop by the ByteToMessageHandler. This loop continues until one of two cases occurs:

1. The input ByteBuffer has no more readable bytes (i.e. .readableBytes == 0); OR

2. The decode method returns .needMoreData.

The reason this method is invoked in a loop is to ensure that the stream-like properties of inbound data are respected. It is entirely possible for ByteToMessageDecoder to receive either fewer bytes than a single message, or multiple messages in one go. Rather than have the ByteToMessageDecoder handle all of the complexity of this, the logic can be boiled down to a single choice: has the ByteToMessageDecoder been able to move the state forward or not? If it has, rather than containing an internal loop it may simply return .continue in order to request that decode be invoked again immediately. If it has not, it can return .needMoreData to ask to be left alone until more data has been returned from the network.

Essentially, if the next parsing step could not be taken because there wasn’t enough data available, return .needMoreData. Otherwise, return .continue. This will allow a ByteToMessageDecoder implementation to ignore the awkward way data arrives from the network, and to just treat it as a series of decode calls.

decodeLast is a cousin of decode. It is also called in a loop, but unlike with decode this loop will only ever occur once: when the ChannelHandlerContext belonging to this ByteToMessageDecoder is about to become invalidated. This invalidation happens in two situations: when EOF is received from the network, or when the ByteToMessageDecoder is being removed from the ChannelPipeline. The distinction between these two states is captured by the value of seenEOF.

In this condition, the ByteToMessageDecoder must now produce any final messages it can with the bytes it has available. In protocols where EOF is used as a message delimiter, having decodeLast called with seenEOF == true may produce further messages. In other cases, decodeLast may choose to deliver any buffered bytes as “leftovers”, either in error messages or via channelRead. This can occur if, for example, a protocol upgrade is occurring.

As with decode, decodeLast is invoked in a loop. This allows the same simplification as decode allows: when a message is completely parsed, the decodeLast function can return .continue and be re-invoked from the top, rather than containing an internal loop.

Note that the value of seenEOF may change between calls to decodeLast in some rare situations.

Implementers Notes

ByteToMessageHandler will turn your ByteToMessageDecoder into a ChannelInboundHandler. ByteToMessageHandler also solves a couple of tricky issues for you. Most importantly, in a ByteToMessageDecoder you do not need to worry about re-entrancy. Your code owns the passed-in ByteBuffer for the duration of the decode/decodeLast call and can modify it at will.

If a custom frame decoder is required, then one needs to be careful when implementing one with ByteToMessageDecoder. Ensure there are enough bytes in the buffer for a complete frame by checking buffer.readableBytes. If there are not enough bytes for a complete frame, return without modifying the reader index to allow more bytes to arrive.

To check for complete frames without modifying the reader index, use methods like buffer.getInteger. You MUST use the reader index when using methods like buffer.getInteger. For example calling buffer.getInteger(at: 0) is assuming the frame starts at the beginning of the buffer, which is not always the case. Use buffer.getInteger(at: buffer.readerIndex) instead.

If you move the reader index forward, either manually or by using one of buffer.read* methods, you must ensure that you no longer need to see those bytes again as they will not be returned to you the next time decode is called. If you still need those bytes to come back, consider taking a local copy of buffer inside the function to perform your read operations on.

The ByteBuffer passed in as buffer is a slice of a larger buffer owned by the ByteToMessageDecoder implementation. Some aspects of this buffer are preserved across calls to decode, meaning that any changes to those properties you make in your decode method will be reflected in the next call to decode. In particular, moving the reader index forward persists across calls. When your method returns, if the reader index has advanced, those bytes are considered “consumed” and will not be available in future calls to decode. Please note, however, that the numerical value of the readerIndex itself is not preserved, and may not be the same from one call to the next. Please do not rely on this numerical value: if you need to recall where a byte is relative to the readerIndex, use an offset rather than an absolute value.

Using ByteToMessageDecoder

To add a ByteToMessageDecoder to the ChannelPipeline use

channel.pipeline.addHandler(ByteToMessageHandler(MyByteToMessageDecoder()))