read-sequence*

An enhanced read-sequence

1 Introduction

Applications that care about I/O performance really want to read data into buffers by means of read-sequence instead of calling read-byte for each byte or read-char for each character. However, when implementing network protocols this might be impractical.

1.1 Short Reads

A message might not come with some header indicating the length but may be delimited otherwise. read-sequence as specified would block until the whole requested buffer has been filled. The effect is that the application may block although a whole message has arrived already.

We would need a variation of read-sequence that would block only once for the first element to arrive and would read as many elements as are immediately available and fit into the given buffer and return. Which is the behavior of the POSIX read(2) system call.

read-sequence* provides this facility with the :short keyword argument.

1.2 Timeouts

Further, a robust implementation of some network protocol may want to establish meaningful timeouts when waiting for data. Not all timeouts need to be fatal; it could also be that some connection handling loop may have scheduled tasks at some time.

read-sequence* offers a :timeout argument, which when given specifies a timeout for the read operation. Any data read before reaching this timeout is reported together with a second return value indicating that read-sequence* returned prematurely because of a timeout. So no data has been lost.

Note: The latter, no data has been lost, cannot be achieved with alternative mechanisms some Lisp have for having either a global timeout or a stream specific timeout which would signal a condition in case timeout was reached. At the time we are about to handle the condition we have no means to know about the data that actually might have been already read.

2 Dictionary

2.1 Stream Functions

read-sequence*
sequence stream
&key start end timeout short
Function
→ first-not-done, status

    

Arguments

sequence

a sequence to read to.

stream

an input stream.

start, end

bounding index designators of sequence. The defaults for start and end are 0 and nil, respectively.

timeout

a non-negative real specifying the timeout in seconds. Or nil for no (infinite) timeout.

short

a generalized boolean indicating whether read-sequence* may return even if not all requested elements are read.

Returns

first-not-done

Sequence index of the first element not read.

status

One of nil, :eof, or :timeout:

Description

Successively reads elements from the input stream stream and destructively deposits them into the designed subsequence of sequence.

If timeout is not nil and that timeout was reached while waiting for any available stream element, the primary return value is the the index of the first sequence element not read and the second return value is :timeout.

If an end of file is reached while waiting or reading a stream element the end of file is consumed and the function returns the index of the first sequence element not read and :eof.

If all the requested stream elements have been read the function returns the given end and nil indicating normal completion.

If short is not nil the behavior is modified for waiting for and reading stream elements after the first stream element was read and deposited into sequence so that no further blocking occurs while reading the remaining requested stream elements. Stream elements are then successively copied into sequence until either all of the specified subsequence has been filled or no more input is immediately available. The primary return value is the first sequence index not written to and the second value is nil indicating normal operation.

Notes

In case short is true it is expected that actual I/O in form of a system call or similar happens at most once.

The end of file might not be a permanent condition and may be cleared by reporting it. Callers can tell whether end of file has been cleared (consumed) by checking the status return value for being :eof; only in this case the end of file condition has been cleared or consumed.

read-byte-no-hang
stream
&optional eof-error eof-value
Function
→ byte or nil

    

Like read-char-no-hang but for reading bytes.

read-byte*
stream
&key eof-value eof-error-p timeout timeout-value timeout-error
Function
read-char*
stream
&key eof-value eof-error-p timeout timeout-value timeout-error
Function

2.2 Gray Stream Extension

stream-read-char-with-timeout stream timeout Generic Function
stream-read-byte-with-timeout stream timeout Generic Function

    

Reads a character or a byte from the input stream stream but blocking at most timeout seconds.

In case of end of file :eof is returned. In case the timeout was reached nil is returned. Otherwise, returns the requested character or byte.

The timeout argument should be a non-negative real number.

stream-read-sequence stream buffer start end timeout short-p Generic Function

stream-buffered-chars-count stream → count definitive-p Generic Function
stream-buffered-bytes-count stream → count definitive-p Generic Function

    

Description

Returns the minimum number of characters (bytes) that can be read from the internal buffer of stream without any need to initiate any system call. The second return value definitive-p is a generalized boolean indicates whether the answer given is an accurate count of the buffered stream elements.

Notes

Note that 0 is always a legitimate answer as the returned count is the minimum. Returning 0 does not indicate that there is no buffered data.

Indeed, the fallback implementation just returns 0 for it cannot know anything about the stream's buffer.

3 Discussion

3.1 listen and read-byte-no-hang

Somewhat related is the issue that listen as specified is impractical. In case of end of file listen would return nil as listen indicates whether further stream elements are available, not whether a read-byte or read-char would hang.

Common Lisp has a read-char-no-hang function to mitigate this. However, there is no read-byte-no-hang. We provide one.

ANSI-CL says: "If an end of file is encountered, listen returns false."

However, it also says "listen is intended to be used when input-stream obtains characters from an interactive device such as a keyboard.". We could argue that hitting ^D is input. We could also argue that closing a TELNET connection is input.

And in fact CCL (as version 1.12) has listen implemented so that with interactive streams it returns true on end of file. Network connections are considered interactive streams as well. This choice of CCL makes perfect sense, but is not how the majority would read the ANSI-CL spec.

Hence, I want a new function that can substitute listen and is well specified. A natural choice to name it would be input-available-p but I fear this name would cause confusion again for the "end of file is no input" faction. The other option would be input-would-not-block-p but there is a negation which is bad.

4 Future

We will need the same for write-sequence.

5 TODO