Compression (defalate, gzip)

It is a common case to compress big pieces of data for transmitting it between server and client. Since of v.0.4.4 RESTinio adds a concept of data transformators to its tooling. While itself the transform mechanics is independent it comes nicely integrated with RESTinio.

In this section, zlib transformation is introduced. It allows you to perform compress/decompress operation on the data using zlib library. Zlib supports the following compression algorithms: deflate and gzip.

Zlib transformator is not included with <restinio/all.hpp>, to use it one needs to include <restinio/transforms/zlib.hpp>. All functions and classes are located in restinio::transforms::zlib namespace. The basic usage of zlib transformator will start with something like the following:

 #include <restinio/transforms/zlib.hpp>
 // ...

 void my_handler( auto req )
 {
   namespace rtz = restinio::transforms::zlib;

   // ...
 }

zlib_t

The main class for zlib transform is restinio::transform::zlib::zlib_t. It handles all the low level staff of using zlib. All higher level functionality that minimizes boilerplate code and makes compression and decompression logic less verbose is based on zlib_t.

params_t

zlib_t must be initialized with parameters (restinio::transform::zlib::params_t) that define the operation: compress or decompress and destination format: deflate, gzip or identity. Note though identity is an imaginary compression format it means that no compression will be used and no header/trailer will be applied. identity is very helpful for cases when compression (decompression) might or might not be used and you want a single piece of code to handle these cases.

Class params_t keeps the following transform parameters:

operation: Types of transformation. Enum params_t::operation_t with the following items: compress, decompress;
format: Formats of compressed data. Enum params_t::format_t with the following items: deflate, gzip, identity.
level: Сompression level (makes sense only for compression operation). Must be an integer value in the range of -1 to 9. -1 means default level and 0 means no compression but zlib header and trailer are still present.
window_bits: Must be an integer value in the range of 8 to MAX_WBITS (constant defined by zlib, usually 15). For decompress operation, it is better to keep the default value of windows beats (MAX_WBITS). Also for decompress operation window_bits can be zero to request the use of window size obtained from zlib header of the compressed stream (See inflateInit2() description for details).
mem_level: Defines memory usage of compression operation (makes sense only for compression operation). Must be an integer value in the range of 1 to 9. 1 stands for minimum memory usage and 9 stands for maximum memory usage. The amount of memory that can be used affects the quality of compression.
strategy: Compression strategy (makes sense only for compression operation). Must be an integer value defined by one of zlib constants: Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_DEFAULT_STRATEGY. (See deflateInit2() description for details).
reserve_buffer_size: The initial size of buffer used for output. When using zlib transformator the outout buffer is used. The initial size of such buffer must be defined. zlib_t instance will use this parameter as the initial size of out buffer and as an increment size if out buffer must be enlarged.

params_t supports fluent interface for setting all the mentioned parameters except operation and format that must be in constructor. params_t has a default constructor that defines identity transformation. Note that in case of identity transformation all parameters are ignored as there is no place for them to be used.

RESTinio provides several functions for making parameters for a given operation:

params_t make_deflate_compress_params( int compression_level = -1 );

params_t make_deflate_decompress_params();

params_t make_gzip_compress_params( int compression_level = -1 );

params_t make_gzip_decompress_params();

params_t make_identity_params();

So instead of of writing something like the following:

namespace rtz = restinio::transforms::zlib;

params_t params{
  rtz::params_t::operation_t::compress,
  rtz::params_t::format_t::gzip,
  -1 };

params.mem_level( 7 );
params.window_bits( 13 );
params.reserve_buffer_size( 64 * 1024 );

It is better to write the following:

params_t params =
  restinio::transforms::zlib::make_gzip_compress_params()
    .mem_level( 7 )
    .window_bits( 13 )
    .reserve_buffer_size( 64 * 1024 );

zlib_t API

Having parameters zlib_t object can be instantiated. zlib_t is non-copiable and non-movable class with a single constructor available:

zlib_t( const params_t & transformation_params ): Initialize zlib transformator with given parameters.

As previously mentioned zlib_t hides all low level details of working with zlib. Based on parameters it initializes all appropriate structs and calls necessary functions for doing compression or decompression and deinitializes structs when the stream is completed.

The interface of zlib transform routine is the following:

const params_t & params() const

Get parameters of current transformation. Gives an access to parameters a given object is about.

void write( string_view_t input )

Append input data. Pushes given input data to zlib transform. Parameter input points to data to be compressed or decompressed.

void flush()

Flushes underlying zlib stream. All pending output is flushed to the output buffer.

void complete()

Complete the stream. All pending output is flushed and a proper trailer data is produced and inserted into stream.

std::string giveaway_output()

Get current accumulated output data. On this request, a currently accumulated output data is returned. Move semantics is applied to the output buffer. Once current output is fetched zlib_t object resets its internal out buffer. In the following code:

 namespace rtz = restinio::transforms::zlib;
 rtz::zlib_t z{ rtz::make_gzip_compress_params() };

 z.write( A );
 consume_out( z.giveaway_output() );

 z.write( B );
 z.write( C );
 consume_out( z.giveaway_output() );

function consume_out() on line(9) receives a string that is not an appended version of a string received on line(5).

auto output_size() const

Get accumulated output data size.

bool is_completed() const

Gives operation completion state.

Usage examples

Simple usage example:

namespace rtz = restinio::transforms::zlib;
rtz::zlib_t z{ rtz::make_gzip_compress_params( 9 ) };
z.write( input_data );
z.complete();

// Get compressed input_data.
auto gziped_data = z.giveaway_output();

Advanced usage example:

namespace rtz = restinio::transforms::zlib;
rtz::zlib_t z{ rtz::make_gzip_compress_params( 9 ) };

std::size_t processed_data = 0;
for( const auto d : data_pieces )
{
  z.write( d );

  // Track how much data is already processed:
  processed_data += d.size();

  if( processed_data > 1024 * 1024 )
  {
    // If more than 1Mb  is processed do flush.
    z.flush();
  }

  if( z.output_size() > 100 * 1024 )
  {
    // If more than 100Kb of data is ready, then append it to something.
    append_output( z.giveaway_output() );
  }
}

// Complete the stream and append remeining putput data.
z.complete();
append_output( z.giveaway_output() );

Easy interface

RESTinio provides a number of helper classes and functions for doing compression/decompression easier.

Single function for compression/decompression

RESTinio has a set of handy functions helping to perform zlib transform in one line:

//! Do a specific transformation.
std::string transform( string_view_t input, const params_t & params );

std::string deflate_compress( string_view_t input, int compression_level = -1 );

std::string deflate_decompress( string_view_t input );

std::string gzip_compress( string_view_t input, int compression_level = -1 );

std::string gzip_decompress( string_view_t input );

So instead of writing something like this:

namespace rtz = restinio::transforms::zlib;
rtz::zlib_t z{ rtz::gzip_compress() };
z.write( data );
z.complete();
body = z.giveaway_output();

It is possible to write the following:

body = restinio::transforms::zlib::gzip_compress( data );

Body appenders

RESTinio provides a special template class body_appender_t<Response_Output_Strategy> which is tied with response_builder_t<Response_Output_Strategy> (see Response builder). The purpose of body_appender_t for a given response builder is to help to set a compressed body data. It not only compresses the data but also sets an appropriate value for Content-Encoding field (deflate, gzip, identity).

For making body appender there is a set of factory functions:

// Create body appender with given zlib transformation parameters.
// Note: params must define a compression operation.
template < typename Response_Output_Strategy >
body_appender_t< Response_Output_Strategy > body_appender(
    response_builder_t< Response_Output_Strategy > & resp,
    const params_t & params );

// Create body appender with deflate transformation and a given compression level.
template < typename Response_Output_Strategy >
body_appender_t< Response_Output_Strategy > deflate_body_appender(
    response_builder_t< Response_Output_Strategy > & resp,
    int compression_level = -1 );

// Create body appender with gzip transformation and a given compression level.
template < typename Response_Output_Strategy >
inline body_appender_t< Response_Output_Strategy > gzip_body_appender(
    response_builder_t< Response_Output_Strategy > & resp,
    int compression_level = -1 );

// Create body appender with gzip transformation and a given compression level.
template < typename Response_Output_Strategy >
inline body_appender_t< Response_Output_Strategy > identity_body_appender(
    response_builder_t< Response_Output_Strategy > & resp,
    int = -1 );

There are 3 specializations of body_appender_t for each type of response builder.

body_appender_t< restinio_controlled_output_t >

Helper class for setting the body of response_builder_t<restinio_controlled_output_t> (see RESTinio controlled output response builder). It combines semantics of setting body data with response builder and zlib compression. the main functions of it are:

body_appender_t & append( string_view_t input). Appends data to be compressed.
void complete(). Completes the compression and sets the response body.

Sample usage:

namespace rtz = restinio::transforms::zlib;
auto resp = req->create_response(); // creates response_builder_t<restinio_controlled_output_t> by default.

resp.append_header( restinio::http_field::server, "RESTinio" )
  .append_header_date_field()
  .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" );

auto ba = rtz::gzip_body_appender( resp );
ba.append( some_data );
// ...
ba.append( some_more_data );
ba.complete();
resp.done();

body_appender_t< user_controlled_output_t >

Helper class for setting the body of response_builder_t<user_controlled_output_t> (see User controlled output response builder). Sample usage:

auto resp = req->create_response<user_controlled_output_t>();

resp.append_header( restinio::http_field::server, "RESTinio" )
  .append_header_date_field()
  .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" );

auto ba = restinio::transforms::zlib::gzip_body_appender( resp );
ba.append( some_data );
ba.flush();
// ...
ba.append( some_more_data );
ba.complete();
resp.done();

body_appender_t< chunked_output_t >

Helper class for setting the body of response_builder_t<chunked_output_t> (see Chunked transfer encoding output builder). It combines semantics of appending body data with chunked response builder and zlib compression. It has the following functions:

body_appender_t & append( string_view_t input). Append data to be compressed. Function only adds data to anderlying zlib stream and it doesn’t affect target response right on here.
body_appender_t & make_chunk( string_view_t input = string_view_t{} ). Append data to be compressed and adds current zlib transformator output as a new chunk. After adding data flushes zlib transformator. Then ready compressed data is taken and used as a new chunk of target response.
void flush(). Flushes currently available compressed data with possibly creating new chunk and then flushes target response.
void complete(). Complete zlib transformation operation and appends the last chunk.

Sample usage:

resp.append_header( restinio::http_field::server, "RESTinio" )
  .append_header_date_field()
  .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" );

auto ba = restinio::transforms::zlib::deflate_body_appender( resp );
ba.append( some_data );
ba.append( some_more_data );
ba.make_chunk(); // Flush copressed data and creates a chunk with it.
ba.flush(); // Send currently prepared chunks to client
// ...
// Copress the data and creates a chunk with it.
ba.make_chunk( even_more_data );
ba.flush(); // Send currently prepared chunks to client
// ...
ba.append( yet_even_more_data );
ba.append( last_data );
ba.complete(); // Creates last chunk, but doesn't send it to client.
ba.flush(); // Send chunk created by complete() call
// ...
resp.done();

Handle decompressed body

To simplify handling of the requests with a body that might or might not be compressed RESTinio provides a helper function for handling those cases evenly:

// Call a handler over a request body.
template < typename Handler >
auto handle_body( const request_t & req, Handler handler );

Handler is a function object capable to handle std::string as an argument.

If the body is encoded with either ‘deflate’ or ‘gzip’ then it is decompressed and the handler is called. If the body is encoded with ‘identity’ (or not specified) the handler is called with original body. In other cases, an exception is thrown.

Sample usage:

auto decompressed_echo_handler( restinio::request_handle_t req )
{
  return
    restinio::transforms::zlib::handle_body(
      *req,
      [&]( auto body ){
        return
          req->create_response()
            .append_header( restinio::http_field::server, "RESTinio" )
            .append_header_date_field()
            .append_header(
              restinio::http_field::content_type,
              "text/plain; charset=utf-8" );
            .set_body( std::move( body ) ) // Echo decompressed body.
            .done();
      } );
}

Samples

See compression and decompression samples for full working examples using zlib transformator.

Compression (defalate, gzip)

zlib_t

params_t

zlib_t API

Usage examples

Easy interface

Single function for compression/decompression

Body appenders

Handle decompressed body

Samples

Table Of Contents

Search