-------------------------------
                         spoa-mirror for HAProxy
                     -------------------------------


Summary
------------------------------------------------------------------------

  1. Introduction
  2. Build instructions
  3. Use of the program
  4. HAProxy compatibility
  5. Known bugs and limitations


1. Introduction
------------------------------------------------------------------------

Replicating (mirroring) HTTP requests using the HAProxy SPOP, i.e. Stream
Processing Offload Protocol.

This is a very simple program that can be used to replicate HTTP requests
via the SPOP protocol.  All requests are replicated to the web address (URL)
selected when running the program.

The program originated from a test program located in the contrib/spoa_example
directory.  spoa.c is an original program that comes with HAProxy source.

The program uses the libev event library, as opposed to the original program
that uses the libevent library.


2. Build instructions
------------------------------------------------------------------------

Prerequisites for configuring and compiling the program:

  * GNU autoconf
    https://www.gnu.org/software/autoconf/

  * GNU automake
    https://www.gnu.org/software/automake/

  * POSIX threads library

  * cURL library
    https://curl.haxx.se/

  * libev library
    http://software.schmorp.de/pkg/libev.html


Creating the build subsystem (configure script):

  % ./scripts/bootstrap


Configuring the program:

  % ./configure

The program can be configured with debugging features enabled, but it is
not recommended for actual use:

  % ./configure --enable-debug

The program can also be built with the sanitizers, which is how the SPOP
regression tests detect the memory errors.  The address sanitizer is used
when no list is given, and any list that the compiler accepts may be set:

  % ./configure --enable-sanitizer
  % ./configure --enable-sanitizer=address,undefined

To configure the program, the system must have installed development packages
for cURL and libev, which will be selected automatically when running the
configure script.


Compiling the program:

  % make all

After that, in the directory src will be located spoa-mirror program.

The util directory holds the decode-data program, which decodes the SPOP frame
data copied from the log created by spoa-mirror.  It is described in more detail
in util/README.


3. Use of the program
------------------------------------------------------------------------

The options supported by the program can be found using the '-h' option:

  % ./src/spoa-mirror -h

--- help output -------
Usage: spoa-mirror { -h --help }
       spoa-mirror { -V --version }
       spoa-mirror { -r --runtime=TIME } [OPTION]...

Options are:
  -a, --address=NAME              Specify the address to listen on (default: "0.0.0.0").
  -B, --libev-backend=TYPE        Specify the libev backend type (default: AUTO).
  -b, --connection-backlog=VALUE  Specify the connection backlog size (default: 10).
  -C, --connection-timeout=VALUE  Set the maximum time to establish a connection (default: 20.00s).
  -c, --capability=NAME           Enable the support of the specified capability.
  -D, --daemonize                 Run this program as a daemon.
  -F, --pidfile=FILE              Specifies a file to write the process-id to.
  -h, --help                      Show this text.
  -i, --monitor-interval=TIME     Set the monitor interval (default: 5.00s).
  -l, --logfile=[MODE:]FILE       Log all messages to logfile (default: stdout/stderr).
  -m, --max-frame-size=VALUE      Specify the maximum frame size (default: 16384 bytes).
  -n, --num-workers=VALUE         Specify the number of workers (default: 10).
  -p, --port=VALUE                Specify the port to listen on (default: 12345).
  -r, --runtime=TIME              Run this program for the specified time (0 = unlimited).
  -T, --timeout=VALUE             Set the maximum time for a single transfer operation (default: 10.00s).
  -t, --processing-delay=TIME     Set a delay to process a message (default: 0).
  -u, --mirror-url=URL            Specify the URL for the HTTP mirroring.
  -I, --mirror-interface=NAME     Specify the interface/address for outgoing connections.
  -P, --mirror-local-port=VALUE   Specify the local port range for outgoing connections.
  -V, --version                   Show program version.

Supported libev backends: select, poll, epoll, linuxaio.

Supported capabilities: fragmentation, pipelining, async.

The fragmentation and async capabilities are supported only by HAProxy older
than 3.1; newer versions negotiate pipelining only.

Allowed logging file opening modes: a, w.  The 'a' mode allows openning or
creating file for writing at end-of-file.  The 'w' mode allows truncating
the file to zero length or creating a new file.  If a capital letter is used
for the mode, then line buffering is used when writing to the log file.

The time delay/interval is specified in milliseconds by default, but can be
in any other unit if the number is suffixed by a unit (us, ms, s, m, h, d).

Copyright 2018-2026 HAProxy Technologies
SPDX-License-Identifier: GPL-2.0-or-later
--- help output -------


The program simultaneously supports short and long format options.


The '-a' option accepts an IPv4 address, an IPv6 address, or a host name.  The
agent tries IPv4 addresses first, preserving IPv4 connections to names such as
'localhost'.  It tries IPv6 addresses only if no IPv4 address can be bound.

The test directory contains the configuration files for HAProxy (haproxy.cfg)
and SPOE (spoe.cfg).  haproxy-ssl.cfg adds an HTTPS frontend, haproxy-mw.cfg
starts HAProxy in master-worker mode and declares the agent as a program, and
ssl-cert.pem is a certificate intended for testing the HTTPS frontend.  The
get-absolute-uri.sh script sends a request with an absolute-form target.

The spop-test.sh script runs the SPOP regression tests.  It starts the program
itself, talks to it with the spop-client.py client, which implements the SPOP
protocol independently of the program, and checks the frames that it gets back.
The program that is tested is chosen with the '-a' option, the cases to run may
be named on the command line, and the '-l' option lists them.  Besides the frame
exchange, the script also checks that the program refuses command line arguments
that are out of range.

A case fails when the client reports an error, when a sanitizer writes to the
log of the program, or when the program is no longer running once the client is
done; the logs of a run that failed are kept.  The cases that send a fragmented
frame or a truncated header block, and the one that leaves the mirrored requests
running while the program is stopped, are reported as skipped unless the program
is built with the address sanitizer, which is what makes the overflow and the
leaks that they leave behind visible.  A case that needs an option the program
was not built with is skipped as well.

The test/debug subdirectory holds the same configurations extended with the
SPOE variable captures, and its spoe.cfg also refers to the check-client-ip
and test messages.  Those messages are commented out there and are meant to
be enabled only when debugging the program, which is most useful when it is
configured with the --enable-debug option.


First, the spoa-mirror program should be launched by referring to the URL to
which the HTTP request will be copied.  In the example below, URL is set to
http://localhost:8100/ .

  % ./src/spoa-mirror -r0 -u"http://localhost:8100/"

'-r' is the option that must be specified when launching a program.  It is
used to prevent the program from starting unnecessarily when testing the
program options, or printing the program help.  In addition, with this option,
the program runtime is set.  If the time is set to 0, the program runtime is
unlimited.


After that, the HAProxy should be started (as the root user), using haproxy.cfg
configuration file:

  # /path-to/haproxy -f haproxy.cfg


Note related to SPOE configuration: in the 'spoe-message' section (in the
given example, this section is called 'mirror') the 'args' keyword must always
(and only) contain the following arguments (after the sign '=' there is a
permitted argument type):

  arg_method=STR arg_path=STR arg_ver=STR arg_hdrs=(BIN|STR) arg_body=BIN

'arg_hdrs' can have a binary (req.hdrs_bin) or string (req.hdrs) data type
argument specified.  In case we want to make some changes in the HTTP headers
that are sent to the mirror URL, then we will use the string data type for
that argument to which we can add the regsub() converter, as in the example:

  .. arg_hdrs=req.hdrs,regsub(foo,bar)


4. HAProxy compatibility
------------------------------------------------------------------------

The program implements version 2.0 of the SPOP protocol and works with all the
HAProxy versions that support it.  Starting with HAProxy 3.1 the protocol is
handled by a dedicated multiplexer, while the SPOE filter keeps the agent and
message configuration.

That multiplexer announces and accepts the pipelining capability alone, so the
fragmentation and async capabilities are negotiated only with older versions.

The backend used by the SPOE agent should be declared with 'mode spop', but a
backend left in 'mode tcp' is still converted automatically, so the shipped
configurations keep working.

The following spoe-agent keywords are still parsed by the newer versions but
they no longer have any effect: 'option async', 'option send-frag-payload',
'maxconnrate', 'maxerrrate' and 'max-waiting-frames'.


5. Known bugs and limitations
------------------------------------------------------------------------

HAProxy should be used in SSL termination mode if HTTPS is used in the HAProxy
frontend part.

All the HTTP methods are mirrored.  The body of a request is sent with every
method that carries one, and the agent leaves the framing of the request to
the cURL library, so the length that is announced always matches the body that
is really sent.  A request that has no body is mirrored as it came.

The agent removes client framing and hop-by-hop headers before mirroring the
request.  This includes every header named in any client Connection header,
regardless of its position or case.  Other client headers are preserved.

The 'option http-buffer-request' must be set in the HAProxy configuration,
because otherwise the contents of the HTTP body are not transferred to the
agent, and the mirrored request carries no body.