Repository navigation
Application Layout
As LibEMP is our orchestration layer for our monitoring platform, we are using Erlang and taking advantage of OTP's hard-learned lessons. Our basic application layout is as follows in figure 1. Note that we maintain three subtrees for each of the three LibEMP Components (Monitors, Buffers, Sinks). The only interesting subtree is the Monitor supervisory tree; both the Buffers' and Sinks' subtree tend to only ever have one supervisor and worker each (vastly depends on implementation).
The Buffer tree is initialized first, Monitors and then Sinks start afterward (in that order). This makes a difference, as our primary supervisor uses a rest_for_one pattern. That is, if the Buffer crashes, the whole system must be brought down, but if the Sink goes down, we do not want the Monitors to be restarted (as Sink issues are potentially extremely common). LibEMP will orchestrate the connection of the Buffer implementation to the individual Monitors and Sinks, thus if that connection fails as the Buffer crashed, we need to re-orchestrate the system.
Monitors are separated into two types, the custom (application specific monitors) and the built-ins (LibEMP provided). The built-ins can be seen as simplistic gen_server style workers (even if they are implemented with more complication in the future). You can toggle them on and off via configuration or ask that they shut down during runtime.
LibEMP makes no assumptions about the application specific monitors, and they can be supervisory trees themselves, additionally, as we will see in the next section, they could be simple wrappers around some external (outside the Erlang VM) service or application.
Monitor crashes are handled on a one_for_one basis and will just restart the Monitor. Sinks on the other hand are potentially more complicated due to the semantic guarantees needed by the system on it's event processing (i.e. recall "at-most-once", "at-least-once"). The supervisory tree here therefore is much more mailable to crash handling.
As LibEMP is just an orchestration layer, we want to make it fairly simple to use it as such, and not be tied to an ecosystem that isn't right for your application. The application may also not have a choice as to what language/system/architecture it uses, or even have access to it (in the case of external services).
LibEMP thus is being tooled with a level of indirection in mind for plugging in components using (TCP/UDP/ZeroMQ) Sockets and Erlang's Native Interfaces. These plugs will provide standard RPC mechanisms for triggering actions on the Component well as provide the standard LibEMP functionality such as pushing events to a Buffer or pulling from it.
As the image shows, we are targeting Plug-In functionality for Monitors first, but Sinks and Buffers will follow shortly after. However, much of their functionality can be temporarily implemented using a Distributed LibEMP Buffer (as we see in the next section).
Distributed Computing in Erlang opens up a number of concerns (including Security issues). Rather than rely on Erlang's implementation and tie ourselves into a language implementation, we enforce (or suggest really) the following: Distributed Buffers.
Buffers are what implement the storage mechanism of Events. These events do not have to be stored on the local machine, and for extreme fault-tolerance purposes, this could be seen as a horrible idea. Thus, you can implement a distributed buffer either using Erlang's built-in Mnesia DB or a service such as AWS SQS or host a distributed queue (of which there are many possibilities).
In addition to this, we see how LibEMP can break up its event capturing from its event processing. A second LibEMP Node could be started without any Monitors and just a set of Sinks. This Event Processing style node could then be initialized on multiple computers to become a fleet of Event Processing Engine. Event Capture Nodes can then be light weight and loaded onto embedded platforms using a single Monitor for its purpose and a distributed Buffer implementation.
This layout is up to the discretion of the Application designer, but LibEMP's goal is to make those designs easier by implementing a test bed, fault-tolerant application initializer, and simple orchestration framework right out of the gate.
