SObjectizer
5.5
|
Until v.5.5.4 SObjectizer Run-Time was a "black-box". There was no possibility to look inside it and ask for some run-time statistics. How many agents are registered? How many demands in an event queue? How many delayed messages waiting? There were no answers to those questions.
That was not a problem for most cases because application level monitoring is much more useful than low-level monitoring of SObjectizer's internals. For example the collecting statistics about time spent on DB-related operation can give you more precise information about problems in your doman-specific code than quantity of delayed messages in SObjectizer's timer thread.
But it is a big plus if a framework like SObjectizer can provide some kind of run-time statistics. It could be used for debugging purposes for example. Or for collecting statistics by the tools like statsd and graphite and viewing/analyzing it conjunctions with other information about your application.
Because of that a very simple mechanism for collecting and distributing of run-time monitoring information is implemented in v.5.5.4. Please note that it is a very first step in implementation of full-fledged run-time monitoring solution for SObjectizer. Version 5.5.4 provides basic statistics about SObjectizer's internals. There are some plans for extending such functionality in the future versions.
There are several new concepts related to run-time monitoring in v.5.5.4:
The stats_controller and data-sources are created by SObjectizer automatically, there is no need to do that manually.
The stats_controller can be turned on or off. By default it is turned off. In that mode no run-time information collected nor distributed. It means there is no any performance penalty when stats_controller doesn't work.
If the stats_controller is turned on it works on separate thread. This thread is created when the stats_controller is turned on and terminated when the stats_controller is turned off.
The stats_controller sleeps for some time. Then wakes up and tells every data-source object to distribute the information they currently have. The distribution if performed on the thread of stats_controller. When distribution finishes the stats_controller is going to sleep again.
The time the stats_controller sleeps can be adjusted. By default stats_controller distributes run-time statistics every 2 seconds. But is can be changed by so_5::rt::stats::controller_t::set_distribution_period() method.
Every instance of SObjectizer Environment has its own stats_controller instance. This instance can be accessed via so_5::rt::environment_t::stats_controller() method. For example:
Data-sources distribute their values by ordinary messages. Types of that messages are defined in so_5::rt::stats::messages namespace. Those messages are sent to mbox which can be obtained by so_5::rt::stats::controller_t::mbox() method.
There is only one message type in so_5::rt::stats::messages namespace in v.5.5.4: it is a template so_5::rt::stats::messages::quantity. This message is used to spread information about quantities of something (like quantity of registered cooperations/agents, quantity of single-shot or periodic timer messages, quantity of working threads for a dispatcher, size of an event queue and so on).
Only so_5::rt::stats::messages::quantity<std::size_t> type is used in v.5.5.4 for all types of run-time statistics. Other message types could be introduced in future versions.
Data-sources have unique names. Every name is built from two parts: prefix and suffix. Concatenation of prefix and suffix makes unique name for data-source.
Prefix is a size-limited null-terminated string. Prefix defines group of related data-sources. For example all data-sources related to a particular dispatcher instance will have the same prefix.
Suffix is a constant pointer to immutable null-terminated string. Suffix identified data-source inside a group of related data-sources.
For example here are some full names of data-sources:
mbox_repository/named_mbox.count: 0 coop_repository/coop.reg.count: 2 coop_repository/coop.dereg.count: 0 coop_repository/agent.count: 41 timer_thread/single_shot.count: 1 timer_thread/periodic.count: 0 disp/ot/DEFAULT/agent.count: 6 disp/ot/DEFAULT/wt-0/demands.count: 23 disp/ot/0x3be520/agent.count: 5 disp/ot/0x3be520/wt-0/demands.count: 19 disp/ao/0x24911b0/agent.count: 5
Substrings like mbox_repository
, disp/ot/DEFAULT
and disp/ao/0x24911b0
are prefixes. Substring like /named_mbox.count
, /agent.count
and demands.count
are suffixes.
A name of data-source is included in every message with actual data from that data-source. For example the so_5::rt::stats::messages::quantity is defined as:
Where quantity::m_prefix and quantity::m_suffix are parts of data-source name.
There are some predefined prefixes. They could be obtained by functions defined in so_5::rt::stats::prefixes namespace. But all the prefixes for dispatcher-related data-sources are generated at run-time.
All suffixes are predefined. They could be obtained by functions defined in so_5::rt::stats::suffixes namespace.
To receive run-time monitoring information it is necessary to do the two things:
Those can be done like this:
Please note that SObjectizer doesn't provide any ready-to-use tools for working with run-time statistics. If you want to store it anywhere you must write the appropriated code by youself.
Please note also that there is no possibility to narrow a stream of monitoring information. If you create subscription to quantity
message you will receive messages of that type from all data-sources. And if you want to narrow the stream you must filter those messages in your event:
There are several predefined data-sources in SObjectizer Environment. And there are some data-sources for every dispatcher and every dispatcher's working thread. This section briefly describes those data-sources.
There are three data-sources related to agents and cooperations repository. They all have prefix defined by so_5::rt::stats::prefixes::coop_repository() function but the suffixes for data-source are different:
There is one data-source that informs about the current quantity of named mboxes. This data-source has prefix so_5::rt::stats::prefixes::mbox_repository() and suffix so_5::rt::stats::suffixes::named_mbox_count().
There are two data-sources related to timer thread. They have prefix so_5::rt::stats::prefixes::timer_thread() and one of the following suffixes:
Every dispatcher creates several data-sources. Those data sources can be grouped in several groups. Every group will have its own prefix.
There is common rules for creation of prefixes for dispatchers' data-sources:
disp
;/ot
– for one_thread dispatcher;/ao
– for active_obj dispatcher;/ag
– for active_group dispatcher;/tp
– for thread_pool dispatcher;/atp
– for adv_thread_pool dispatcher;For example:
disp/ot/DEFAULT/agent.count disp/ot/DEFAULT/wt-0/demands.count disp/ot/0x3be520/agent.count disp/ot/0x3be520/wt-0/demands.count disp/ao/0x24911b0/agent.count disp/ao/0x24911b0/wt-0x2491260/demands.count disp/ag/0x2495930/group.count disp/ag/0x2495930/wt-group#1/agent.count disp/ot/stats_listener/wt-0/demands.count disp/tp/workers/aq/0x53acf0/demands.count
Note. If a dispatcher has been added to the environment as named dispatcher its name will be used for creation of data-source prefixes. A name for private dispatcher can be specified by using the appropriate create_private_disp
function.
one_thread dispatcher creates two data-sources:
disp/ot/<DISP-ID>
and suffix so_5::rt::stats::suffixes::agent_count(). It informs quantity of agents bound to that dispatcher;disp/ot/<DISP-ID>/wt-0
and suffix so_5::rt::stats::suffixes::work_thread_queue_size(). It informs event queue length.Example:
disp/ot/DEFAULT/agent.count: 6 disp/ot/DEFAULT/wt-0/demands.count: 23 disp/ot/0x3be520/agent.count: 5 disp/ot/0x3be520/wt-0/demands.count: 19
active_obj dispatcher creates two types of data-sources:
disp/ao/<DISP-ID>
and suffix so_5::rt::stats::suffixes::agent_count(). It informs quantity of agents bound to that dispatcher. This data-source is created always, even if there is no agents bound to the dispatcher;disp/ao/<DISP-ID>/wt-<THREAD>
and suffix so_5::rt::stats::suffixes::work_thread_queue_size(). It informs event queue length for a working thread (e.g. for one agent bound to the dispatcher).Example:
disp/ao/0x24911b0/agent.count: 5 disp/ao/0x24911b0/wt-0x2491260/demands.count: 2 disp/ao/0x24911b0/wt-0x2491d30/demands.count: 4 disp/ao/0x24911b0/wt-0x24931f0/demands.count: 2 disp/ao/0x24911b0/wt-0x24934b0/demands.count: 2 disp/ao/0x24911b0/wt-0x24955c0/demands.count: 0
active_group dispatcher creates several data-sources:
disp/ag/<DISP-ID>
and suffix so_5::rt::stats::suffixes::disp_active_group_count(). It informs quantity of active groups handled by that dispatcher. This data-source is created always, even if there is no agents bound to the dispatcher;disp/ag/<DISP-ID>/wt-<GROUP>
and one of the suffixes:Example:
disp/ag/0x2495930/group.count: 5 disp/ag/0x2495930/wt-db_handler/agent.count: 3 disp/ag/0x2495930/wt-db_handler/demands.count: 4 disp/ag/0x2495930/wt-logger/agent.count: 1 disp/ag/0x2495930/wt-logger/demands.count: 0
thread_pool/adv_thread_pool dispatchers create several data-sources:
disp/tp/<DISP-ID>
and suffix so_5::rt::stats::suffixes::disp_thread_count(). It informs about working thread count for the dispatcher;disp/tp/<DISP-ID>
and suffix so_5::rt::stats::suffixes::agent_count(). It informs about the quantity of agents bound to that dispatcher;disp/tp/<DISP-ID>/cq/<COOP>
. If FIFO queue is created for an individual agent then the corresponding data-sources will have prefix like disp/tp/<DISP-ID>/aq/<AGENT-PTR>
. The suffix for those data-sources can be one of the following:Note. For adv_thred_pool dispatcher all prefixes will start from disp/atp
.
Example:
disp/tp/0x24974b0/threads.count: 4 disp/tp/0x24974b0/agent.count: 5 disp/tp/0x24974b0/cq/db_ops/agent.count: 5 disp/tp/0x24974b0/cq/db_ops/demands.count: 9 disp/tp/0x2499ab0/threads.count: 4 disp/tp/0x2499ab0/agent.count: 5 disp/tp/0x2499ab0/aq/0x2495670/agent.count: 1 disp/tp/0x2499ab0/aq/0x2495670/demands.count: 4 disp/tp/0x2499ab0/aq/0x2495880/agent.count: 1 disp/tp/0x2499ab0/aq/0x2495880/demands.count: 1 disp/tp/0x2499ab0/aq/0x2495ca0/agent.count: 1 disp/tp/0x2499ab0/aq/0x2495ca0/demands.count: 0 disp/tp/0x2499ab0/aq/0x2495d50/agent.count: 1 disp/tp/0x2499ab0/aq/0x2495d50/demands.count: 1 disp/tp/0x2499ab0/aq/0x24962d0/agent.count: 1 disp/tp/0x2499ab0/aq/0x24962d0/demands.count: 2 disp/atp/0x24a3c10/threads.count: 4 disp/atp/0x24a3c10/agent.count: 5 disp/atp/0x24a3c10/cq/crypto_ops/agent.count: 5 disp/atp/0x24a3c10/cq/crypto_ops/demands.count: 15 disp/atp/0x24971a0/threads.count: 4 disp/atp/0x24971a0/agent.count: 2 disp/atp/0x24971a0/aq/0x24a7e10/agent.count: 1 disp/atp/0x24971a0/aq/0x24a7e10/demands.count: 4 disp/atp/0x24971a0/aq/0x24a8a70/agent.count: 1 disp/atp/0x24971a0/aq/0x24a8a70/demands.count: 4