Monitoring message flow performance with the WebSphere Message Broker Explorer Accounting and Statistics facility

Introduction

IBM® WebSphere® Message Broker (hereafter called Message Broker) provides a new Accounting and Statistics facility to help you understand the processing profile of a broker or execution group at the message flow, node, or terminal level.
Accounting and statistics data is available over periods as short as 20 seconds and as long as 240 hours. This data can help you determine message flow performance and resource usage. Snapshot data is particularly useful for problem determination -- for example, when determining why a flow is slower or consumes more CPU time than expected. WebSphere Message Broker Explorer now has a graphical viewer for this data, which makes the data much easier to use and lets you quickly assess the performance of a message flow to catch poorly written code, slow calls to subsystems, poor database response, and so on. The Accounting and Statistics facility enables message flow developers and testers to quickly get an assessment of message flow performance in a running broker. It is aimed at short-term problem determination and is not intended to be a long-term performance monitor.

Accounting and Statistics options

WebSphere Message Broker Accounting and Statistics provides two types of data: snapshot and archive. Snapshot data is collected for an interval of about 20 seconds -- the exact interval depends on the level of activity in the system and in Message Broker. At the end of an interval the data is written to the destination and the next interval is started. Archive data is collected over a longer period, which can vary from 10 to 14400 minutes (240 hours).
When requesting collection of accounting and statistics data, you can specify where to send the data:

• Publish the data using the Publish/Subscribe facility. The data is published as an XML message and is available to subscribing applications that register with the appropriate topic.
• Write the data to the user trace log. The data is written even if trace is turned off.
• Write the data to SMF (z/OS only). WebSphere Message Broker V5 writes accounting and statistics data as Type 117 records, which can be processed by any tool or facility capable of processing SMF records.

Collecting Accounting and Statistics data is optional. By default it is switched off -- you must specifically request it at the message flow, execution group, or message broker level. You can start or stop collection dynamically using the mqsichangeflowstats command.
Although the data provided in Accounting and Statistics contains a lot of useful information, it is not easily human readable, so a new facility has been added to the Message Broker Explorer to make it easier to use. For detailed information on how accounting and statistics data is collected and formatted when not using the Broker Explorer, see the WebSphere Message Broker information center.

Using the Message Broker Explorer

The Message Broker Explorer provides an easy and convenient way to collect and present snapshot accounting and statistics data on all platforms on which Message Broker runs. The Broker Explorer is shipped as WebSphere Message Broker SupportPac ISO2.
A subscription request to receive accounting and statistics data is issued by Broker Explorer when you request the collection of accounting and statistics data. You do not need to manage the subscription explicitly. After the snapshot period has elapsed, data is presented in both graphical and tabular form. You can view the data at the message flow, thread, or node level and you can sort the columns in the table to order the data by number of invocations, CPU time used, or other parameters.

Getting started with Accounting and Statistics

You first need to install Message Broker Explorer if you have not already done so, and then read the ReadMe file shipped with it for instructions on how to configure your MQ Queue Manager to support JMS. To start using the Accounting and Statistics facility in Message Broker Explorer, open a new view: selecting Window => Show View in the toolbar menu and select the Broker Statistics view:

Figure 1. Opening the Message Broker Statistics view

Selecting the Broker Statistics option will open two new windows in Message Broker Explorer: the Broker Statistics window, which consists of a table showing the collected data and the Statistics Graph, which presents the data as a graph. At first, both windows will be grayed out and say "Waiting for data." When this view is opened, Message Broker Explorer will attempt to connect to all brokers in the workspace. An error message will be produced for any broker that Message Broker Explorer cannot connect to. If you do get connection problems, check the access permissions to the Message Broker server and WebSphere MQ Queue Manager as well as the JMS setup.
To start collecting data, select a message flow in the Navigator window, right-click, and select Statistics => Start Statistics. Message Broker Explorer will then arrange for the collection of thread, flow, and node level snapshot accounting and statistics data for this flow. To stop collecting data, right-click on the message flow and select Statistics => Stop Statistics:

Figure 2. Starting and stopping the collection of Accounting and Statistics data

At the end of a snapshot period, Message Broker Explorer will automatically update the current view to present this data. The screen can be paused using the Pause button above the graph window, as shown in Figure 3 below. The pause button stops further updates to the screen as more data is obtained at the end of subsequent snapshot periods. After you release the pause button, data will once again be automatically updated as each snapshot period expires.

Figure 3. Pause button

While the display is paused, you can move between different screens presenting different views of the same set of data, which lets you see activity at a message flow level or individual nodes within a given message flow:

Figure 4. Activity at the message flow level

Understanding the Message Broker Explorer Accounting and Statistics data

The Accounting and Statistics information presented in Message Broker Explorer is spread over three panes, as shown in Figure 4 above. The Navigator pane on the left lets you select an individual Message Broker artifact (the whole broker, an execution group, or an individual message flow) for which you wish to start or stop collecting statistics.
Selecting a broker will display thread statistics for that broker. Selecting an execution group will show statistics for all message flows within it. Selecting an individual message flow will show node statistics for the nodes within that message flow. For more information on thread, flow, and node statistics see the WebSphere Message Broker information center.
The bottom right pane shows the Accounting and Statistics data as numeric values. Data is presented as a series of rows and columns. The meaning of a row and the columns associated with it depends on the level of component being displayed. You can sort on any of the columns by clicking on the column header, and you can resize the width of any column as needed by dragging the edge of the column header.
The top right pane shows the currently selected rows from the table view in graphical form. You can dynamically change the selection of data in the table view by pressing the Ctrl key and clicking rows to add or remove them from the selection. Alternatively, you can use the Shift key to select a range of rows to graph.

Selecting columns

To prevent the graph window from becoming cluttered, not all numeric values are shown at the same time in the graph. You can change the default values that are presented by clicking Filter above the graph and then selecting from the list of data values that are available to be graphed. The values available depend on the type of broker object selected in the Navigator window. Figure 5 below shows the data Values for flow data:

Figure 5. Data values for flow data

The selection of data values in the Filter window is remembered for each type of broker object, and restored when the same type is reopened.

Selecting graphing options

There are three types of graph available: Stacked, Linear, and Logarithmic, to let you choose how values are compared:

• The Stacked view gives a good indication of the relative size of the same data value for different instances of the same object type.
• The Linear view gives an accurate comparison of similar-sized numbers.
• The Logarithmic view helps you make comparisons where there is a wide range of data values. Figures 6, 7, and 8 show the Stacked, Linear, and Logarithmic views:

Figure 6. Stacked graph of node-level data

Figure 7. Linear graph of thread-level data

Figure 8. Logarithmic graph of flow-level data

Usage tips

To obtain accounting and statistics data, you will need to process messages within a message flow after installation and configuration are complete. If you have your own tools to generate input messages, you may want to use those, such as:

• The Test Client provided with the Message Broker toolkit in WebSphere Message Broker V6.0.2 or later.
• The WebSphere Message Broker V6 Message display, test, and performance utilities provided in SupportPac IH03.
• The IBM Performance Harness For JMS, which allows load generation for WebSphere MQ and JMS message flows.

All of these tools are worth investigating. The Test Client and the WebSphere Message Broker V6 Message display, test, and performance utilities make it easy to send a small number of messages. The display, test, and performance utilities, and the IBM Performance Harness For JMS are both good for generating a large volume of messages.
After you get used to generating statistics, you may wish to run a message flow and then a modified version of it, and compare the different processing profiles. You should process more than one message at a time when collecting and viewing accounting and statistics data. Run tens, hundreds, or even thousands of messages to produce a processing time covering more than one reporting period. The number of messages needed in any situation depends on the complexity of the message flow being monitored. When comparing performance, use the same set of input messages, and process the same number of messages each time to get a valid comparison.

Performance overhead

Collecting accounting and statistics data from a broker affects its performance, so you may want to restrict its use to development and test systems. The performance overhead depends on the level of data collected and the complexity of the message flow. For example, running the Large Messaging Sample shipped with WebSphere Message Broker V6 with basic thread level and advanced node level accounting activated resulted in an 8% processing overhead in one test. The processing cost associated with the presentation and graphing of data is incurred on the same machine as the toolkit in which the data is displayed, which may be the same machine that the broker is running on or not, dependent on the configuration. This overhead is negligible.
WebSphere Message Broker Explorer cannot determine whether Accounting and Statistics is enabled or not, so you should ensure that it is turned off when you have finished analyzing message flow performance. The only way to check whether Accounting and Statistics is turned on is by invoking the mqsichangeflowstats command on the broker machine. For more information on this command, see the WebSphere Message Broker information center.

Limitations

Currently you cannot change the type of statistics that are collected, the data collection interval, or the level of data collected. Nor can you save results to file.
Connection to the broker currently requires that the local user that is used with the toolkit also exist on the server machine, and that this user has access to the broker’s WebSphere MQ Queue Manager.

Conclusion

The Accounting and Statistics data collection and viewing facility of WebSphere Message Broker Explorer provides an excellent way to view message flow performance on both local and remote brokers. Its snapshot statistics let you determine in real-time the resource usage profile in any message flow running in a broker. You can choose graphical or tabular presentation of the data, as well as how much data is displayed. By sorting tabular data, you can quickly identify the message flow or node consuming the most CPU resources or having the longest elapsed time, and you can then focus your performance improvement efforts where they will do the most good.
This new feature is highly recommended to message flow developers and testers to speed diagnosis and resolution of performance problems.