Ads Header

Monday, September 8, 2014

IBM Integration Bus and WebSphere Message Broker FAQ for Memory

Technote (FAQ)


Question

The following is a list of answers to frequently asked questions (FAQ) about Memory related problems in IBM Integration Bus (IIB) and WebSphere Message Broker (WMB) for new and experienced users

Answer

Index of questions in this document:


1. Is there a memory leak in my DataFlowEngine?

During message processing, it is possible for execution groups to continue to grow in memory usage. This could be due to a number of factors and does not necessarily indicate that there is a memory leak. The following link provides information on the Execution group memory growing behavior and the reasons.
Back to top


2. Why is my DataFlowEngine memory usage not reducing on the UNIX systems?

DataFlowEngine acquires memory while processing messages through its message flows. The amount of memory required to process messages depends on the size of the messages. After some processing, the DataFlowEngine acquires enough memory that it can internally re-use the memory for all the subsequent messages that are processed through its message flows. The UNIX process holds on to the memory anticipating that it could be required for subsequent processing of messages and to avoid performance impact due to releasing and reacquiring memory.
Back to top
 


3. How can I monitor JVM memory usage of a DataFlowEngine process?

JVM memory usage by an execution group can be monitored using resource statistics. The following Information center page gives more information on this.
Use MBX explorer and/or WebUI to enable and view resource statistics
- Register subscriptions and use command line parameters to monitor the resource statistics
http://pic.dhe.ibm.com/infocenter/wmbhelp/v9r0m0/topic/com.ibm.etools.mft.doc/bj43320_.htm
- Export the following env variable in the broker service Id's profile and restart the broker to monitor the garbage collection (GC) in broker -
export MQSIJVERBOSE=-verbose:gc
OR
- You may run the following command at the execution group level and restart just the execution group -
mqsichangeproperties <broker> -e <EG> -o ComIbmJVMManager -n jvmSystemProperty -v"-verbose:gc -Xverbosegclog:<path to gc.trc>
Once the output is collected, you can use the IBM Support Assistant (ISA) workbench, a GUI based application to view the GC activity in graphical form. The ISA workbench can be downloaded from the following IBM site:
http://www.ibm.com/software/support/isa/download.html
After download, you can go to Update > Find updates and add ons to install "IBM Monitoring and Diagnostic Tools for Java - Garbage collection and Memory Visualizer". This will install the tool that can now be used to view the garbage collection activity.  You can use this tool to analyze the stdout file.
Back to top



4. How can I tune JVM heapsize for a DataFlowEngine, bipbroker and biphttplistener processes?


JVM maximum and minimum heapsize can be tuned using mqsichangeproperties command. The following online documentation gives more information
Please see the following link for bipbroker and DataFlowEngine processes:
http://www-01.ibm.com/support/docview.wss?uid=swg21198739
Back to top



5. How can I monitor memory usage on Windows and UNIX?


At any given point, you can check the memory usage for processes in the following way:
Windows:
Ctrl-Alt-Delete > Start Task Manager > Processes > Show processes for all users and go to the process "DataFlowEngine" and look at the field "Memory (Private working set)
If you want to continuously monitor the memory usage, then check the following link for Windows sysinternals for process utilities: http://technet.microsoft.com/en-us/sysinternals/
UNIX:
ps -aelf | grep <PID for DataFlowEngine>
If you want to continuously monitor the memory usage, then the above command may have to be incorporated into a simple shell script.
Back to top
 


6. I see an abend with an out of memory error. What could be causing this failure?

An out of memory error in an abend file indicates the process has terminated as it could not acquire more memory to continue processing. The memory required for processing could be native memory or java heap memory. Investigation would be required to identify why the process was unable to acquire more memory. If using java compute nodes, it is a good practice to tune JVM Heapsize for the environment. Refer to the questions below to gain better understanding of this subject to be able to carry out the investigation.
Back to top



7. How do I isolate the problem of memory growth to a particular message flow?


Separating the message flows that are deployed to an Execution group into multiple Execution Groups would allow you to recreate the problem with isolated sets of message flows deployed to their own execution groups and thus allow you to identify the specific message flows that may be contributing towards memory growth for that execution group.
To identify a potential memory leak, the same input message should be repeatedly sent into the message flow. When a message flow processes the same input message over and over again, then this should drive the same message flow paths each time. When this happens then we would expect the DataFlowEngine process to plateau very quickly unless the flow had experienced an expected growth.

Back to top


8. How does WebSphere Message Broker (WMB) handle the memory usage within a DataFlowEngine (DFE) Process?

When considering memory usage within a DataFlowEngine process there are two sources that the storage is allocated from, and these are :
1. The DataFlowEngine main memory heap
2. The operating system
When message flow processing requires some storage, then an attempt is first made to allocate the memory block required from the DataFlowEngine's heap. If there is not a large enough contiguous block of storage on the heap, then a request will be made to the operating system to allocate more storage to the DataFlowEngine for the message flow. Once this is done, then this would lead to the DataFlowEngine's heap growing with the additional storage, and the message flow will use this extra storage.
When the message flow has completed its processing, then it issues a "free" on all its storage and these blocks will be returned to the DataFlowEngine's heap ready for allocation to any other message flows of this DataFlowEngine. The storage is never released back to the operating system, because there is actually no programmatic mechanism to perform such an operation. The operating system will not retrieve storage from a process until the process is terminated. Therefore the user will never see the size of the DataFlowEngine process decrease, after it has increased.
When the next message flow runs, then it will make requests for storage, and these will then be allocated from the DataFlowEngine heap as before. Therefore there will be a re-use within the DataFlowEngine's internal storage where possible, minimizing the number of times that the operating system needs to allocate additional storage to the DataFlowEngine process. This would mean there may be some growth observed on DataFlowEngine's memory usage which is of the size of the subsequent allocations for message flow processing. Eventually we would expect the storage usage to plateau, and this situation would occur when the DataFlowEngine has a large enough heap such that any storage request can be satisfied without having to request more from the operating system.

Back to top


9. What is memory fragmentation in a DataFlowEngine process?

As explained in the above question, at the end of each message flow iteration, storage is freed back to the DataFlowEngine memory heap ready for re-use by other threads. However, there are objects that are created within the DataFlowEngine that last the life of the DataFlowEngine and therefore reside at that point in the heap for that time. This leads to what is known as fragmentation and as a result reduces the size of contiguous storage blocks available in the DataFlowEngine when an allocation request is made. This means that DataFlowEngine process has the memory blocks for allocation but are fragmented to be allocated to requests made during message processing. In most of the cases, the requesters of storage require a contiguous chain of blocks in memory. Therefore, it is possible for a message flow to make a request for storage against the DataFlowEngine's heap that does not have enough free storage to satisfy the request for this contiguous chain of blocks, but the storage is fragmented, such that the contiguous block does not fit into any of the "gaps". In this situation, a request would have to be made to the operating system to allocate more storage to the DataFlowEngine so that this block can be allocated.
However, when unfreed blocks remain on the DataFlowEngine's heap then this will fragment the heap. This means that there will be smaller contiguous blocks available on the DataFlowEngine's heap. If the next storage allocation cannot fit into the fragmented space, then this will cause the DataFlowEngine's memory heap to grow to accommodate the new request.
This is why small increments may be seen in the DataFlowEngine even after it has processed thousands of messages. In a multi-threaded environment there will be potentially many threads requesting storage at the same time, meaning that it is more difficult for a large block of storage to be allocated.
For example,
Some message flows implement BLOB domain processing which may result in the concatenating of BLOBs. Depending on how the message flow has been written, this may lead to fragmentation of the message heap due to the fact that when a binary operation takes place such as concatenation, both the source and target variables need to be in scope at the same time.
Consider a message flow that reads in a 1MB BLOB and assigns this to the BLOB domain. For the purposes of demonstration, this ESQL will show a WHILE loop that causes the repeated concatenation of this 1MB BLOB to produce a 57MB output message. Consider the following ESQL :
DECLARE c, d CHAR;
SET c = CAST(InputRoot.BLOB.BLOB AS CHAR CCSID InputProperties.CodedCharSetId);
SET d = c;

DECLARE i INT 1;
WHILE (i <= 56) DO
  SET c = c || d;
  SET i = i + 1;
END WHILE;

SET OutputRoot.BLOB.BLOB = CAST(c AS BLOB CCSID InputProperties.CodedCharSetId);
As can be seen, the 1MB input message is assigned to a variable c and then this is also copied to d. The loop then concatenates c to d and assigns the result back to c on iteration. This means that c will grow by 1MB on every iteration. Since this processing generates a 57MB blob, one may expect the message flow to use around 130MB of storage. The main aspects of this are the ~60MB of variables in the compute node, and then 57MB in the Output BLOB parser which will be serialised on the MQOutput node.
However this is not the case. This ESQL will actually cause a significant growth in the DFE's storage usage due to the nature of the processing. This ESQL encourages fragmentation in the memory heap. This condition means that the memory heap has enough free space on the current heap, but has no contiguous blocks that are large enough to satisfy the current request. When dealing with BLOB or CHAR Scalar variables in ESQL, these values need to be held in contiguous buffers in memory.
Therefore when the ESQL SET c = c || d; is executed, in memory terms this is not just a case of appending the value of d, to the current memory location of c. The concatenation operator takes two operands and then assigns the result to another variable, and in this case this just happens to be one of the input parameters. So logically the concatenation operator could be written SET c = concatenate(c,d). This is not valid syntax but is being used to illustrate that this operator is like any other binary operand function. The value contained in c cannot be deleted until the operation is complete since c is used on input. Furthermore, the result of the operation needs to be contained in temporary storage before it can be assigned to c.

Back to top


10. How can I tell if the out of memory is caused due to the DataFlowEngine running out of JVM Heapsize or total process memory?


- When a DataFlowEngine reaches the JVM heap limitations, it typically generates a javacore, heapdump along with a java out of memory exception in the EG stderr/stdout files.
When the DataFlowEngine runs out of total memory, it may cause the DataFlowEngine to go down or the system to become un-responsive.
 
Back to top


11. Are there any useful references in the area of broker/execution group memory usage?

See the following references:

Back to top


12. What should I do in a broker to process a very large message?

See the following references:

Back to top


13. What is MQSI_THREAD_STACK_SIZE and do I need to set it?

MQSI_THREAD_STACK_SIZE
Purpose : For any given message flow, a typical node requires about 2KB of the thread stack space. Therefore, by default, there is a limit of approximately 500 nodes within a single message flow on the UNIX platform and 1000 nodes on the Windows platform. This limit might be higher or lower, depending on the type of processing being performed within each node. If a message flow of a larger magnitude is required, one can increase this limit by setting the MQSI_THREAD_STACK_SIZE environment variable to an appropriate value( broker must be restarted for the variable to be effective). This environment variable setting applies to brokers, therefore the MQSI_THREAD_STACK_SIZE is used for every thread that is created within a DataFlowEngine process. If the execution group has many message flows assigned to it, and a large MQSI_THREAD_STACK_SIZE is set, this can lead to the DataFlowEngine process requiring a large amount of storage for the stack. In WMB, it is not just execution of nodes that can cause a build up on a finite stack size. It follows from the same principles for any processing that leads to a large amount of nested or recursive processing and might cause extensive usage of the stack. Therefore, you may need to increase the MQSI_THREAD_STACK_SIZE environment variable in the following situations: a) When processing a large message that has a large number of repetitions or nesting. b) When executing ESQL that recursively calls the same procedure or function. This can also apply to operators. For example, if the concatenation operator was used a large number of times in one ESQL statement, this could lead to a large stack build up.
However, it should be noted that this environment variable applies to all the message flow threads in all the execution groups, as it is set at the broker level. For example, if there are 30 message flows and this environment variable is set to 2MB then that would mean that 60MB would be reserved for just stack processing and thus taken away from the DataFlowEngine memory heap. This could have an adverse effect on the execution group rather than yielding any benefits. Typically, the default of 1 MB is sufficient for most of the scenarios. Therefore we would advise that this environment variable NOT be set unless absolutely necessary.
Please refer to the following link for more information:
http://pic.dhe.ibm.com/infocenter/wmbhelp/v9r0m0/topic/com.ibm.etools.mft.doc/ac55020_.htm  

Back to top
 


14. Are there any needs for modifying the system kernel parameters for WMB/IIB?

In WMB/IIB there are no suggested kernel settings for the tuning of an operating system. However, the WebSphere MQ and some database products do, and WMB/IIB runs under the same environment as these. Hence, its best to check and tune your environment as guided by these applications.
Back to top
 


15. How do you set the limit of total memory consumed by the DataFlowEngine process?

There is no functionality within the Broker product to set a maximum memory limit on an execution groups. Operating systems may offer such functionality and the user should look into the ways of doing this on HP-UX. However, it should be noted, if a DataFlowEngine limit is set to a value that it needs to exceed, then it will terminate when the request for storage is refused.
Back to top
Cross reference information
Segment Product Component Platform Version Edition
Business Integration WebSphere Message Broker High CPU/Memory growth/System related issues AIX, HP-UX, Linux, Solaris, Windows 7.0, 8.0 All Editions

Product Alias/Synonym

WMB MB WebSphere Message Broker IBM Integration Bus IIB IBMIB MQ Integrator WBIMB WBI-MB MQSI WMQI
Posted by at
Labels:

1 comment:

  1. UnknownSeptember 4, 2019 at 11:11 PM

    Thank you so much for the post you do. I like your post and all you share with us is up to date and quite informative, i would like to bookmark the page so i can come here again to read you, as you have done a wonderful job. south africa brokers forex

    ReplyDelete

Subscribe to: Post Comments (Atom)