How-to: Analyzing Out-Of-Memory issues in WebLogic 10.3.3 with JRockit 4.0 Flight Recorder

Oracle WebLogic Server 10.3.3 provides out-of-the box support for JRockit Flight Recorder (JFR); the new enhanced run-time JVM analyzer in JRockit 4.0 positioned as the replacement for JRA with the following points of improvement Always on, Better data, third-party application integration through an API and low-to-zero overhead. JFR integrates seamlessly with WLS 10.3.3 to produce recording images on demand or event-based to analyze and solve all kinds of JVM issues.

In this blog posting, I show how to capture automatically an overall WLS system image, including a JFR image, after an out-of-memory (OOM) exception has occured in the JVM hosting WLS 10.3.3.

Setting up WLS Diagnostic framework
To enable event generation by WLS for JFR, the Weblogic Diagnostic Volume property has to be set to the value low, medium or high indicating the amount of recorded events. The
Diagnostic Volume can be set in the WLS Administration console -> Environment -> servers -> YourServer.



Now we have started event generation by WLS for JFR, we have to configure a WLS Diagnostic system module with a watch rule and a notification so that image capturing is triggered whenever an OOM error happens. The image capturing mechanism captures the WLS system state together with the JFR buffered event data and generates a zip file in containing the JFR file in the image folder. The image folder is specified in the WLS adminstation console -> Diagnostics -> Diagnostic images -> YourServer



Go in the WLS Administration console to WLS adminstation console -> Diagnostics -> Diagnostic modules and create a new diagnostic module. I have called it JRFDiagnosticeModule. Click on the created diagnostic module and target it to the designated server (tab Targets). Go back to configuration tab and click on the Watches and Notifications tab to create a watch rule and a notification with following specs:

Watch rule

• Type = Server Log


• Expression = (MESSAGE LIKE '%OutOfMemoryError%')


• Use an automatic alarm so that the rule is re-enabled each time it is triggered after a defined period
  

Notification

• Type = Diagnostic Image

Make sure everything is enabled and the notification is associated with the watch. Leave every other setting to their defaults.

This is all we have to do in WLS 10.3.3. Before we proceed with triggering a OOM with a sample application, we first start the JRockit Mission Control (JRMC) application to verify that the JFR recording has been started. Execute the jrmc file in the folder /bin folder to start JRMC. Open the JVM browser and right-click on the WLS jvm and select view reports. In the lower-right panel you'll see that one recording has been started:



Generate an OOM error
To generate an OOM I've created a simple web application constisting of a simple page with a button that triggers a servlet that will execute the following code-snippet to generate an OOM in WLS 10.3.3.

 List list = new ArrayList();

        while(true){

          list.add("test string");

Deploy the web application to the WLS server and trigger the OOM by pressing the button on the page. After a few seconds you'll see this in console logging:



The console logging shows that the watch rule has been triggered and a image capture has been generated in the folder /servers/AdminServer/logs/diagnostic_image. Unzip the file and open the JRockitFlightRecorder.jfr file in JRMC.

In the JRMC console you are now able to analyze the root cause of the problem. For this obvious OO example (you can also get the root cause from the console output..but the intention here is to show the capabilities of JFR in general), you can have a look at the allocation tab in the Memory panel to drill down to class that causes the String object creation (It's just an example and the JFR contains a lot more information about Threads/CPU utilization and GC executions for example):




Also the Hot Method tab in the Code panel shows the servlet doPost method as a top listed hot method:



This simple and obvious example shows how easy it is to let WLS diagnostic framework continuously produce monitoring data for JFR that can be dumped to a JFR image when required, e.g. in case of OOM exception or other events. The default WLS diagnostic framework can be configured to collect a specific amount of events by using the coarse-grained Diagnostic Volume property. If you want extra events to JFR image you can start extra Recording by using the JRMC or using the command line. Also it is possible to use Java startup parameter -XX:+|-FlightRecordingDumpOnUnhandledException to trigger a JFR dump after a unhandled exception in the JVM.

Documentation:

• JRockit Flight Recorder run-time guide
  
• WebLogic Server Diagnostic Framework guide

How-to: Building REST/JSON services with OSB 11g and JAX-RS

Last couple of weeks, I have read a lot of blog postings about REST services in combination with the Oracle Service Bus (like this posting). I've even made a first attempt to write a posting about this subject, but I got really constructive comments in return on that posting that I decided to write a complete new one. Thanks for the comments :)

I decided to play around with REST and OSB myself and choosed for JAX-RS (Jersey) as the Java technology to build my REST service and the brand new OSB 11g release for proxing this REST service. In this posting, I will show you how easy it is to build a REST/JSON service with the feature rich and highly flexible JAX-RS standard, deploy it to WebLogic 10.3.3 and proxy it with an OSB 11g service.

REST service in JAX-RS
Required libraries (can be downloaded from here):

• jersey-bundle-1.1.5.1.jar


• jsr113-api-1.1.jar


• asm-3.1.jar

I build in JDeveloper a very simple REST product service with a single method to find a product by its id. The REST method returns a JSON representation of the product.

First you have to create a new web application project in JDeveloper, because the JAX-RS product service is deployed as a web application to the WebLogic Server. In the web.xml file the following things has to be specified:

• the JAX-RS servlet which will handle all requests and forward them to the appropriate REST Service class.


• The context path to access the servlet


• The mime type application/json

The web.xml has the following content:

         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"  
         version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">  
    web.xml for 
      
        10  
      
      
        html  
        text/html  
      
      
        json  
        application/json  
          
      
        txt  
        text/plain  
      
      
        JAX-RS Servlet  
        jersey-servlet  
        com.sun.jersey.spi.container.servlet.ServletContainer
          
             com.sun.jersey.config.property.resourceConfigClass  
             com.sun.jersey.api.core.PackagesResourceConfig  
           
           
             com.sun.jersey.config.property.packages  
             nl.oracle.com.fmw.rest;nl.oracle.com.fmw.rest.model  
                 
      
      
        jersey-servlet  
        /*  
      
  

Now the web.xml file is in place, two Java classes have to be implemented:

• ProductResource


• Product

The ProductResource class will contain the method to find a product by its id:

package nl.oracle.com.fmw.rest;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;

import nl.oracle.com.fmw.rest.model.Product;

@Path("/products")
public class ProductResource {

@GET 
@Path("{id}")
@Produces("application/json")
public Product getProductById(@PathParam("id") int id){  
     //Return a simple new product with the provided id 
     return new Product(id, "DummyProduct");
    }
}

The code listing above shows that that getProductById method is only accessible through the HTTP GET method. Also the use of the @Path annotation on the class and method level makes it possible to set a specific relative-uri with which the service can be accessed. The {id} serves as a placeholder for the product id and can be accessed through @PathParam annotation

The Product class is shown in the following code listing and uses the JAXB XmlRootElement binding annotation to automatically map the class structure to a JSON structure. Isn't that cool :). More JSON serialization and deserialization options in JAX-RS can be found here

package nl.oracle.com.fmw.rest.model;

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class Product {
    
    private int id;
    private String name;
    
    public Product(){
        
    }
    
    public Product(int id, String name) {
      this.id=id;
      this.name=name;
    }

    public void setId(int id) {
        this.id = id;
    }

    public int getId() {
        return id;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }
}

Coding is completed and it should be clear now that the use of annotations makes it very easy and flexible to create REST services with JAX-RS. Create a WAR deployment descriptor and make sure you add the jersey libraries to the WAR file and set the JEE web context root to services. Finally, deploy the WAR file to the WebLogic 10.3.3 server. I target the application to the osb_server1 managed server, which hosts my OSB 11g installation.

After deployment you can test the REST service using this URL where I use 1 as the id:

http://localhost:port/services/products/1

OSB 11g proxy
I used the brand new OSB 11g installation for creating a proxy service for my REST service.

The implementation is fairly simple and straight forward and follows more or less the same steps used in this excellent posting:

Business Service
The business service invokes the REST product service. Make sure you use the messaging service type with the HTTP transport protocol. Also set the HTTP method to GET.

Proxy Service
Create a proxy service in Eclipse and use the messaging service type with the HTTP transport protocol. I used request type none and Response type text. In the message flow I added a routing action to invoke the business service.
I need to mention two important things about the my Proxy service implementation

• Set the endpoint /osb-services/products. This enables you to add anything to this context path, for example /{id}. This makes it also possible to use the OSB service as a proxy for different types of REST calls (make sure you can switch between HTTP methods in the message flow) to the same product service


• Use the transport request element from the $inbound variable to append the relative path after /osb-services/products to the REST service endpoint. I've used an insert operation for this:

Deploy the OSB service to the OSB 11g server and use the test console to test the service. Make sure you set the relative-uri attribute in the test console transport panel:



The $body element in the response message should look like this:



The OSB service just passes the JSON response forward to the client. It is fairly simple to convert the JSON output to XML and vice versa using JSON lib in a Java service callout. How to do this is described in this posting.

Also the JSON structure returned by the product service is very simple. In most cases you have to do more work with JAX-RS in order to construct the required JSON structure. The JAX-RS libraries contain options for configuring JSON