Wayne Adams' Blog

Asynchronous Java SE Web Services: 1. Implementation

2012-01-04T22:00:00.000-08:00

A few years ago, I posted a how-to on Java-SE-based Web Services. More recently, I've become interested in asynchronous web-service invocation, and, as it turns out, Java SE supports that, too. This post, then, is the asynchronous version of that older post. How I got to the structure of this post is a story in itself. To make things simpler, I will first go through all the steps to deploy
an asynchronous Java SE web service. Then, I will explain the route I chose, and what I see as the positives and negatives of the results.

Here's the outline:

Create a WSDL definition file
Using the WSDL file, generate server artifacts; implement the web service operations
Create an external JAX-WS binding definitions file
Generate client-side artifacts using the WSDL and binding definitions files
Demonstrate synchronous and asynchronous client-side invocations of the web-service operations

Let's cut to the chase.

Create a WSDL definition file

Here I'll create a minimal WSDL file, describing a web service which returns the exchange
rate of two currencies. Here's the file, called exchange-rate.wsdl:

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
          xmlns:wsp="http://www.w3.org/ns/ws-policy"
          xmlns:wsp1_2="http://schemas.xmlsoap.org/ws/2004/09/policy"
          xmlns:wsam="http://www.w3.org/2007/05/addressing/metadata"
          xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
          xmlns:tns="http://async.ws.adamsresearch.com/"
          xmlns:xsd="http://www.w3.org/2001/XMLSchema"
          xmlns="http://schemas.xmlsoap.org/wsdl/"
          targetNamespace="http://async.ws.adamsresearch.com/"
          name="ExchangeRateService">
      
<types>
 <xsd:schema xmlns:tns="http://async.ws.adamsresearch.com/"
             xmlns:xs="http://www.w3.org/2001/XMLSchema"
             version="1.0"
             targetNamespace="http://async.ws.adamsresearch.com/">
   <xsd:element name="getExchangeRate"  type="tns:getExchangeRate"></xsd:element>
   <xsd:element name="getExchangeRateResponse"  type="tns:getExchangeRateResponse"></xsd:element>
   <xsd:complexType name="getExchangeRate">
     <xsd:sequence>
       <xsd:element name="arg0" type="xsd:string" minOccurs="0"></xsd:element>
       <xsd:element name="arg1" type="xsd:string" minOccurs="0"></xsd:element>
     </xsd:sequence>
   </xsd:complexType>
   <xsd:complexType name="getExchangeRateResponse">
     <xsd:sequence>
       <xsd:element name="return" type="xsd:double"></xsd:element>
     </xsd:sequence>
   </xsd:complexType>
 </xsd:schema>
</types>
<message name="getExchangeRate">
 <part name="parameters" element="tns:getExchangeRate"></part>
</message>
<message name="getExchangeRateResponse">
 <part name="parameters" element="tns:getExchangeRateResponse"></part>
</message>
<portType name="ExchangeRate">
 <operation name="getExchangeRate">
   <input wsam:Action="http://async.ws.adamsresearch.com/ExchangeRate/getExchangeRateRequest"
message="tns:getExchangeRate"></input>
   <output wsam:Action="http://async.ws.adamsresearch.com/ExchangeRate/getExchangeRateResponse"
message="tns:getExchangeRateResponse"></output>
 </operation>
</portType>

<binding name="ExchangeRatePortBinding" type="tns:ExchangeRate">
 <soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"></soap:binding>
 <operation name="getExchangeRate">
   <soap:operation soapAction=""></soap:operation>
   <input>
     <soap:body use="literal"></soap:body>
   </input>
   <output>
     <soap:body use="literal"></soap:body>
   </output>
 </operation>
</binding>

<service name="ExchangeRateService">
 <port name="ExchangeRatePort" binding="tns:ExchangeRatePortBinding">
   <soap:address location="http://localhost:8282/exchangeRate"></soap:address>
 </port>
</service>

</definitions>

The web service has a single operation -- getExchangeRate -- which takes a String representation of two currencies and returns a double. There's no need to flesh this service out with a lot of operations -- I just want to demonstrate the dev cycle required to deploy a web service with asynchronous operations.

Generate the service-side artifacts

Next, we'll create a Java web-service project using Maven. In a convenient directory, enter

mvn archetype:generate -DgroupId=com.adamsresearch.ws.async -DartifactId=AsyncService

(substituting your values, of course) and accept all the default values. This creates the skeleton of what will be our web service. Next, we'll modify the POM file to read the WSDL file and generate the service-side artifacts. For this, we will use the JAX-WS utility wsimport (found in the Java SE SDK; we will be using the Maven wsimport plugin, however). First, we have to decide where to put the WSDL file. We can put it anywhere we want, of course, but note from the jaxws:import docs page that the default location is ${basedir}/src/wsdl. I prefer to put mine in a directory src/main/resources/wsdl, so I'll need to specify that directory below, in the arguments to wsimport.

Open the POM file in the top-level directory of the project and add the following, after the dependencies element:


<build>
 <finalName>ExchangeRateWebService</finalName>
 <plugins>
   <plugin>
     <artifactId>maven-compiler-plugin</artifactId>
     <configuration>
       <source>1.6</source>
       <target>1.6</target>
     </configuration>
   </plugin>
   <plugin>
     <groupId>org.codehaus.mojo</groupId>
     <artifactId>jaxws-maven-plugin</artifactId>
     <executions>
       <execution>
         <goals>
           <goal>wsimport</goal>
         </goals>
         <configuration>
           <wsdlDirectory>${basedir}/src/main/resources/wsdl</wsdlDirectory>
           <keep>true</keep>
           <packageName>com.adamsresearch.ws.async.generated</packageName>
           <sourceDestDir>${basedir}/src/main/java</sourceDestDir>
         </configuration>
       </execution>
     </executions>
   </plugin>
 </plugins>
</build>

The first dependency is an acknowledgment that Maven defaults to 1.3. For those of you without white hair, that's a really old version of Java.

I've decided to override a number of wsimport defaults, to create the service in the desired package and to drop the files in the desired directory. Note that packageName does not have a default setting. Now enter mvn install from the project top-level directory.

I now have an interface -- ExchangeRate -- as well as some additional supporting classes. It is time to write our service implementation. Here is my first cut:


package com.adamsresearch.ws.async;

import javax.jws.WebMethod;
import javax.jws.WebService;
import javax.xml.ws.Endpoint;
import com.adamsresearch.ws.async.generated.ExchangeRate;

@WebService(serviceName="ExchangeRateService", portName="ExchangeRatePort", endpointInterface="com.adamsresearch.ws.async.generated.ExchangeRate")
public class ExchangeRateEndpoint implements ExchangeRate
{
 public static void main(String[] args)
 {
     if (args.length != 1)
     {
         System.out.println("Usage: java -cp <jarFile> com.adamsresearch.ws.async.ExchangeRateEndpoint publishURL");
         System.exit(-1);
     }
     ExchangeRateEndpoint wsInstance = new ExchangeRateEndpoint();
     Endpoint.publish(args[0], wsInstance);
     System.out.println("Published endpoint at URL " + args[0]);
 }

 @WebMethod
 public double getExchangeRate(String fromCurrency, String toCurrency)
 {
   if (fromCurrency.equals("AS1") && toCurrency.equals("GMD"))
   {
     return 2.78;
   }
   else
   {
     return 0.0;
   }
 }
}

I then run mvn install once more, then launch the web service with the following:

C:\dev\AsyncWSDev\AsyncService>java -cp target\ExchangeRateWebService.jar com.adamsresearch.ws.async.ExchangeRateEndpoint

http:// localhost:8282/exchangeRateService Published endpoint at URL http://localhost:8282/exchangeRateService

If we open a browser at the specified URL with "?wsdl" appended, we'll see the web-service WSDL file, verifying that we successfully deployed the service. Note that the Java runtime cleverly extracts the XML Schema from the WSDL file and references it via import. Both the WSDL file and the XML Schema file (as retrieved by the HTTP request in the browser) are dynamically generated by the Java runtime.

Create an external JAX-WS binding definitions file

Why are we performing this step? To produce a more-fully-functional client-side API for our to-be-created web service client. You can provide two types of binding definitions files to wsimport -- JAXB-related files, and a file that specifies some customizations of the web service (which is why it's called a binding customization).

Here is our binding customization file:


<?xml version="1.0" encoding="UTF-8"?>

<bindings
 xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
 wsdlLocation="http://localhost:8282/exchangeRateService?wsdl"
 xmlns="http://java.sun.com/xml/ns/jaxws">

 <!-- applies to wsdl:definitions node, that would mean the entire wsdl -->
 <enableAsyncMapping>false</enableAsyncMapping>

 <!-- wsdl:portType operation customization -->
 <bindings node="wsdl:definitions/wsdl:portType [@name='ExchangeRate']/wsdl:operation[@name='getExchangeRate']">
     <enableAsyncMapping>true</enableAsyncMapping>
 </bindings>
</bindings>

Note that the outer envelope of the file is called bindings, and that there is an inner element also called bindings. Not to go into too much detail, but elements can be applied at the global level, and at a portType or even an operation level. In this file, I've disabled asynchronous mapping at the global level, but turned it on just for the getExchangeRate operation. This precaution prevents potential new operations from being inadvertently exposed as asynchronous operations. More on the wsdlLocation later.

Generate the client-side artifacts

We're going to create a new Maven project for the web-service client (more on this later), so next we'll create another project:

mvn archetype:generate -DgroupId=com.adamsresearch.ws.async -DartifactId=AsyncClient

As with the service-side artifacts, I will create a resources directory, but I'll do this a little differently this time.

Instead of referencing the WSDL file in the project filesystem, I'll point wsimport to it via URL. Note this is what I did above in the binding customization file, since it also references the WSDL location. In the client project directory structure, I'll create a src/main/resources/jaxws directory and put the binding customization file (which I called async-bindings.xml) in the
directory.

Here's the relevant section of the client-project POM file, modified to point to the binding customization file and to access the WSDL file from the service itself:


<build>
 <finalName>ExchangeRateWebService</finalName>
 <plugins>
   <plugin>
     <artifactId>maven-compiler-plugin</artifactId>
     <configuration>
       <source>1.6</source>
       <target>1.6</target>
     </configuration>
   </plugin>
   <plugin>
     <groupId>org.codehaus.mojo</groupId>
     <artifactId>jaxws-maven-plugin</artifactId>
     <executions>
       <execution>
         <goals>
           <goal>wsimport</goal>
         </goals>
         <configuration>
           <wsdlUrls>
             <wsdlUrl>http://localhost:8282/exchangeRateService?wsdl</wsdlUrl>
           </wsdlUrls>
           <bindingDirectory>${basedir}/src/main/resources/jaxws</bindingDirectory>
           <keep>true</keep>
           <packageName>com.adamsresearch.ws.async.generated</packageName>
           <sourceDestDir>${basedir}/src/main/java</sourceDestDir>
         </configuration>
       </execution>
     </executions>
   </plugin>
 </plugins>
</build>

Let's do a first build, before we actually write a client, and see what we get:


  mvn install

Now take a look at the generated ExchangeRate Java interface. When we generated the artifacts for the server, we had a single method declaration for our one operation in this file:


 @WebMethod
 @WebResult(targetNamespace = "")
 @RequestWrapper(localName = "getExchangeRate", targetNamespace = "http://async.ws.adamsresearch.com/", className = "com.adamsresearch.ws.async.generated.GetExchangeRate")
 @ResponseWrapper(localName = "getExchangeRateResponse", targetNamespace = "http://async.ws.adamsresearch.com/", className = "com.adamsresearch.ws.async.generated.GetExchangeRateResponse")
 public double getExchangeRate(
     @WebParam(name = "arg0", targetNamespace = "")
     String arg0,
     @WebParam(name = "arg1", targetNamespace = "")
     String arg1);

When we implemented this interface in our endpoint, it was a simple matter to process the input parameters and return the double value. If you open the artifact that we just generated, you'll see there are two additional declarations:


 @WebMethod(operationName = "getExchangeRate")
 @RequestWrapper(localName = "getExchangeRate", targetNamespace = "http://async.ws.adamsresearch.com/", className = "com.adamsresearch.ws.async.generated.GetExchangeRate")
 @ResponseWrapper(localName = "getExchangeRateResponse", targetNamespace = "http://async.ws.adamsresearch.com/", className = "com.adamsresearch.ws.async.generated.GetExchangeRateResponse")
 public Response getExchangeRateAsync(
     @WebParam(name = "arg0", targetNamespace = "")
     String arg0,
     @WebParam(name = "arg1", targetNamespace = "")
     String arg1);

 @WebMethod(operationName = "getExchangeRate")
 @RequestWrapper(localName = "getExchangeRate", targetNamespace = "http://async.ws.adamsresearch.com/", className = "com.adamsresearch.ws.async.generated.GetExchangeRate")
 @ResponseWrapper(localName = "getExchangeRateResponse", targetNamespace = "http://async.ws.adamsresearch.com/", className = "com.adamsresearch.ws.async.generated.GetExchangeRateResponse")
 public Future getExchangeRateAsync(
     @WebParam(name = "arg0", targetNamespace = "")
     String arg0,
     @WebParam(name = "arg1", targetNamespace = "")
     String arg1,
     @WebParam(name = "asyncHandler", targetNamespace = "")
     AsyncHandler asyncHandler);

What happened here is that we got two additional options to retrieve the data -- one which returns a pollable object, and one which allows you to specify an asynchronous handler (note that Response is a subinterface of Future).

Implement the client-side logic

At this point, you're probably wondering why we will be invoking an asynchronous operation when we haven't yet implemented it on the server. Bear with me for a moment, while we write our client.

Here's my version of a client which exercises the three different available method signatures.


package com.adamsresearch.ws.async;

import java.net.MalformedURLException;
import java.net.URL;
import javax.xml.namespace.QName;
import javax.xml.ws.AsyncHandler;
import javax.xml.ws.Response;
import com.adamsresearch.ws.async.generated.ExchangeRate;
import com.adamsresearch.ws.async.generated.ExchangeRateService;
import com.adamsresearch.ws.async.generated.GetExchangeRateResponse;

public class ExchangeRateClient
{
protected ExchangeRateClient theClient = null;
protected String wsdlUrl = null;
protected double rate = 0.0d;
ExchangeRate excRate = null;

public static void main(String args[]) throws MalformedURLException, InterruptedException
{
 if (args.length != 1)
 {
   System.out.println("Usage java -cp <jarFile> com.adamsresearch.ws.async.ExchangeRateClient serviceWsdlUrl");
   System.exit(-1);
 }
 ExchangeRateClient client = new ExchangeRateClient(args[0]);
 Thread.sleep(5000L);
}

public ExchangeRateClient(String urlStr) throws MalformedURLException
{
 theClient = this;
 wsdlUrl = urlStr;
 URL url = new URL(wsdlUrl);
 QName qname = new QName("http://async.ws.adamsresearch.com/", "ExchangeRateService");
 ExchangeRateService exchangeRateService = new ExchangeRateService(url, qname);
 excRate = exchangeRateService.getExchangeRatePort();

 // synchronous:
 System.out.println("Airstrip One / Ganymede exchange rate, retrieved synchronously, is: " + excRate.getExchangeRate("AS1", "GMD"));

 // asynchronous with polling:
 try
 {
   Response = excRate.getExchangeRateAsync("AS1", "GMD");
   Thread.sleep (2000L);
   GetExchangeRateResponse output = response.get();
   System.out.println("--> retrieved via polling: " + output.getReturn());
 }
 catch (Exception exc)
 {
   System.out.println(exc.getClass().getName() + " polling for response: " + exc.getMessage());
 }

 // asynchronous with callback:
 excRate.getExchangeRateAsync("AS1", "GMD", new AsyncHandler()
 {
   public void handleResponse(Response response)
   {
     System.out.println("In AsyncHandler");
     try
     {
       theClient.setCurrencyExchangeRate(response.get().getReturn());
     }
     catch (Exception exc)
     {
         System.out.println(exc.getClass().getName() + " using callback for response:" + exc.getMessage());
     }
   }
 });
}

protected void setCurrencyExchangeRate(double newRate)
{
   rate = newRate;
   System.out.println("--> via callback, updated exchange rate to " + rate);
}
}

The Thread.sleep() in main is just to make sure we're still around when the web service responds. Finally, invoking the client:


c:\dev\AsyncWSDev\AsyncClient>java -cp target\ExchangeRateWebService.jar
com.adamsresearch.ws.async.ExchangeRateClient http://localhost:8282/exchangeRateService?wsdl
Airstrip One / Ganymede exchange rate, retrieved synchronously, is: 2.78
--> retrieved via polling: 2.78
In AsyncHandler
--> via callback, updated exchange rate to 2.78

So there we have it -- a Java SE client which hits a Java SE web service three ways: synchronously, asynchronously with polling, and asynchronously with a callback handler.

There's a lot to discuss about these results, some of which, frankly, I did not expect (for example, why did we not have to explicitly implement asynchronous service-side logic?). That is the topic of another post, which I hope you'll find interesting, too.

Asynchronous Java SE Web Services: 2. Discussion

2012-01-04T20:13:00.001-08:00

In an earlier post, we stepped through the building of an asynchronous web service, deployed in Java SE. I saved my comments for this post to keep things a little cleaner. But there are some loose ends to discuss, especially when you hear my motivation for building an asynchronous web service in the first place.

Normally, you want to invoke a service asynchronously because you expect potentially long server response times. This, I would think, is the most likely reason for wanting asynchronous invocation. For example, in a user-facing application, you don't want to block on a call, especially if your front end has other things it could be doing.

Lately, I've been spending a lot of time on "server push" applications, using frameworks like DWR (Direct Web Remoting) and CometD; these frameworks provide a clean façade for logic which either maintains long-lived HTTP connections or polls for data, for example. I'd rather not have to rely on a servlet-based framework to do this, however; if possible, I'd prefer to simply leverage the Java SE. So my ulterior motive was to see if I could use it to implement server push. Unfortunately, server push and asynchronous response are not equivalent, of course, although there's some obvious overlap. The overlap lies in that first (and, for async, only) response.

What I found most interesting about this exercise is that the asynchronous logic (at least, the logic visible to us, the developers) was manifested only on the client side. In other words, the asynchronous versions of the operation were entirely a client-side artifact -- we did nothing on the web-service side to implement asynchronicity! I did not expect that result. We just implemented a synchronous Java method on the service side, and the Java SE runtime took care of the asynchronous-invocation details for us, simply by virtue of our using the asynchronous operations of the client-side API.

I really expected, when I started this exercise, that I would generate asynchronous service-side stubs and actually implement the asynchronous logic myself. In that case, it would have been natural to assume that I could play some neat trick and continue sending data "down the pipe", rather than providing just one response and seeing the connection get closed afterwards. I'm not sure how I thought I was going to do this (groundless optimism?), but I planned on figuring that out once I got the basics working.

What we ended up with is exactly what JAX-WS advertised, and that is actually quite handy; all you have to do is to make the asynchronous call and you can go on with whatever else you need to do; the runtime takes the response, when it returns, and updates your pollable object (or calls your callback).

One thing that is appealing about this result is that you really don't need to build your web service by starting with a WSDL file, since the asynchronicity is purely a client-side artifact. In other words, if you prefer to start with a Java bean and use wsgen to generate the web service, that is fine -- you don't need to do anything else on the service side to support asynchronous invocation. It's only when you create the client that you need to worry about asynchronous invocation, but by then you already have a deployed web service, so you can generate the client-side, asynchronous API by running wsimport against the deployed web service's published WSDL file. Overall, a rather convenient result!

So, to recap:

JAX-WS in the Java SE provides relatively straightforward support for asynchronous web-service development. The service can be developed starting Java-first or WSDL-first, as the asynchronicity doesn't come into play until the client-side artifacts are generated.
JAX-WS does not (as far as I know at this point) support the concept of server push, by which I mean a long-lived output stream over which data can be pushed to the client.

Examining variables in JDI

2011-12-17T11:11:00.000-08:00

As I have mentioned in earlier posts, I am using the Java Debug Interface (JDI) to create a Java process-monitoring tool. I'm retracing my steps from an earlier such effort a few years ago, and as I wade through the details, I'm posting details in the hope they'll be helpful to someone else out there. I sometimes wonder who would really be interested in this material, but I have been noticing that my old posts on JDI, DTrace, etc. continue to get hits every week, even the old ones, so I know you're out there. I hope you find this article helpful, too.

Current Status:

At this stage in the development of my application, I have set up breakpoint requests and am stepping through events as they arrive. You'd probably do this in a loop similar to the following:


  EventQueue evtQueue = virtualMachine.eventQueue();
  EventSet evtSet = null;
  while (!stopRequested)
  {
      try
      {
          evtSet = evtQueue.remove();
          EventIterator evtIter = evtSet.eventIterator();
          while (evtIter.hasNext())
          {
          ...
          }
      }
      catch (Exception exc)
      {
          ...
      }
      finally
      {
          evtSet.resume();
      }
  }

In my application, I handle debugger events in a thread, so I need a way for the thread to know if it should stop (that's the purpose of the boolean stopRequested). Note that an EventSet actually contains an EventIterator which can contain several events. Also note that it is quite important to ensure that resume() is always called on the EventSet, regardless of any exceptions that are thrown. Otherwise, you could end up with some permanently hung threads and have to restart your target application.

Inspecting Variables:

At a breakpoint, you'll probably want to extract some of the current state of your target application. There are two places to find variables in the JDI API -- instance/class variables, and the stack. Static and instance variables can always be listed by inspecting the class's ReferenceType -- they're always there, as long as the class is loaded, even if the application isn't running (you may still have to be at a breakpoint to look at their values, of course, but the point is you can also see what a class's static and instance variables are simply by looking at its definition).

On the other hand, the stack variables aren't defined unless you are, well, on the stack where they're defined (e.g. local variables in a method). In JDI, you find these two types of variables in different parts of the API, in locations which will make sense to you as we continue our discussion. You see the same division in a debugger. For example in Eclipse, when you are at a breakpoint and look at the "Variables" window, you will see class/instance variables in a tree structure under "this", and all local (stack) variables listed peer-level to "this" under their local names.

The following sections assume that you have extracted an Event from an EventIterator, verified that it is a BreakpointEvent, extracted the EventRequest from the Event, and verified that it is a BreakpointRequest (these are all JDI classes).

Inspecting Static and Instance Variables:

Given a BreakpointEvent and a variable name (defined in your breakpoint request specification), how would you search for the variable in the static and instance variables list? Every BreakpointEvent has a Location (obtained via the location() method). In addition to having the usual information you would expect a Location object to have (the name of the source, the line number, method name, etc.), a Location has a reference to the class's ReferenceType. The ReferenceType has a method called allFields() which, as the API documentation says, "Returns a list containing each Field declared in this type, and its superclasses, implemented interfaces, and/or superinterfaces. All declared and inherited fields are included, regardless of whether they are hidden or multiply inherited. " Exactly what you want, right?

allFields() returns a List. The JDI Field type has a name field (method: name()) which can be used, while iterating over the List, to find the target variable. Once you have retrieved the correct Field, you get its Value by appealing to the ReferenceType again, as in:

Value val = theReferenceType.getValue(theField);

The only other detail at this point is actually extracting the value of the Value. This is a slightly messy topic, which we'll discuss after we look at the other location of variables -- the stack.

Inspecting Stack Variables:

Stack variables, and specifically their values at your breakpoint, are associated with the JDI StackFrame and Frame classes. They are referred to as LocalVariables in JDI, you get reach them through the BreakpointEvent, by accessing its ThreadReference, then the StackFrame at the top level (that is, element #0) of the stack. You can then query the visible variables on the StackFrame by name to find the target variable. In other words, you could do something like this:


 StackFrame stackFrame = breakpointEvt.thread().frame(0);
 LocalVariable localVar = stackFrame.visibleVariableByName(varName);
 Value val = stackFrame.getValue(localVar);

Note that you must get the LocalVariable from the StackFrame, but having retrieved it, you must again access the StackFrame to get the Value of the LocalVariable. Note also that we are now at the same stage we were with class/instance variables -- we have a Value, but we need to find out what its "value" is. As I said, this is a little messy and is the subject of the next section.

Extracting Values:

In JDI, a Value isn't just something you can meaningfully display with toString() (unless you find hash codes especially meaningful). The JDI API documentation contains a very useful table under com.sun.jdi.Value which describes the family of subinterfaces of Value. Basically, all the interfaces which represent a usable value contain a value() method, but this method is only defined in the sub-interfaces (since not all of them define one). This means that if you want to inspect a Value that at heart is a boolean, or a String, you can call value() on it, but you must first determine if it is an instance of the type you need, then cast it to that type and call value() on the cast instance. I don't know if there's any more practical way to do this that doesn't involve writing extremely clever (in other words, unmaintainable) code, so here's how I do it:


  if (value instanceof BooleanValue)
  {
    return ((BooleanValue)value).value();
  }
  else if (value instanceof ByteValue)
  {
    return ((ByteValue)value).value();
  }
  ...

and so on.

There are two main subinterfaces of Value -- PrimitiveValue and ObjectReference, along with a little more hierarchy which will be important to you. For example, if you are using code such as the example above to extract values, you will want to check for StringReference before ObjectReference. Why? StringReference has a value() method, as you would expect and hope, but StringReference is a subinterface of ObjectReference, which does not have a value() method. If you check for ObjectReference first, and you're looking at a String Value, you will miss out on the available cast to StringReference and miss out on a chance to inspect the value (yes, I accidentally did this; I did it this time, and I remembered when I did it that I did the same thing a few years ago).

Drill-down:

If you, too, are writing what I would call a "real-time" JDI-based tool, you'll probably want to provide some drill-down at a breakpoint. For example, you might want to look at a specific field of a variable on the stack, rather than just the Value of the variable. The way I handle this is to allow a "dot" notation, where "a.b.c" allows my users to specify that they want to look at field c of field b of variable/field a. Things get a little interesting here, too, as the following example illustrates.

Suppose you are monitoring the traffic of an application and you just want to know the length of Strings being processed (e.g. returned from a service) at a breakpoint. You don't really want to display the String itself; this could be enormous (and remember, your target application is halted while this breakpoint is being processed, so I/O on a large String could cripple the application). All you want to know is the length of the String. But you don't want to write code to actually invoke general methods on your stack/instance variables. And in this particular case, there is a field available to you from JDI on the String class -- count. So your users should be able to specify, for example, incomingMessage.count, and get back an integer with the length of the String.

The only problem is, not everyone knows that there is a field in java.lang.String called count -- they're more accustomed to calling the method length(). In an interactive debugger, this issue isn't a problem, as you don't specify what you want in advance -- you just inspect the state when the breakpoint occurs, and the debugger will show all the fields for you. In a debugger of the type I'm developing, users would need to know what is available. For a stack variable, this is a problem because you can't inspect it until the application is running. For class/instance variables, once you have the ReferenceType of the class, you can at least inspect those variables and drill down at will.

The way I'm handling this issue (for stack variables) is to provide a diagnostic version of a breakpoint request, where the user can request a dump of the variables on the stack, to a specified depth. Note such a feature should be used with some care; an application with a very large number of stack variables, all traversed to (say) 10 levels deep, can end up being suspended for seconds, minutes, or tens of minutes just dumping the state of the stack (yes, I've done that before too, but will not repeat that this time!). Suffice to say that if you want to inspect an application for nested variables, you should do so on a non-production system.

How do you drill down into fields? Unsurprisingly, there's a different procedure for class/instance variables than for stack variables. Unfortunately, some things in JDI are not immediately obvious (in fact, every time I write code to do this, I have to "re-discover" how to do it), which is why I illustrate below.

For class/instance variables, you got the fields by getting the ReferenceType and looking at its Fields. Having retrieved a reference to the Field, you can simply repeat the process -- there is no method on the Field itself to drill down, but you can get the ReferenceType of each Field and then get the Fields of each Field from each Field's ReferenceType. Here's an example:


   List childFields = bpEvt.location().declaringType().allFields();
   for (Field childField: childFields)
   {
     List grandChildFields = childField.declaringType().allFields();
     for (Field grandChildField: grandChildFields)
     {
       System.out.println(grandChildField.name() + " is a grandchild field");
     }
   }

The situation for stack variables is as follows. If you are trying to drill down into a variable, by definition it is an ObjectReference, right? In other words, there's no drilling down into a primitive. If the Value on the StackFrame is an instance of an ObjectReference, then you can get the ReferenceType of the Value and retrieve its Fields just as we did above when we retrieved the ReferenceType of the class/instance Field. Note that since StringReference is a subinterface of ObjectReference, we could do something like the following:


    else if (value instanceof StringReference)
    {
      // look at the fields:
      List childFields = ((StringReference) value).referenceType().allFields();
      for (Field childField: childFields)
      {
        System.out.println(">> child field: '" + childField.name() + "'");
      }
      return ((StringReference)value).value();
    }

If we run this code, we would see something like the following output for a String:

>> child field: 'value'
>> child field: 'offset'
>> child field: 'count'
>> child field: 'hash'
>> child field: 'serialVersionUID'
>> child field: 'serialPersistentFields'
>> child field: 'CASE_INSENSITIVE_ORDER'

(Note our count Field, which happens to be an IntegerValue). Of course, you can continue drilling down, using the same procedure, on each of these Fields.

That's all I have for now -- look for more in the coming weeks as the project comes along.

From OO to FP: Haskell I/O, Part 2

2011-12-07T20:24:00.000-08:00

In my Part-1 post on this topic, we actually did all the I/O I'm going to do here. We lazily read in the entire sample data file, a file containing data describing events generated by a process monitor. My next goal was to re-hydrate my Events from the Strings serialized to the file. These Strings were generated by calling the function show on my List of Events.

Time to back up a little. We got show for free, but we had to ask for it. Leaving aside the Event data type for a moment and looking at the simpler Property data type, remember how we defined it:

     data Property = Property {
     key :: Key,
     value:: Value }
     deriving (Show)

At that time we discussed "deriving (Show)" by simply noting it allowed ghci to know how to output Property values. Of course, there's more to it than that. The "more" is Haskell typeclasses. A Haskell typeclass defines a set of functions that can be defined to operate on a data type. The show function is a member of the Show typeclass and has the following type:

     *EventProcessor> :type show
   show :: Show a => a -> String

For a moment, let's see what had happened if we had not specified that Property derives Show:

     -- file:  c:/HaskellDev/eventProcessor/simpleProperty.hs
  type Key = String
  type Value = String
  data Property = Property {
    key :: Key,
    value:: Value }

If we load this file, then create a Property, then try to show it, we see the following:

     Prelude> :load simpleProperty.hs
   [1 of 1] Compiling Main             ( simpleProperty.hs, interpreted )
   Ok, modules loaded: Main.
   *Main> let prop1 = Property "key1" "value1"
   *Main> show prop1
   :1:1:
   No instance for (Show Property)
     arising from a use of `show'
   Possible fix: add an instance declaration for (Show Property)
   In the expression: show prop1
   In an equation for `it': it = show prop1

That's considerably more helpful than a typical compiler error message. The key is the hint to add an instance declaration. In order for the functions of a typeclass to be applicable to your data type, you need to declare an instance of that typeclass that handles your data type. In our case, we can get Show simply by stating that we derive it. So we go back to our old definition:

     -- file:  c:/HaskellDev/eventProcessor/simpleProperty.hs
   type Key = String
   type Value = String
   data Property = Property {
     key :: Key,
     value:: Value }
     deriving (Show)

Repeating our earlier test, now we get the following:

     *Main> :load simpleProperty.hs
   [1 of 1] Compiling Main             ( simpleProperty.hs, interpreted )
   Ok, modules loaded: Main.
   *Main> let prop1 = Property "key1" "value1"
   *Main> show prop1
   "Property {key = \"key1\", value = \"value1\"}"

That's definitely an improvement. But, back to the subject -- what I want is to be able to read a String and turn it in to a Property. First, let's try to read a String into an Integer:

     *Main> read "5"
   :1:1:
   Ambiguous type variable `a0' in the constraint:
   Read a0) arising from a use of `read'
   Probable fix: add a type signature that fixes these type variable(s)
   In the expression: read "5"
   In an equation for `it': it = read "5"

Again, a helpful hint from ghci. It basically doesn't know what we are trying to create and suggests adding a type signature. Here's how you would do it:

     *Main> let anInt = (read "5")::Integer
   *Main> :type anInt
   anInt :: Integer
   *Main> show anInt
   "5"

Haskell now knows we're trying to read an Integer, so it recognizes that anInt is an Integer and knows how to show it. All of this should tell you that Integer has defined instances for both Show and Read.

Can we get Read "for free" simply by stating that our data type derives it? It can't hurt to try:

     -- file:  c:/HaskellDev/eventProcessor/simpleProperty.hs
   type Key = String
   type Value = String
   data Property = Property {
     key :: Key,
     value:: Value }
     deriving (Read, Show)

Next, I'll create a Property and output it with show, then see if that format (which, incidentally, is how I created my sample Event file -- using show on a List of Events) is "read"-able:


   *Main> let prop1 = Property "key1" "value1"
   *Main> show prop1
   "Property {key = \"key1\", value = \"value1\"}"
   *Main> let prop2 = (read "Property {key = \"key1\", value = \"value1\"}")::Property
   *Main> show prop2
   "Property {key = \"key1\", value = \"value1\"}"

This is great news -- I didn't have to write a parser, and I'm perfectly happy to use, as my format, the same format that Haskell uses to show a Haskell data structure.

What I would really like to do is to be able to read a full Event, as defined in my earlier posts, including a List of Events. Here's an example Event:

     "Event {timestamp = 1320512200548, className = \"java.lang.String\", lineNumber = 1293, message = \"NPE in substring()\", properties = [Property {key = \"userId\", value = \"smith\"},Property {key = \"sessionId\", value = \"ABCD1234\"}]}"

Let's try the same trick with Event that we used with Property:

     -- file:  c:/HaskellDev/eventProcessor/notSoSimpleProperty.hs

   type Timestamp = Integer
   type ClassName = String
     type LineNumber = Integer
     type Message = String
     type Key = String
     type Value = String
     type Properties = [Property]

     data Property = Property {
       key :: Key,
       value:: Value }
       deriving (Read, Show)

     data Event = Event {
       timestamp :: Timestamp,
       className :: ClassName,
       lineNumber :: LineNumber,
       message :: Message,
       properties :: Properties }
       deriving (Read, Show)

Was it enough simply to declare that Event derives Show? Let's see:

     *EventProcessor> :load notSoSimpleProperty.hs
   [1 of 1] Compiling Main             ( notSoSimpleProperty.hs, interpreted )
   Ok, modules loaded: Main.
   *Main> let event1 = (read "Event {timestamp = 1320512200548, className = \"java.lang.String\", lineNumber = 1293, message = \"NPE in substring()\", properties = [Property {key = \"userId\", value = \"smith\"},Property {key = \"sessionId\", value = \"ABCD1234\"}]}")::Event
   *Main> show event1
   "Event {timestamp = 1320512200548, className = \"java.lang.String\", lineNumber = 1293, message = \"NPE in substring()\", properties = [Property {key = \"userId\", value = \"smith\"},Property {key = \"sessionId\", value = \"ABCD1234\"}]}"
   *Main> show (properties event1)
   "[Property {key = \"userId\", value = \"smith\"},Property {key = \"sessionId\", value = \"ABCD1234\"}]"

This is great. We can take a String representation of an Event, as output by show, and use it directly with read to instantiate an Event variable.

This discussion in no way described how to provide an instance of a typeclass; maybe in another post. Right now I still want to create Events from my sample Event file.

My first cut at this is the following:


   import System.IO
   import EventProcessor

   main :: IO ()
   main = do
    inh <- openFile "sampleEvents" ReadMode
    inContents <- hGetContents inh
    let eventList = processData (lines inContents)
    hClose inh

   processData :: [String] -> [Event]
   processData = map createEvent

   createEvent :: String -> Event
   createEvent event = (read event)::Event

While this appears to be correct, it doesn't do anything that proves I've been able to parse my example-file lines into a List of Events. Unfortunately, at this point I'm a little stuck, as I haven't yet figured out how to correctly extract elements from my Events (I can do so interactively in ghci, but I am plagued by compiler errors if I try to do so in the do block).

I'm going to leave this for a while and go back to some online resources/tutorials, then revisit. The problem for me is that I still don't understand the I/O monad, as it's called. It's becoming clear to me that the concept of a monad -- apparently so integral to Haskell -- isn't easily grasped, as evidenced by the overwhelming number of "Here's my take on monads" tutorials on-line (one author, I see, jokes that everyone who learns about monads appears to post a tutorial on the subject shortly thereafter). So I'm off to the Haskell Wiki to learn about monads, then learn some more about Haskell I/O, and then pick up where I left off.

From OO to FP: Haskell I/O, Part 1

2011-11-22T11:15:00.000-08:00

I'm about halfway through Real World Haskell, and I've spent a week trying to decide when to write this post. As the authors point out, Haskell I/O is easy to work with. However, understanding the significance of what is going on in Haskell I/O requires a little more than simply outputting "Hello, World". The reason: once you include I/O, you start mixing pure and impure code, and a good understanding of what's going on will additionally include some discussion of monads. While my understanding of these topics is starting to gel, what I'm about to say should be taken (like everything else in this series) more as a snapshot of my transition from object-oriented, imperative-style development to functional programming, rather than an authoritative resource.

I'm going to continue with the example I've been using to-date, which involves working with data that represents events coming from a process-monitoring system. Instead of including the events themselves in my source code, this time I'm going to read them from a file. Here's a rehash of what the data structures look like. as output by ghci:

     Event {timestamp =  1320512200548, className = \"java.lang.String\", lineNumber = 1293,  message = \"NPE in substring()\", properties = [Property {key =  \"userId\", value = \"smith\"},Property {key = \"sessionId\", value =  \"ABCD1234\"}]}

My first goal is simply to read a file of these items, one per line, and echo them to the console. I'm using GHC Haskell and will be using the ghci interpreter as well. I'm going to start with my original Haskell file (which I won't reproduce here); suffice to say I would like to turn it into a module and export my data type and type synonym definitions, so I can use them from my new file processor.

I'm starting slowly because I have to backtrack a little to fit Haskell's requirements. My original file was called eventProcessor.hs, and I see now that a Haskell module name must start with a capital letter, and that the name of the module must match the base name of the source file. My first change, then, is to rename eventProcessor.hs to EventProcessor.hs (this will be enjoyable for those of you working on a Windows system). Then, inside my file, I add the following module declaration at the top of my file (note: this also comes before any import statements):


module EventProcessor
  (
    Timestamp,
    ClassName,
    LineNumber,
    Message,
    Key,
    Value,
    Properties,
    Property,
    Event,
    isUserWithId,
    eventGeneratedByUser,
    getEventsGeneratedByUser
  ) where

I've decided to export basically everything I defined in this file. Don't forget, of course, that the above indentation of the parentheses is significant, rather than cosmetic. My original file also contained a number of variable definitions. I'll take those out once I'm convinced I can actually retrieve them from a file!

Leaving the EventProcessor module for a moment, let's read the contents of my events file and output them to the screen. This example won't do much, but it will provide some notation we need to examine. Here is the source code:


     import System.IO

     main :: IO ()
     main = do
       inh <- openFile "sampleEvents" ReadMode
       inContents <- hGetContents inh
       putStrLn inContents
       hClose inh

This is the first Haskell program I have run from a console. Note the System.IO import, and the familiar main function. Not as familiar -- the type of main. It is an "IO ()". The IO in the type not only lets you know I/O is involved, but lets you know you are dealing with a function (main) that may have side effects (is not pure). The "()" part just means that main does not return a value ("()" is an empty tuple, called "unit"). So, "main :: IO ()" is a function which has side effects and does not return a value. Additionally, of course, main is the function which is called when you execute the compiled version of the source code for this file.

The function gets an input handle on my events file, uses a special function to get the entire contents of the file (remember that Haskell's lazy evaluation allows such a function to be practical), and puts the output to the console, finally closing the handle. Before we inspect the logic any further, here is how we compile and run the program, and what we see for output:


     c:\HaskellDev\eventProcessor>ghc --make processEventsFile.hs
     [1 of 1] Compiling Main             ( processEventsFile.hs, processEventsFile.o )
     Linking processEventsFile.exe ...
    
     c:\HaskellDev\eventProcessor>processEventsFile.exe
     Event {timestamp =  1320512200548, className = \"java.lang.String\", lineNumber = 1293,  message = \"NPE in substring()\", properties= [Property {key =  \"userId\", value = \"smith\"},Property {key = \"sessionId\", value =  \"ABCD1234\"}]}
     ...
     Event {timestamp =  1320512265147, className = \"javax.swing.JPanel\", lineNumber = 388,  message = \"initialized\", properties = [Property {key = \"userId\",  value = \"smith\"},Property {key = \"sessionId\", value =  \"ABCD1234\"}]}]"

The next interesting thing you see in the sample code is the do. Before we discuss do, let's discuss main again, with its type of IO (). In Haskell, anything with a type starting with IO is called an I/O action. I/O actions are lazily evaluated, too, and are executed when their parent I/O actions are executed. In this program, main is an I/O action, so putStrLn, also an I/O action, will be executed when main is executed. In other words, their side effects do not occur when they are evaluated, but when they are actually performed. A do block is used to chain (join) actions together; the do is necessary only if you have more than one action. The value of a do block is the last action in the block to be performed.

Note also the <- notation. Assignment from an I/O action looks a little different than assignment from a pure function. The <- operator takes the result from performing an I/O action and stores it in a variable. As an example, let's look at openFile's type in ghci:


     *EventProcessor> :type System.IO.openFile
     System.IO.openFile
       :: FilePath
          -> GHC.IO.IOMode.IOMode
          -> IO GHC.IO.Handle.Types.Handle

So openFile is an I/O action of type "IO GHC.IO.Handle.Types.Handle." In our sample program, the openFile I/O action is performed when main is performed, and at that point the result of openFile is assigned to the inh handle using the <- operator.

If the operator <- takes the result of an I/O action out of System.IO, then you might naturally ask what takes values in the opposite direction. The return keyword provides this functionality. Don't attempt to correlate the Haskell return with the return keyword in any imperative programming language you have used. As an example, suppose we have a function which returns an I/O action:


     import System.IO
   
     main :: IO ()
     main = do
       inh <- openFile "sampleEvents" ReadMode
       inContents <- getContentsHereInstead inh
       putStrLn inContents
       hClose inh
   
     getContentsHereInstead :: Handle -> IO String
     getContentsHereInstead handle = do
       inContents <- hGetContents handle
       return inContents

I have modified my earlier program and now delegate the actual "getting" of the contents to a function. Notice that the type of this function is a "IO String" -- it has side effects as does main, only this time it returns an I/O action which, when performed, returns a String containing the file contents.

Interestingly, note the assignment statements. In getContentsHereInstead, I use return to wrap inContents in an IO String, as that is the return value type of the function. Because hGetContents handle defines an I/O action, I could just as easily have defined the function as follows:

     getContentsHereInstead :: Handle -> IO String
     getContentsHereInstead handle = do
       hGetContents handle

and saved myself the trouble of (1) extracting the I/O action result from the IO system, and (2) re-wrapping it in IO for the purposes of returning the correct type. An interesting situation exists in main, where I use the <- operator to assign the results of getContentsHereInstead to a pure value, then pass the pure value to putStrLn. Note, however, that I really do need the pure value to pass to putStrLn. From ghci:

     Prelude> :type putStrLn
     putStrLn :: String -> IO ()

putStrLn returns IO (), but it actually does expect a pure String as input. So, it would be incorrect to try to "putStrLn" the actual output of getContentsHereInstead; if I try something like

     putStrLn (getContentsHereInstead inh)

I will get the following complaint from ghci:

     processEventsFile2.hs:7:13:     
         Couldn't match expected type `[Char]' with actual type `IO String'
         Expected type: String
           Actual type: IO String
         In the return type of a call of `getContentsHereInstead'
         In the first argument of `putStrLn', namely
           `(getContentsHereInstead inh)'

The problem is that putStrLn expected a pure String and instead got an IO String. So we see that <- maps the result of an I/O action to a pure value, and return wraps the pure value in System.IO, and that we need to ensure we're passing around values of the expected data types.

I still haven't unmarshalled my lines in my data file to Event objects, which was the motivation of this post. To accomplish this goal, I first want to get the contents of the file, as before, and this time, pass a pure value to a non-I/O function which will re-hydrate the Event objects. To keep the post to a manageable length, I'm going to call it a day, and call this Part 1. In Part 2, coming later this week, I'll finish up.

Monitoring classloading in JDI

2011-11-14T20:44:00.000-08:00

For the last few weeks, I have been building a Java process monitoring tool based on the Java Debug Interface. Although I've done much of this work before, it has been a few years, and so now I'm retracing my steps. As I remember the details and pitfalls, I've been posting my notes in the hope that you'll find them useful.

Today I'm going to talk about ClassPrepareEvents, after a little background. As you probably already know, you can attach a debugger to an already-running Java process, or launch the target process itself from your debugger (using various command-line switches). In my project, I'm always going to be attaching to a running process, as the point is to collect process data on an as-needed basis. The reason JDI's ClassPrepareEvent is interesting is that, when you launch a debug target process, or even when you attach to an already-running process, it's likely that some of your desired breakpoints lie in classes which have not yet been loaded.

In my usual scenario, I call the com.sun.jdi.VirtualMachine's allClasses() method to get a list of all loaded ReferenceTypes. One way to think of a ReferenceType is as a chunk of a Java class definition. If your Java class has inner classes, then they will be broken out by JDI into separate ReferenceTypes. Each ReferenceType contains a collection of line locations; these correspond to lines of code on which breakpoints can be set and are identified by (among other things) the source-code line number. If a line of source code cannot be the target of a breakpoint, then there will not be a line location for it in the ReferenceType. In my debugger-based applications, I step through the line locations of all the ReferenceTypes, matching up line locations with breakpoint specifications, and then register my breakpoint requests.

As you can guess, I have a potential problem: what should I do if a class I need has not yet been loaded at the time I'm constructing my breakpoint requests? The answer is: JDI's ClassPrepareEvent. The entry point for using this part of the API is the EventRequestManager's createClassPrepareRequest() method. Having made our request, the same event-listener loop we use to wait for breakpoint events can also be used to wait for class prepare events (see the JVM specification for a definition of class preparation).

One thing I remember from my previous development on this API is that there is a timing risk here. You probably want to create the class prepare request before you iterate over the list of currently-loaded classes. The reason is that you don't want to fall into this trap:

Iterate over a set of the currently-loaded classes, processing and making breakpoint requests.
Suddenly, a class you need is loaded!
You register your class-prepare event and start getting events as classes are loaded, but you miss the class that loaded in between step #1 and step #3.

Here's another possible trap:

Register for class-prepare events so you don't get caught by the above issue.
Iterate over the currently-loaded classes, requesting breakpoints as necessary.
Process newly-loaded classes, requesting breakpoints as necessary.

The problem with this second approach is that you may process the same breakpoint twice. Why? By the time you iterate over the currently-loaded classes, some of the classes in that list are very likely going to be classes which have shown up in your class-prepare listener. Neither of these problems can be fixed by slapping a synchronized keyword somewhere.

Whether you launch your target application from your debugger or attach to it after the fact, you will have to deal with some variation of this issue. The way I deal with it is to add some state to the class I use to define each breakpoint specification. As each corresponding loaded class is found and the breakpoint request is made, I set a flag on the specification so that I know the request was registered. Further, I follow the second approach outlined above (better to have duplicates than to miss one). If I see a class-prepare event for a class I've already processed from the VM's ReferenceType list, then I simply skip over it. I do the same for the reverse situation, in which my list of ReferenceTypes contains ReferenceTypes which I have just processed in my ClassPrepareEvent listener.

Finally, one issue I have not looked at before (either for this development effort, or in my previous development in this area) -- what happens when a class is unloaded, especially a class on which you have registered breakpoint requests. For example, will a registered breakpoint request prevent a class from being unloaded? Do you care about a stranded breakpoint request if the class isn't even loaded? (Answer: yes, I suppose, if it gets reloaded and you no longer have a valid breakpoint request for it). JDI does have a ClassUnloadEvent, for which you can also register a listener. As I said, I have not dealt with this (possible) issue, having never seen a target class get unloaded before, but it's good to know "there's an API for that".

More on Haskell/FP function organization; lazy evaluation

2011-11-12T09:37:00.000-08:00

After my last post scrolled off the bottom of the page, I realized I missed a couple of opportunities: one related to some additional code optimization, and one related to the topic of lazy (or nonstrict) evaluation.

First, let me review what I was doing. I was processing a data structure I was using to represent events generated from a process monitor. The structure includes a timestamp, a class name and source line number, a text message, and a set of properties. Displaying one of these in ghci yields the following:


Event {timestamp = 1320513130333, className = "com.adamsresearch.jarview.JarView", lineNumber = 388, message = "fileNotFound", properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]}

My goal was to write -- in authentic FP style -- some code to extract, from a List of these events, events whose userId property value matched a particular user ID. Here is the code (minus type definitions, etc.) I ended up using:


eventGeneratedByUser :: String -> Event -> Bool
eventGeneratedByUser id event =
 if (containsUserPropertyWithValue (properties event) id)
   then True
   else False
 where
   containsUserPropertyWithValue (x:xs) id =
     if (isUserWithId x id)
       then True
       else containsUserPropertyWithValue xs id
     where
       isUserWithId prop id =
         if (key prop == "userId" && value prop == id)
           then True
           else False
   containsUserPropertyWithValue [] _ = False

Finally, in ghci, I retrieved the list of events I wanted with the following:


    *Main> let filterPred = eventGeneratedByUser "adams"
    *Main> let eventsForAdams = filter filterPred eventList

One theme object-oriented developers hear about repeatedly in FP is to stop thinking in loops, and start thinking in terms of recursion. An additional theme is (as best as I can state it) to avoid recursion when you can use a mapping-style function, that is, a function which structurally is defined to operate on all the elements of a list. When you set up a recursive function to process a list, a large part of what you are building is boilerplate. Boilerplate is the developer's curse: some live with it (care to pound out the boilerplate for Java's GridBagLayout and GridBagConstraints, anyone?), while others replace it with "frameworks" that the rest of us usually find even less appealing than writing the boilerplate code ourselves.

As you can tell from the above source code, I used recursion to create the boolean function which determines if an event is associated with a user. However, when I actually filtered the list, instead of using a recursive function I used Haskell's filter, passing it my partial function. For that last step, I feel I finally was writing in the spirit of Haskell and FP in general, but now let's take a look at that source code again and see if there's room for improvement.

Note first my innermost nested function, isUserWithId:


isUserWithId prop id =
 if (key prop == "userId" && value prop == id)
   then True
   else False

Note from my old post that I used Haskell record syntax to define the Property data type and that Haskell has used type inference to recognize that I am passing a Property structure to the isUserWithId function. All this function does is look at a key-value pair and return True if the key is userId and the value matches the parameter I pass in. I think this function is fine "as is".

The enclosing function, found in the where clause of the function eventGeneratedByUser, is a recursive function. Let's repeat the whole function:


eventGeneratedByUser :: String -> Event -> Bool
eventGeneratedByUser id event =
 if (containsUserPropertyWithValue (properties event) id)
   then True
   else False
 where
   containsUserPropertyWithValue (x:xs) id =
     if (isUserWithId x id)
       then True
       else containsUserPropertyWithValue xs id
     where
       isUserWithId prop id =
         if (key prop == "userId" && value prop == id)
           then True
           else False
   containsUserPropertyWithValue [] _ = False

What I am doing with this function is essentially iterating over the Property elements of an Event, returning True as soon as I find a Property that matches the desired condition.

Can this be converted from a recursive function to something a little more compact? As it turns out, yes, easily. Haskell contains a function called any which acts on a List and returns True if any element of the List is True. To use this function in my example, I would rearrange my code to calculate the isUserWithId value for each Property in the data structure, wrap those boolean values in a List, and apply the Haskell function any to the List. If you are wondering if I will be wasting cycles calculating values I may not need, bear with me for a few minutes.

The first question is how to generate a List of boolean values, the results of applying isUserWithId to a List of properties. To start, let me promote isUserWithId to a standalone function, but before we do that, let's do the same thing we did in my last post -- plan ahead to be able to use it as a partial function. By this I mean we'll want to apply this function to a list of Property values, where the "user ID" will be built-in, so to speak, to the function definition. I'll redefine the function to take the user ID first, so I can create a partial function later which I can apply directly to a Property (note this involves re-ordering the arguments in the isUserWithId equation, also):


isUserWithId :: String -> Property -> Bool
isUserWithId id prop =
 if (key prop == "userId" && value prop == id)
   then True
   else False

We can verify the function, both in its "full form", where we pass in both the user ID and the Property and in its partial form, where we first create a partial function with the user ID specified, and then pass in only a Property:


    *Main> prop4
    Property {key = "userId", value = "smith"}
    *Main> isUserWithId "smith" prop4
    True
    *Main> let containsSmith = isUserWithId "smith"
    *Main> containsSmith prop4
    True

Next, I'll take the existing three, short Property Lists from my example and concatenate them into a larger list:


    *Main> let bigPropList = concat [propList1,propList2,propList3]
    *Main> bigPropList
    [Property {key = "userId", value = "smith"},Property {key = "sessionId", value = "ABCD1234"},Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"},Property {key = "userId", value = "jobim"},Property {key = "sessionId", value = "WXYZ9876"}]

Now let's use any to return True if any item in the list matches our partial function, which is testing to see if a Property has user ID "smith":


    *Main> any containsSmith bigPropList
    True

and that is literally all we had to do to get our result. Note the interesting invocation of any: our partial function takes a Property, not a List of Property values. The any function steps through the elements of the list and returns True as soon as the invocation of containsSmith against a single Property returns True.

To integrate this code into our new version of eventGeneratedByUser, we could do the following:


eventGeneratedByUser :: String -> Event -> Bool
eventGeneratedByUser id event =
 any (isUserWithId id) (properties event)

and then test this in ghci:


    *Main> head eventList
    Event {timestamp = 1320512200548, className = "java.lang.String", lineNumber = 1293, message = "NPE in substring()", properties = [Property {key = "userId", value = "smith"},Property {key = "sessionId", value = "ABCD1234"}]}
    *Main> eventGeneratedByUser "smith" (head eventList)
    True

Note the code is considerably more compact, and (unlike in some languages where "terse" = "dense") actually more readable. In fact, things are so clean we can go a step farther than we did before and provide a function to return a subset of the List, something we just did interactively in ghci the last time:


getEventsGeneratedByUser :: String -> [Event] -> [Event]
getEventsGeneratedByUser id events =
 filter (eventGeneratedByUser id) events

and, testing in ghci:


    *Main> getEventsGeneratedByUser "jobim" eventList
    [Event {timestamp = 1320512200699, className = "javax.swing.JPanel", lineNumber = 388, message = "initialized", properties = [Property {key = "userId", value = "jobim"},Property {key = "sessionId", value = "WXYZ9876"}]},Event {timestamp = 1320512203699, className = "javax.swing.JList", lineNumber = 1255, message = "model replaced", properties = [Property {key = "userId", value = "jobim"},Property {key = "sessionId", value = "WXYZ9876"}]},Event {timestamp = 1320512259333, className = "javax.lang.Integer", lineNumber = 133, message = "number format exception", properties = [Property {key = "userId", value = "jobim"},Property {key = "sessionId", value = "WXYZ9876"}]}]

Note how compact and readable our resulting code is. We have one function which determines if a Property contains a user ID of a specified value -- 5 lines of code, including the type declaration. We have two additional functions, one of which returns True if an Event is generated by a specified user, and one which returns a subset of a List of Events which satisfy this condition -- each 3 lines of code, including the type declaration. And as I mentioned earlier, in this case, compact actual means readable.

Lazy Evaluation

In the above code, you might have been wondering "What is the performance impact of looping through a List of, say, 5000 properties and returning True if only one of them meets the test? Especially if the first property meets the test?". Haskell evaluates expressions only when they are needed. Technically in Haskell this is called nonstrict evaluation, and it allows you to do some things that would truly look odd in Java. For example, in Haskell you can define a list as a range of elements using ..; the following defines a list of integer values from 1 through 9:


    *Main> [1..9]
    [1,2,3,4,5,6,7,8,9]

While this may look normal, what do you think of this?


    Prelude> [1..]
    [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40
,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60
,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80
,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120
,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135
    ...

where I have added the line breaks to make it easier to read. Big surprise: this is an infinite list starting with the integer 1, and ghci will happily continue printing this output to the screen forever, or until Something Really Bad Happens, whichever comes first. I had to hit Control-C to stop the output. But, you can assign this infinite list to a variable:


    Prelude> let infiniteList = [1..]
    Prelude>

with no problems. The reason is that infiniteList has not yet been evaluated (populated). If you were to try to show this in ghci, you'd be back to the infinite output again. Nonstrict evaluation allows us to do some interesting things. For example, you can take the first 11 items of an infinite list:


    Prelude> take 11 infiniteList
    [1,2,3,4,5,6,7,8,9,10,11]
    Prelude>

This feature allows us to write code that (for example) returns True if "any" element in a List satisfies some predicate. Haskell isn't going to sift through the entire List, evaluating every expression, and then step through the List and apply the predicate. So it doesn't matter how many properties my events have; I'll return a True as soon as I get a match. Note that there are cases where all the elements will indeed be evaluated; for example, if you are summing the elements of a List (of course, you didn't expect Haskell to give you the sum of the elements of an infinite List, right?).

Nonstrict evaluation is already one of my favorite features of Haskell, right up there with standard library functions that operate on all the elements of a List. Next post, I expect to discuss I/O in Haskell, continuing along with the process-event-monitoring example I've been using to date.

Haskell From an OO Developer's Perspective, Part 3

2011-11-05T08:54:00.000-07:00

Today I'm going to process a set of structured data using Haskell, tainted by years of Smalltalk, C++, Java and C# experience. I've been following the book Real World Haskell, which I chose for a couple of reasons:

From the little I know of functional programming, Haskell appears to be a relatively "pure" FP language, and
The book is very comprehensive, covering FP topics of interest in depth.

While there are shorter introductions to FP, or good books on hybrid OO/FP languages, my preference is to make a clean break with OO and quickly become proficient in FP.

The structured data

I'm going to process a set of events generated by a Java program (they could be generated by any program, but this choice allows me to leverage one of my other ongoing projects). Each event consists of:

A timestamp, the epoch time measured in the usual Java milliseconds
A fully-qualified Java class name
The line number in the class at which the event is generated
A short descriptive message
A list of key-value pairs

The key-value pairs are analogous to the properties of a JMS message. In my application, I'll assume that some keys are present in every message, for the purposes of filtering messages during processing.

If you, too, are coming to Haskell or FP in general from an OO background, you don't need to see how this data structure would be implemented as an OO object, so I will focus instead of implementing it in Haskell. Here is my first cut:


    -- file:  c:/HaskellDev/eventProcessor/eventProcessor.hs

    type Timestamp = Integer
    type ClassName = String
    type LineNumber = Integer
    type Message = String
    type Key = String
    type Value = String
    type Property = (Key, Value)
    type Properties = [Property]

    data Event= Event Timestamp ClassName LineNumber Message Properties
      deriving (Show)

I have defined Properties to be a list of Property structures, where a Property is a 2-tuple consisting of a String Key and a String Value. Hopefully this structure will be interesting enough to provide a non-trivial example of an event processor in Haskell. Note that I included deriving (Show) in the type definition so that ghci will know how to output a value of my new type.

Now that we've defined the data type, let's create an Event value:


    *Main> let prop1 = ("sessionId", "ABCD1234")
    *Main> :type prop1
    prop1 :: ([Char], [Char])
    *Main> let props = [prop1]
    *Main> let evt1 = Event 1320512200548 "java.lang.String" 1293 "NPE in substring()" props
    *Main> evt1
    Event 1320512200548 "java.lang.String" 1293 "NPE in substring()" [("sessionId","ABCD1234")]

Next, we'll look at what we might want to do with this kind of data, and how we could process it in Haskell.

Generating the data

Suppose for a moment that we have a large application running, with many concurrent user sessions, producing hundreds or thousands of these values (our Events above) per minute. Occasionally a session will "come off the rails", in which case it would be really useful to be able to extract all messages specific to that user's session to see what is going on. We may want to do this in real time, or we may just process the data "after the fact", restricting ourselves to a time window during which the issues arose

In this example, I'm going to concentrate on the post-processing case. One reason is that I don't want to get into event-driven Haskell programming just yet, and a second reason is that I can arrange to skip (for now) the "non-FP-pure" code used to get the Events (file I/O, for example) and focus only on the "pure-FP" code. So I'll assume that we have a ready-made Haskell List of Event values to process in our example. I'll set the data up in my eventProcessor.hs file, entering as many values as I can before being overcome by boredom. I will put two properties in each property list: a session ID and a user ID (on the assumption that sessions come and go for the same user, and a user may even have multiple sessions open simultaneously). In my example data set there will be three user sessions going at any one time. Note that the let in variable assignment is a ghci artifact, so it isn't necessary if you make your assignments in the Haskell source file, below my Event data type definition:


    prop1 = ("sessionId", "ABCD1234")
    prop2 = ("sessionId", "EFGH5678")
    prop3 = ("sessionId", "WXYZ9876")
    prop4 = ("userId", "smith")
    prop5 = ("userId", "adams")
    prop6 = ("userId", "jobim")

    propList1 = [prop4, prop1]
    propList2 = [prop5, prop2]
    propList3 = [prop6, prop3]

    eventList = [
      Event 1320512200548 "java.lang.String" 1293 "NPE in substring()" propList1,
      Event 1320512200699 "javax.swing.JPanel" 388 "initialized" propList3,
      Event 1320512203699 "javax.swing.JList" 1255 "model replaced" propList3,
      Event 1320513130333 "com.adamsresearch.jarview.JarView" 388 "fileNotFound" propList2,
      Event 1320513255342 "com.adamsresearch.jarview.FileFilter" 79 "initialized" propList1,
      Event 1320513257324 "com.adamsresearch.jarview.ArchiveSearch" 193 "search started13255342" propList2,
      Event 1320512259333 "javax.lang.Integer" 133 "number format exception" propList3,
      Event 1320512260122 "com.adamsresearch.jarview.JarView" 725 "search started" propList2,
      Event 1320512263122 "com.adamsresearch.jarview.JarView" 779 "search completed" propList2,
      Event 1320512265147 "javax.swing.JPanel" 388 "initialized" propList1
      ]

Note a common beginner mistake, which I made initially -- I know that indentation is important in Haskell, but I still made the old-style indentation mistake of putting the final, closing square bracket in column 1. This results in an error because the closing bracket is itself part of the List assignment statement. Hence the indentation, above, on the last line of the eventList assignment. The indentation means we're still in the assignment statement.

Processing the data

Given this List of sample data, in imperative-style programming you would probably resort to a loop statement to process the individual Events. Haskell does not have a loop construct; instead, learn to think about loop-style processing of a List as

Recursively operating on each element of the List, or
Using Haskell library functions which operate on all elements of a List.

Although we'll settle on the second case, most FP books discuss the first case first, as it seems a more natural transition from imperative programming, so we will follow that approach here, too.

To introduce the idea of processing a List of data with a recursive function, let's first aim low: let's write a function that just returns a new List of the same data. This will be a simple function and looks as follows:


    -- file:  c:/HaskellDev/eventProcessor/clone.hs

    clone :: [a] -> [a]

    clone (x:xs) = x : clone xs
    clone [] = []

clone takes as an argument a List, and returns a List. The first equation of clone performs a pattern match on the incoming List using the List constructor (:). If this pattern matches, the head of the List (x) is extracted from the list, and clone is called on the tail of the List. The head of the list and the return of clone will be used, as the constructor indicates, to form a new List. So we're just cloning the input List here. As recursion continues, eventually the tail will be an empty List, at which point the second equation will match and the function will return an empty List (appended to the List it has been building on each recursive call).

To see this function in action, we load it into ghci, then inspect the value of the List returned when we pass a List to clone:


    *Main> :load clone.hs
    [1 of 1] Compiling Main             ( clone.hs, interpreted )
    Ok, modules loaded: Main.
    *Main> let clone1 = clone [1,2,3,4,5]
    *Main> clone1
    [1,2,3,4,5]

Now, on to something non-trivial: I want to extract, from our sample data, all Event values with a specific user ID. In clone, we took each value indiscriminately because we wanted all of them; in this case, we'll need to pattern match on the attributes of the Event. Not only that, but we'll have to find the user ID in the List of properties nested within the Event. These requirements will allow me to show an example with a little more substance than a "Hello, World" example.

Recall the format of our Event:

Event 1320512200548 "java.lang.String" 1293 "NPE in substring()" [("userId","smith"),("sessionId","ABCD1234")]

Let's say we are interested in extracting every Event with a specified user ID, and let's call the function getMessagesForUser. For each Event, once we determine that the user ID matches, we append the itemit to a List which will be the function's return value.

The way I approached this is to recognize that I first need a function that will iterate over a list of key-value pairs, searching for a key of userId and a value (actually, the second element of a 2-tuple) matching the passed-in user ID. I'll call this function matchesUserId.

Before I get started, though, I'm going to revisit my original algebraic data type Event and make it a little more user-friendly. The reason: I want to easily access the fields of an Event without having to write boilerplate accessor functions. By boilerplate I mean something like the following, to access the timestamp of an Event:


    timestamp (Event ts _ _ _ _) = ts

where you would have to supply one of these for each field you want to access. Instead, we can use Haskell record syntax to supply default accessors while defining the data type, which I modify here. While I'm at it, I'll perform a similar change to Property:


    data Property = Property {
      key :: Key,
      value:: Value }
      deriving (Show)

    data Event= Event {
      timestamp :: Timestamp,
      className :: ClassName,
      lineNumber :: LineNumber,
      message :: Message,
      properties :: Properties }
      deriving (Show)

Having done this, I'll need to change my initialization of my Property values, as they are now no longer simple 2-tuples:


    prop1 = Property "sessionId" "ABCD1234"
    prop2 = Property "sessionId" "EFGH5678"
    prop3 = Property "sessionId" "WXYZ9876"
    prop4 = Property "userId" "smith"
    prop5 = Property "userId" "adams"
    prop6 = Property "userId" "jobim"

Remember that my eventProcessor.hs file creates a List of sample Event objects. Now I can use the head function to retrieve the first Event in the list, and then use the new timestamp accessor function to retrieve the timestamp, as in the following. Note also that timestamp's type signature ("takes an Event; returns a Timestamp") can be displayed in ghci:


    *Main> let evt1 = head eventList
    *Main> evt1
    Event {timestamp = 1320512200548, className = "java.lang.String", lineNumber = 1293, message = "NPE in substring()", properties = [("userId","smith"),("sessionId","ABCD1234")]}
    *Main> :type timestamp
    timestamp :: Event -> Timestamp
    *Main> timestamp evt1
    1320512200548

Now we can go back to our function matchesUserId. I decided that before I create this function, I need an additional small function that looks at a single Property and returns True if the key is userId and the value is the passed-in user ID. I called this function isUserId and placed it in the eventProcessor.hs file, directly below the Event data type definition:


    isUserWithId :: Property -> String -> Bool
    isUserWithId prop id =
      if (key prop == "userId" && value prop == id)
        then True
        else False

Note the indentation. I have defined the type signature in a way that we'll need to discuss later, when we talk about partial functions and currying. Suffice to say the function takes a Property and a String and returns a Bool. We can test this function against some of the properties I create in this file:


    *Main> prop1
    Property {key = "sessionId", value = "ABCD1234"}
    *Main> isUserWithId prop1 "smith"
    False
    *Main> prop4
    Property {key = "userId", value = "smith"}
    *Main> isUserWithId prop4 "smith"
    True
    *Main> prop6
    Property {key = "userId", value = "jobim"}
    *Main> isUserWithId prop6 "smith"
    False

With that function completed, we can work on containsUserPropertyWithValue, which looks for the user ID in a List of properties. Here's what I came up with:


    containsUserPropertyWithValue :: [Property] -> String -> Bool
    containsUserPropertyWithValue (x:xs) id =
      if (isUserWithId x id)
        then True
        else containsUserPropertyWithValue xs id
    containsUserPropertyWithValue [] _ = False

and we can test this against a couple of the Property Lists that we created in our sample data:


    *Main> containsUserPropertyWithValue propList3 "jobim"
    True
    *Main> containsUserPropertyWithValue propList3 "smith"
    False

Before we move on, note that Haskell has a where clause, in which I can add my isUserWithId function directly to this function. It makes sense in this case; I am unlikely to need this function anywhere else besides the containsUserPropertyWithValue function. Let's try this:


    containsUserPropertyWithValue :: [Property] -> String -> Bool
    containsUserPropertyWithValue (x:xs) id =
      if (isUserWithId x id)
        then True
        else containsUserPropertyWithValue xs id
      where
        isUserWithId prop id =
          if (key prop == "userId" && value prop == id)
            then True
            else False
    containsUserPropertyWithValue [] _ = False

and I can verify that everything works as before, except that I now no longer have an isUserWithId function in scope:


    *Main> isUserWithId prop4 "smith"

    :1:1: Not in scope: `isUserWithId'

This function was defined within another function only and so is no longer available to me as a standalone function.

The final step is to use this function while iterating over the entire list of Events. As I described earlier, we won't be iterating in the usual sense. First I'll experiment with using a recursive function. On each call, the function will add an Event to the list if it matches the specified user ID. Here's what my first cut looks like:


    listEventsForUser :: [Event] -> String -> [Event]
    listEventsForUser (x:xs) id =
      if (containsUserPropertyWithValue (properties x) id)
        then x : listEventsForUser xs id
        else listEventsForUser xs id
    listEventsForUser [] _ = []

If I run this against the sample set of events I generate in this file, I get exactly what I expected:


    Prelude> :load eventProcessor.hs
    [1 of 1] Compiling Main             ( eventProcessor.hs, interpreted )
    Ok, modules loaded: Main.
    *Main> let evtList1 = listEventsForUser eventList "adams"
    *Main> evtList1
    [Event {timestamp = 1320513130333, className = "com.adamsresearch.jarview.JarView", lineNumber = 388, message = "fileNotFound",
properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]},
    Event {timestamp = 1320513257324, className = "com.adamsresearch.jarview.ArchiveSearch", lineNumber = 193, message = "search started13255342",
properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]},
    Event {timestamp = 1320512260122, className = "com.adamsresearch.jarview.JarView", lineNumber = 725, message = "search started",
properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]},
    Event {timestamp = 1320512263122, className = "com.adamsresearch.jarview.JarView", lineNumber = 779, message = "search completed",
properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]}]
    *Main>

(with a little bit of output formatting to make it easier to see that the correct Events have been selected).

This post has become chapter-length (although note how compact the actual working code is!), but I have one more topic to discuss. FP languages like Haskell recognize the above pattern occurs so often that there is built-in support in the language to perform operations on every element in a List, leading to extremely compact but very readable code. For example, there is the Haskell map function, which takes a function as an argument and applies that function to each element in a list. Some such functions return Lists, while others reduces Lists to a scalar value. What would be helpful in our case is the filter function, which applies a function to each List element and returns a List of items for which the filter is true.

Here is how I would have used filter in this example. First, I would like to define a function that would apply to an entire event. To make things a little more compact, I will include (in nested where clauses) my previously-defined functions, as follows:


    eventGeneratedByUser :: String -> Event -> Bool
    eventGeneratedByUser id event =
      if (containsUserPropertyWithValue (properties event) id)
        then True
        else False
      where
        containsUserPropertyWithValue (x:xs) id =
          if (isUserWithId x id)
            then True
            else containsUserPropertyWithValue xs id
          where
            isUserWithId prop id =
              if (key prop == "userId" && value prop == id)
                then True
                else False
        containsUserPropertyWithValue [] _ = False

This is a good time for a "teaser" on partial functions. Note how I defined my function signature -- I put the String user ID first, then the Event. The reason for this organization is that I want to use Haskell's filter, which takes a predicate. In most examples, the predicate is something simple like the Haskell odd function, which needs no arguments. If you say filter odd [1,2,3,4,5,6], you get back [1,3,5].

What do you do if you need to pass an argument to a function (in this case, the user ID) to create the predicate? The answer lies in that odd signature, in this case


    eventGeneratedByUser :: String -> Event -> Bool

I pass in a user ID and an Event. In that order, specifically. You've probably noticed by now that the signature (whether I define it, or request it from ghci) is not


    eventGeneratedByUser :: String  Event -> Bool

as you would expect from a function taking two arguments. The function appears (from the actual signature) to be a series of single-argument function invocations. The reason: all functions in Haskell actually take only one argument! Someone might correct my description, but I look on it this way: this signature defines a function that takes a String, and that newly defined function is itself a function that takes an Event. In my case, this helps a lot, because in Haskell you can take a multi-argument function and create a partial function for which you have defined some but not all of the arguments. Hence the choice for my ordering of my arguments. That ordering allows me to create a predicate which is a partial function where "user ID" has already been defined:


    *Main> let filterPred = eventGeneratedByUser "adams"

Since I have not fully specified the arguments to this function, what I get back is a function which requires only an Event to return True or False. Now I have the predicate I need for filter, and here is how I use it:


    *Main> let eventsForAdams = filter filterPred eventList
    *Main> eventsForAdams
    [Event {timestamp = 1320513130333, className = "com.adamsresearch.jarview.JarView", lineNumber = 388, message = "fileNotFound", properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]},
    Event {timestamp = 1320513257324, className = "com.adamsresearch.jarview.ArchiveSearch", lineNumber = 193, message = "search started13255342", properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]},
    Event {timestamp = 1320512260122, className = "com.adamsresearch.jarview.JarView", lineNumber = 725, message = "search started", properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]},
    Event {timestamp = 1320512263122, className = "com.adamsresearch.jarview.JarView", lineNumber = 779, message = "search completed", properties = [Property {key = "userId", value = "adams"},Property {key = "sessionId", value = "EFGH5678"}]}]

If I could not define a partial function in Haskell, I could not have used the Haskell filter function for this exercise, as it takes a single argument -- the predicate. This predicate (by now a single-argument function looking for an Event) is applied to each element of the Event List, and I get exactly what I wanted.

I really enjoyed creating this post. For those of you who are at the same level of FP understanding as I am, I hope you found this post both interesting and helpful.

Breakpoint processing in JDI

2011-11-03T21:27:00.000-07:00

I've been working on a JDI (Java Debug Interface) project lately and have been posting helpful tips as I go along. It has been a few years since I've worked with this API, but although I know there have been a few enhancements, the API is quite consistent with what I remember. In this post I'm going to discuss how to use the API to inspect variable values at a breakpoint.

As in a previous post, I'll be restricting myself to breakpoint requests (as opposed to method-entry, method-exit, exception requests, etc.), mainly because I know they can be processed rather quickly in near-real-time. Some other types of exceptions significantly slow down the target application, even if you process them as quickly as practical and immediately resume all threads. In a typical, interactive debugger, a human is in the loop, inspecting variables and making decisions, and a speed difference between JDI event types would not be noticeable. But in the application I'm developing, I want to pause execution just long enough to collect data and then continue execution immediately. So I'm going to use only breakpoint requests for my project.

Finding Values at a Breakpoint

Given an instance vm of a VirtualMachine, you might configure your event-consuming code as follows:


EventQueue evtQueue = vm.eventQueue();
while(true)
{
 EventSet evtSet = evtQueue.remove();
 EventIterator evtIter = evtSet.eventIterator();
 while (evtIter.hasNext())
 {
   try
   {
     Event evt = evtIter.next();
     EventRequest evtReq = evt.request();
     if (evtReq instanceof BreakpointRequest)
     {
       BreakpointRequest bpReq = (BreakpointRequest)evtReq;
       BreakpointEvent bpEvt = (BreakpointEvent)evt;
       ThreadReference threadRef = bpEvt.thread();
       StackFrame stackFrame = threadRef.frame(0);
       ...

At this point, you can find variables in two places:

The stack (hence the retrieval of the StackFrame); for example, local variables in a method;
The current ObjectReference, for static and instance variables.

If, at a breakpoint, you are searching for the value of a local variable, let's call it varName, you could find it with the following snippet of code:

  List visVars = stackFrame.visibleVariables();
for (LocalVariable visibleVar: visVars)
{
  if (visibleVar.name().equals(varName))
  {
    Value val = stackFrame.getValue(visibleVar);
    ...

As I've mentioned before, the JDI API isn't big on classes with constructors; most of what you see is interface definitions, with impls supplied behind the scenes. So, you would not instantiate a Field in JDI and search for it; instead, you drill down into the API and compare some aspect of the object you want with the data you have at hand. For example, you list all the LocalVariables on the stack and then compare their .name()s with the name of your target variable.

If you don't find your variable on the stack, then you can check the object's fields (I prefer to check the stack first). To get object field values, you would first get the ObjectReference and then query it:


ObjectReference objRef = stackFrame.thisObject();
Value targetVal = objRef.getValue(field...

Halt. An ObjectReference will get a value for you, but it requires a Field object, for which there is no constructor. This is just the issue I was discussing above. Just where do you get a Field object? You query the available Fields of the object's ReferenceType first, then pass those Field references to the object to get their values for this instance:


ObjectReference objRef = stackFrame.thisObject();
ReferenceType refType = objRef.referenceType();
List<Field> objFields = refType.allFields();
for (int i=0; i<objFields.size(); i++)
{
  Field nextField = objFields.get(i);
  if (nextField.name().equals(varName))
  {
    Value targetVal = objRef.getValue(nextField);
  }
}

So there are your values at the breakpoint, obtained either from the stack or the instance fields of the ObjectReference.

Working with JDI Values

The Value that you get from JDI requires a little effort to be of practical use. For example, if the Value is an instance of a StringReference, you'll need to cast it to a StringReference, then call .value() to get a java.lang.String value. If it's an instance of an IntegerValue, then you'll have to cast the Value to an IntegerValue and call .value(), which in this case will return an int! I find this process to be a little tedious (note: .toString() doesn't return the value!). In my case, I'm going to be generating either log output or key-value pairs for each breakpoint event, so my approach is just to write a convenience method that hides the details and provides the value in String form.

An interesting case is when the Value is itself an ObjectReference. Just as you can expand variables down to their fields in a typical debugger, you can also choose to drill down into objects at a breakpoint and retrieve their fields, to an arbitrary level of depth. The same process can be used for other JDI types, such as StringReference.

These fields might not be quite as familiar as the methods you are more accustomed to using. For example, if you are logging the size of Strings that are being generated in your target program, you can't call .length() on the StringReference, but you can get its instance fields and retrieve count, which is the name of the field holding the String's length.

Practical Considerations

While it is possible to drill down to an arbitrary depth in the stack or in the instance's fields, keep in mind that you have halted your application while this is occurring. It's best to keep processing during this period to a minimum. While you could spawn a thread to finish processing the breakpoint (for example, creating and sending a JMS message), you will need to retrieve all the variables from the stack and instance while the target application is halted. So that drill-down should be quick.

That caveat aside, you can then specify variables in some custom format, such as dot-notation, and then drill down into the fields/variables recursively until you find the value you want. For example, you could have a variable specified as fileName.count, in which case you would locate the fileName StringReference, then get the instance fields of fileName and find count, which is the length of the file name.

For fields like String's count, which might not be an obvious choice to someone configuring a breakpoint specification, some type of browser would be useful. Browsing the stack isn't an option until you've already set breakpoints and reached a breakpoint, as the stack frame isn't in scope until the breakpoint is reached. But for static and instance fields, you only need a ReferenceType to list the available Fields, to some configurable depth, before you even start debugging the application. Since these Fields are available in JDI as soon as the class is loaded, you don't have to halt program execution just to get the static and instance field names for the class.

Haskell From an OO Developer's Perspective, Part 2

2011-10-29T09:17:00.001-07:00

Today I'm looking at Haskell type definition and the use of pattern-matching in functions. Pattern-matching is much more an integral feature of FP, as opposed to OO. But first...

Haskell type definition

According to the HaskellWiki, an algebraic data type is a type which specifies the shape of the elements. One way to approach this topic, if you tend to think in OO, is 1) think C structs, and 2) try to forget behavior, for a moment. Even in OO, we've all been there -- sometimes you really want something like a struct just to organize some data, and the only behavior (in an OO sense, meaning "methods") that you really want is a set of accessors/mutators (getters/setters). So you either swallow a little hard and make the attributes public, or you blast out boilerplate getters/setters. Meanwhile, some people write frameworks of code that automagically expose your attributes, and so on.

Right now we just want to define a data structure. We'll be manipulating the structure with functions later, so we don't care about what it "does". It's just there. Here's an example of a data-type declaration in Haskell; I'm using it to describe some breakpoints I've serialized from a Java program:


-- file:  c:/HaskellDev/typesDemo.hs

type Label = String
type FileName = String
type LineNumber = Int
type MethodName = String
type ExceptionName = String
type Description = String

data Breakpoint = LineNumberBreakpoint Label FileName LineNumber Description |
MethodBreakpoint Label FileName MethodName Description |
ExceptionBreakpoint Label ExceptionName Description
deriving (Show)

This isn't the simplest-possible example, but it's representative of a useful structure and allows me to introduce a few things at once. First, focus on the data keyword; data is the beginning of the definition of an algebraic data type and is followed by the name of the type (which must start with a capital letter). This type name is followed by an equals sign and the type-constructor definition. "deriving (Show)" isn't critical to understanding the type; it provides information ghci needs to output the values we'll create based on this type definition.

The type constructor definition is composed of one or more value constructors. These value constructors must also be labeled by a name beginning with a capital letter and are separated by the pipe "|" symbol. Value constructors and type constructors reside in different "namespaces", so to speak, so if you have a really simple data type, you can use the same label for both the type definition name and the value constructor name and not worry about collisions between the two names. For example:

type User = User String String String Int

Back to our example, let's first get out of the way the list of types at the beginning of the source file. These types are called type synonyms in Haskell and are nothing elaborate; they are available to improve the readability of your code. So, if I want to describe a breakpoint definition as consisting of a label, the name of the file, the line number in the code, and a short description, in Haskell I could say

data Breakpoint = Breakpoint String String Int String

or I could improve the readability/understandability of the definition with some type synonyms:


type Label = String
type FileName = String
type LineNumber = Int
type Description = String

data Breakpoint = Breakpoint Label FileName LineNumber Description

and that's what I've done in our example. In addition, I've defined three different breakpoint types: one based on line number, one based on method call (e.g. method-entry, method-exit) (yes, we're using Haskell to analyze breakpoints in a Java class!), and one based on thrown exceptions.

Going back to our example, the next question would be "what do you do with this type definition?" At this point I want to launch the Glasgow interpreter/debugger, ghci. Once this is up and running, I can load my file. Note that ghci loads by default from the directory in which it was launched; you can set the directory from which it loads with :cd dirName:


*Main> :cd c:/HaskellDev
*Main> :load typesDemo.hs
[1 of 1] Compiling Main             ( typesDemo.hs, interpreted )
Ok, modules loaded: Main.
*Main>

I now have three value constructors for my Breakpoint data type, and I can create instances of each. I do this by referencing the name of the value constructor and providing the arguments that it expects, as specified in the type definition. Additionally, in ghci, to assign a variable a value, we need to use the let keyword. First, imagine that we want to create an exception breakpoint, but all we remember is the name of its value constructor. In ghci, you can get the type info for this value constructor:


*Main> :type ExceptionBreakpoint
ExceptionBreakpoint
:: Label -> ExceptionName -> Description -> Breakpoint

An explanation of the chain of -> symbols will have to wait until a later day, as it is a little foreign to the OO mind. But for now we can use the output to remind ourselves what we need to supply to the value constructor. Let's create an ExceptionBreakpoint, output its value, then use ghci's :type to output its type:


*Main> let bp3 = ExceptionBreakpoint "NPE" "java.lang.NullPointerException" "Break on all NullPointerExceptions"
*Main> bp3
ExceptionBreakpoint "NPE" "java.lang.NullPointerException" "Break on all NullPointerExceptions"
*Main> :type bp3
bp3 :: Breakpoint

Note that :type returns the actual type of the breakpoint, not the name of the value constructor which was used to build it. There are times when we want to know which value constructor was used -- an excellent segue to the 2nd topic of this post.

Pattern-matching in functions

Suppose I want to write a function that extracts the description of a breakpoint definition and further, reminds me what type of breakpoint it is. In Haskell, you accomplish this with pattern matching. In the following fragment:

describeBreakpoint (ExceptionBreakpoint _ _ desc) = desc

I have defined a function -- describeBreakpoint -- which matches only on an ExceptionBreakpoint. I've written it to match on a tuple which consists of

the name of the value constructor
two wildcards ("_", placeholders) for the values I'm not currently interested in (the Label and the ExceptionName)
a parameter for the description, desc.

After the equals sign is the value returned from the function, which is just the description. Applying this function to the breakpoint we created earlier, I get:

     *Main> describeBreakpoint bp3
    "Break on all NullPointerExceptions"

This process is called deconstruction, as we've essentially reversed the process we used to create bp3, by using a pattern which looks like the pattern used to construct the breakpoint value.

Haskell allows us to define a function as a series of predicate/value pairs. For example, we can update our function to extract the description of any Breakpoint as follows:

   describeBreakpoint (LineNumberBreakpoint _ _ _ desc) = desc
  describeBreakpoint (MethodBreakpoint _ _ _ desc) = desc
  describeBreakpoint (ExceptionBreakpoint _ _ desc) = desc

If we did not do this, we would get an error if we tried to use the function to describe a LineNumberBreakpoint:

    *Main> let bp4 = LineNumberBreakpoint "parseParam" "Parser.java" 79 "Break at beginning of parse"
   *Main> describeBreakpoint bp4
   "*** Exception: typesDemo.hs:15:1-56: Non-exhaustive patterns in function describeBreakpoint

Note that in general FP functions prefer that your matching patterns be exhaustive and will complain if they exhaust the list without a match. In this case the message should be self-explanatory. After updating our function as described earlier (and, reloading in ghci, which will require that you re-define the breakpoints), we get the expected answers:

*Main> :load typesDemo.hs
[1 of 1] Compiling Main             ( typesDemo.hs, interpreted )
Ok, modules loaded: Main.
*Main> let bp3 = ExceptionBreakpoint "NPE" "java.lang.NullPointerException" "Break on all NullPointerExceptions"
*Main> let bp4 = LineNumberBreakpoint "parseParam" "Parser.java" 79 "Break at beginning of parse"
*Main> let bp5 = MethodBreakpoint "initBean" "Parser.java" "init()" "Break at parser init() method"
*Main> describeBreakpoint bp3
"Break on all NullPointerExceptions"
*Main> describeBreakpoint bp4
"Break at beginning of parse"
*Main> describeBreakpoint bp5
"Break at parser init() method"

Note that the matches above treat the value as a simple tuple, where the first element is the name of the value constructor and subsequent elements are simply the values passed to the value constructor. Pattern matching can be performed on any tuple, in general. For example, suppose we write a function that returns the 2nd element of a 4-tuple:

     secondElement (a, b, c, d) = b

and then try this:

*Main> secondElement ("Washington", "Lincoln", "Fillmore", "Roosevelt")
"Lincoln"

That's great. Now look at this:

*Main> secondElement ("Washington", "Lincoln", "Fillmore")

:1:15:
   Couldn't match expected type `(t0, t10, t20, t30)'
               with actual type `(t1, t2, t3)'
   In the first argument of `secondElement', namely
     `("Washington", "Lincoln", "Fillmore")'
   In the expression:
     secondElement ("Washington", "Lincoln", "Fillmore")
   In an equation for `it':
       it = secondElement ("Washington", "Lincoln", "Fillmore")

While there is a second element in this list, it is a 3-tuple and there is no pattern in the function secondElement which matches a 3-tuple. We could add another match in the function definition to handle a 3-tuple (and so on), but there are better ways to do this, which we may investigate in later posts.

As with all other topics I'll be investigating in these early posts, this one is far from exhaustive, and it isn't intended to be. Again, this discussion reflects what stands out to me, as an OO developer, looking at all of this for the first time. Next post: more on thinking in FP.

Haskell From an OO Developer's Perspective, Part 1

2011-10-24T22:35:00.000-07:00

I'm learning Haskell by following O'Sullivan, Goerzen and Stewart's Real World Haskell. I've been writing object-oriented code for well over half my career as a developer, and there are things about functional programming that really stand out to me specifically because of my OO background. My goal is to capture that experience as I learn, because I haven't been able to find much material that speaks to the unique experience of transitioning from OO to FP. So here goes.

As you can guess, functions are highly regarded in functional programming. There is an emphasis on "purity", something you don't hear much about in OO. Simply put, a pure function does its work without side effects. Repeatedly invoking a pure function with the same arguments will always yield the same result.

When you hear this topic discussed in FP, it frequently includes references to mutable state. For example, if invoking your function changes the state of a variable, your function is not pure. In fact, if your function even references a global variable, it is not pure. Why? Because you can't guarantee that invoking your function repeatedly, with the same arguments, will yield the same results, because it relies on state over which it has no control.

In functional programming, the goal is to keep code as pure as possible and to isolate impure code. One advantage of pure code is in testing, as you don't have to worry about the effect of other threads on the results produced by your code. In addition to code that employs mutable state, other examples of impure code are: any I/O, pinging a web service for that live data feed, and so on. As I go through my self-training, I will be very interested in how useful an FP application will be in my field. I write mostly middle-tier Java code with a lot of interfaces to web services, and I have to admit at this stage I'm a little skeptical. Will I write a big chunk of FP code that's 99% "impure" and feel that I just performed the equivalent of porting a Fortran program to C++? I hope not. But it's way too early to pass judgment, and I expect a far happier outcome.

Anatomy of a Haskell function

Functions in Haskell look a lot different than Java functions. I've spent a few months trying to become proficient in Erlang, and so the structure of an FP function is starting to become familiar to me. Here's an example Haskell function:


agentName code =
if code == 86
then "adams"
else if code == 99
then "feldon"
else "unknown"

Given an agent's code number, it returns the agent's (real) name. In general, a function consists of:

The name of the function
The argument names, separated by whitespace
The equals "=" sign
The expression that your function evaluates and returns

Note that the example does not have a return statement. In this example, the "if" statement itself is the value returned by the function. For this reason, you cannot leave an unaccounted-for branch of the if statement. In other words, if you try to load the following function definition into the interactive Glasgow Haskell compiler:

agentName code =
if code == 86
then "adams"
else if code == 99
then "feldon"

you will see the following:

     agentName.hs:7:1: parse error (possibly incorrect indentation)

the reason being that there are some inputs for which the function will not be able to return a value at all. If you've ever declared a non-void Java function and forgot to return a value from a branch of an if statement, then this error message seems reasonable, if not quite as informative.

If we go back to the original version of the function, we can test its behavior:

Prelude> :load agentName.hs
[1 of 1] Compiling Main             ( agentName.hs, interpreted )
Ok, modules loaded: Main.
*Main> agentName 86
"adams"
*Main> agentName 99
"feldon"
*Main> agentName 007
"unknown"

Note, in the examples, the indentation, which is required when a statement (in this case, the function definition) extends across a line boundary. The size of the indentation and the characters used (space, tab) are not important, as long as the indentation is there.

There is obviously a lot more to Haskell functions, of course. The above points are what stand out to me as a mostly-OO developer.

I've been using the Glasgow Haskell Compiler and, in the examples above, the interactive interpreter (and debugger) ghci. Although I'm generally following Real World Haskell, I don't really expect to review the entire book in slow-motion on a blog (who would read that?). So I may not write a post about defining types unless it seems unusually foreign to OO. My ultimate goal is to become proficient enough to provide useful and practical advice, oriented toward an OO developer wanting to learn to write production-quality FP code.

OO to FP: Transition

2011-10-23T18:57:00.000-07:00

For the last year or so, I've been trying to come up to speed on functional programming, studying bits and pieces here and there. One interesting source was Bruce Tate's Seven Languages in Seven Weeks, which included a number of FP languages. I reviewed the book on this blog -- in seven days -- a few months ago, if that tells you how much time I've devoted to this topic in general! I've decided to get a little more serious now.

Since I've been doing primarily object-oriented development for many years, I have a perspective that is different than someone just learning to program. I think that a person with an OO background might need a slightly different introduction to FP than someone coming in with a clean slate -- it certainly seems that way to me, as I don't feel a lot of the "intro" material emphasizes the differences that an OO person would find important. The point of my posts on this topic is to capture my own experience learning FP before I forget what my "angle" was (and then myself become useless to an OO person about to go through the same process).

This experience reminds me of my last development paradigm shift, when a number of my co-workers and I decided to learn object-oriented analysis, design and programming (in those pre-agile days, huge emphasis was placed on the difference between those three phases!). I worked in a research lab, and we wrote mostly in C and Fortran. Some of use wanted to become really grounded in OOA/D/P before we started writing code (I was in this camp) and some of us just wanted to start hammering out code. I could not understand these people in the latter camp. How could you develop object-oriented code without learning what object-oriented analysis is? And you're not even going to learn about OOD? You're skipping the analysis and design phases? You're just going to be writing Fortran code in C++! Ahh! I was young and thought I would live forever. Actually, I thought about death a lot (actually, I didn't).

We used Smalltalk and C++. Although I was really happy with my Smalltalk tic-tac-toe program (on a Mac), I knew from the beginning that this platform was not going to rule the OO world. C++ had a lot going for it (and still does): it was a bridge technology from C to object orientation, and the results speak for themselves. Of course Java has been hot for many years, too, but frankly I suspect that the success C++ had in bridging procedural developers to OO may have been the big enabler for Java.

One thing about FP: I'm not sure I see an equivalent bridge technology; not an out-of-the-park winner, at least. I appreciate the FP-inspired languages that run on the JVM; the JVM seems like a potential bridge technology. For example, I could probably convince my co-workers to accept some Scala code on our Java platform. But I don't feel very enthused working with Scala. If you read my posts on Seven Languages in Seven Weeks, you know I like Erlang a lot. I've spent a lot of time learning Erlang, but I'm just now giving up on it for a number of reasons, the main one being that I just don't see it becoming a Java or a C++. I don't feel particularly competent in my ability to predict which FP language is likely to prevail, though. I won't be surprised by what happens, but I also know it won't be something I could have predicted, as things seem a little less clear-cut than the competition between OO languages years ago.

So, here is what I have decided to do: I'm going to learn Haskell, using the O'Reilly book Real World Haskell. My goal is to get a good education in functional programming and not to worry at this time about who's going to "win", and I feel Haskell is a great language to choose to learn FP. As I learn, I'll collect my thoughts and mention what jumps out at me, as an OO person, and hopefully some other OO people will find the commentary useful.

I don't hold strong opinions on any of these topics. Mostly I just want to learn FP and hopefully help other OO developers do the same. I would also like to be especially proficient in one FP language, so I've made my choice. If you are interested in Haskell, or are a Java developer interested in FP, I hope I provide something useful to you. My first post is coming in a couple of days.

JDI: three ways to attach to a Java process

2011-10-15T08:48:00.000-07:00

If you've looked at my recent posts, you know I'm working on a plugin for VisualVM, a very useful tool supplied with the JDK. In one example, I showed how to attach to a waiting Java application using a socket-based AttachingConnector. At that time I said that there were two primary ways of attaching to a process with JDI -- via shared memory, and with a socket.

It turns out there is a "third way". Following is an example of why this way is useful, and why it was provided.

When I last wrote JDI programs (in Java 5), I would notice that my target application would start up and print (to stdout) the port on which it was listening, as in the following:

Listening for transport dt_socket at address: 55779

In Java 5, if you detached your debugger from this process, you would get another line to stdout in the target's console, like this:

Listening for transport dt_socket at address: 55779

and this would go on for as long as you chose to attach and detach, etc.

At some point (and I don't know when this started happening), the port on which the target is listening started changing on each detach of an external debugger. If in Java 6 (I'm using u20), you repeatedly attach and detach from the target process, you'll see the following out in the target's console:

Listening for transport dt_socket at address: 55837
ERROR: transport error 202: recv error: Connection reset by peer
Listening for transport dt_socket at address: 55844
ERROR: transport error 202: recv error: Connection reset by peer
Listening for transport dt_socket at address: 55846
ERROR: transport error 202: recv error: Connection reset by peer
Listening for transport dt_socket at address: 55911

If you're writing an application that attaches using the debug port, each time you attach you need to find out what port the target is using. This information is not available from the process itself; in other words, you have to play the usual unpleasant game of capturing console output to know what the port is. Even if you specify a port at target start, you still need to get your hands on the value.

You can still find the original request for a feature to attach to a process by its process ID if you search around the old Java bug reports. The long and short of it: a new AttachingConnector was created, one which attaches by PID. As you know, sometimes it isn't much fun finding a process's PID either. In my case, however, I am writing a plugin for VisualVM, and one thing you get for free when you do that is Visual VM's API, which as you might expect includes calls to get the PID. My goal, then, is to use this new connector in my VisualVM plugin, and I thought it might be appreciated if I shared the details.

I've adapted my test program from an earlier post so that it now outputs the details of each AttachingConnector; the changed code fragment is shown here:

List<AttachingConnector> attachingConnectors = vmMgr.attachingConnectors();
for (AttachingConnector ac: attachingConnectors)
{
  Map paramsMap = ac.defaultArguments();
  Iterator keyIter = paramsMap.keySet().iterator();
  System.out.println("AttachingConnector:  '" + ac.getClass().getName() + "'");
  System.out.println("  name: '" + ac.name() + "'");
  System.out.println("  description: '" + ac.description() + "'");
  System.out.println("  transport name: '" + ac.transport().name() + "'");
  System.out.println("  default arguments:");
  while (keyIter.hasNext())
  {
    String nextKey = keyIter.next();
    System.out.println("    key: '" + nextKey + "'; value: '" + paramsMap.get(nextKey) + "'");
  }
}

The output from this code is shown below:

AttachingConnector:  'com.sun.tools.jdi.SocketAttachingConnector'
 name: 'com.sun.jdi.SocketAttach'
 description: 'Attaches by socket to other VMs'
 transport name: 'dt_socket'
 default arguments:
   key: 'timeout'; value: 'timeout='
   key: 'hostname'; value: 'hostname=AdamsResearch'
   key: 'port'; value: 'port='
AttachingConnector:  'com.sun.tools.jdi.SharedMemoryAttachingConnector'
 name: 'com.sun.jdi.SharedMemoryAttach'
 description: 'Attaches by shared memory to other VMs'
 transport name: 'dt_shmem'
 default arguments:
   key: 'timeout'; value: 'timeout='
   key: 'name'; value: 'name='
AttachingConnector:  'com.sun.tools.jdi.ProcessAttachingConnector'
 name: 'com.sun.jdi.ProcessAttach'
 description: 'Attaches to debuggee by process-id (pid)'
 transport name: 'local'
 default arguments:
   key: 'pid'; value: 'pid='
   key: 'timeout'; value: 'timeout='

A couple of things I hadn't noticed before is that the socket-based connector comes with the hostname argument pre-set to my machine's hostname, and that all three connectors have a timeout default argument. The first observation brings up an interesting point: if you use the local, PID-based connector, remember that you'll only be attaching to processes on your debugger's host.

I changed my test program to use the local connector and it works as before! Well, no, actually, it does not. Here's what I now get:

java.lang.UnsatisfiedLinkError: no attach in java.library.path
Exception in thread "main" java.io.IOException: no providers installed
at com.sun.tools.jdi.ProcessAttachingConnector.attach(ProcessAttachingConnector.java:86)
at com.adamsresearch.jdiDemo.JDIDemo.main(JDIDemo.java:70)

Does this mean the local connector isn't exactly ready for use? No, but I have been burned by the same issue that has plagued a number of others (scroll down in that page -- the issue was found by a reader of that post and was solved, partially, by another reader of that post). I'm working on a Windows platform, and when you do that you have to be a little careful ;-> . In this case, the problem is caused by 1) using the java interpreter as found on the system path, and 2) not making sure that path points directly to your JDK or JRE directory. The executable will look in a path relative to itself for the needed libraries, and when Windows copies the java executable to C:\Windows\system32 (or similar) -- and if you use that executable -- that relative path is broken. I believe this is the true issue, unlike described in the comments on the above post, where the distinction is made between using the JRE java and the JDK java. I don't think that's the issue. For example, below are the results of my attach test in 3 different scenarios:

Using java from my path, the first hit of which comes from C:\Windows\system32:


java -cp c:\jdk1.6.0_20\lib\tools.jar;. com.adamsresearch.jdiDemo.JDIDemo 10816 863 fileName
...
java.lang.UnsatisfiedLinkError: no attach in java.library.path
Exception in thread "main" java.io.IOException: no providers installed
 at com.sun.tools.jdi.ProcessAttachingConnector.attach(ProcessAttachingConnector.java:86)
 at com.adamsresearch.jdiDemo.JDIDemo.main(JDIDemo.java:70)

Using the full path to the JRE bin java:


c:\jdk1.6.0_20\jre\bin\java -cp c:\jdk1.6.0_20\lib\tools.jar;. com.adamsresearch.jdiDemo.JDIDemo 10816 863 fileName
...
Attached to process 'Java HotSpot(TM) 64-Bit Server VM'

Using the full path to the JDK bin java:

c:\jdk1.6.0_20\bin\java -cp c:\jdk1.6.0_20\lib\tools.jar;. com.adamsresearch.jdiDemo.JDIDemo 10816 863 fileName
...
Attached to process 'Java HotSpot(TM) 64-Bit Server VM'

As you can see, the above seems to support my theory that it's not the JRE vs the JDK, but rather the context-poor placement of the java executable in the "usual" Windows binaries directory, that caused the problem. That posting is several years old, so it is possible that at that time, the needed JDI libraries actually were not included in the JRE, but it is clear that today, you will see the same exception if you use the java executable found in Windows' default binaries directory.

Now, if I run my JDI application against my JarView utility, searching for AttachingConnector in the JDK installation directory, I get the following output:

Breakpoint at line 863:
fileName = 'AttachingConnector.class'
Breakpoint at line 863:
fileName = 'GenericAttachingConnector$1.class'
Breakpoint at line 863:
fileName = 'GenericAttachingConnector.class'
Breakpoint at line 863:
fileName = 'ProcessAttachingConnector$1.class'
Breakpoint at line 863:
fileName = 'ProcessAttachingConnector$2.class'
Breakpoint at line 863:
fileName = 'ProcessAttachingConnector.class'
Breakpoint at line 863:
fileName = 'SharedMemoryAttachingConnector$1.class'
Breakpoint at line 863:
fileName = 'SharedMemoryAttachingConnector.class'
Breakpoint at line 863:
fileName = 'SocketAttachingConnector$1.class'
Breakpoint at line 863:
fileName = 'SocketAttachingConnector.class'

and so have done what I set out to do, which is 1) debug-attach by process ID, and 2) thrash through the inevitable hiccups and share the solutions. Hopefully this will be useful to you, too.

Note: actually, there are even more ways to attach to a Java process. JPDA Connection and Invocation is the definitive guide, from Oracle. If you're going to be writing debuggers, you can't go wrong reading this page first.

Generating a minable event stream with JDI

2011-10-08T09:21:00.000-07:00

Say you've got a good-sized chunk of code, in production, that doesn't always act as expected but it does so often enough that everyone's willing to keep using it (including your customers). You have a lot on your to-do list, and you keep busy enough just handling the serious meltdowns, so much so that you don't really have time to investigate that occasional failed parse-and-load, or that mysterious stack trace that's supposed to be harmless. Besides: your application does so much; statistically, it can't get everything right all the time, can it?

For an issue that isn't easily repeatable but is considered to be a significant problem, running in the debugger can be demoralizing. What if you could produce the equivalent of a "robo-debugger", a process that would run the debugger for you, continuously, and wait with infinite patience for that rare occurrence? And then have the common sense to collect information off the stack and -- gasp -- even tell you about it? If this does not sound revolutionary, then good for you. Why do humans ever sit in front of a monitor, stepping through a debugger manually anyway? We've "manualized" an operation that should be automated.

Of course, logging could do the same thing for you. My interest in this idea arose when I was supporting an application for which I had the source code, but I was not permitted to modify it. I was allowed to recompile the application with the debug switch on, however, and I was allowed to attach with a debugger. After I wrote a JDI-based monitor for this application, I realized it had one additional advantage -- you don't have to add a lot of logging statements for an issue that might only need to be debugged once. Also note that code like this could be embedded into another application (for example, it could be a VisualVM extension) and be used to generate events on demand, another reason to skip the embedded logging statements.

Here's the general approach:

Ensure your target application is compiled with the -g switch.

Start the targeted application as usual, but listening on a port for a debugger connection.

Start your robo-debugger and attach to the target JVM.

Read a list of breakpoint specifications, each of which contains the following information:
- Class name and source line number.
- List of variables on the stack that you want to inspect.
- Optional message in the form of a formatted String with placeholders for said variables retrieved from the stack.
- Optional list of key-value pairs, values again being retrieved from the stack.

At each defined breakpoint, halt execution (briefly!) and generate an event, realized as a log (to file, JDBC, etc) message, JMS message, etc.

Mine the event stream from your application to solve all the issues that have been nagging you since you went to production.

This post will cover everything except the last item, which is the hard part. I also won't write logging or JMS code, as that's not relevant to the discussion. My example will generate some output to stdout.

To get started, pick a target application. I'll be using an application I wrote called "JarView" (just a simple Swing application to search through a directory of .jar files to find a missing class file).

Start the target application

There are two primary transports in JPDA (Java Platform Debugger Architecture): socket-based, and shared-memory-based. I'll start my application using socket-based JPDA and (transport=dt_socket), instruct it to wait for a debugger to attach to it (server=y) and do not suspend while waiting for a connection (suspend=n):

     c:\JarView>java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -cp jarview.jar JarView

You'll see a launch message like the following:

     Listening for transport dt_socket at address: 50069

Attach to the target

Write a program to

Use the JDI Bootstrap class to get an instance of a VirtualMachineManager.

Iterate over the VirtualMachineManager's list of AttachingConnectors until you find a connector supporting transport dt_socket.

Get the port Connector.Argument of the AttachingConnector and set it to the port on which your target application is listening.

Attach to the AttachingConnector and get an instance of a VirtualMachine.

An example piece of code (bare minimum, with no special exception handling) that will perform the above steps follows. You will need to compile and run with the JDK's lib/tools.jar on the classpath (this is not found in the JRE, by the way).

import java.util.List;
import java.util.Map;
import com.sun.jdi.Bootstrap;
import com.sun.jdi.VirtualMachine;
import com.sun.jdi.VirtualMachineManager;
import com.sun.jdi.connect.AttachingConnector;
import com.sun.jdi.connect.Connector;

public class JDIDemo
{
public static void main(String[] args) throws Exception
{
  VirtualMachineManager vmMgr = Bootstrap.virtualMachineManager();
  AttachingConnector socketConnector = null;
  List attachingConnectors = vmMgr.attachingConnectors();
  for (AttachingConnector ac: attachingConnectors)
  {
    if (ac.transport().name().equals("dt_socket"))
    {
      socketConnector = ac;
      break;
    }
  }
  if (socketConnector != null)
  {
    Map paramsMap = socketConnector.defaultArguments();
    Connector.IntegerArgument portArg = (Connector.IntegerArgument)paramsMap.get("port");
    portArg.setValue(Integer.parseInt(args[0]));
    VirtualMachine vm = socketConnector.attach(paramsMap);
    System.out.println("Attached to process '" + vm.name() + "'");
  }
}
}

It is a lot easier to get a Connector.Argument from an existing data structure (as above) than it is to create one from scratch. Also note, there are very few (if any) constructors in this API; just about every reference you get is retrieved ultimately by going through the Bootstrap class and working your way into the API. In my example, there were 3 AttachingConnectors, representing transports dt_socket, dt_shmem, and local. When I run the above example, I see the following output:

    Attached to process 'Java HotSpot(TM) 64-Bit Server VM'

Note that when this program exits, the target VM changes the port on which it is listening, something you should remember if you run again. I don't remember this behavior on Java 5, but it has been a while since I've written a JDI application.

Pause at a breakpoint and generate an event

To conclude this post in a reasonable length, I will just pick a line of code in my target that I know well and give some example code that will pick a variable off the stack and output it to stdout. The details of logging or sending a JMS message aren't really relevant to this topic.

For this example, I want to break at line 863, where I'm about to add the name of a file to my Swing table. This is a file whose name at least partially matches an input class name. Below is a segment of the source:

849:    if (fullName.lastIndexOf("/") > -1)
850:    {
851:      directoryName = fullName.substring(0, fullName.lastIndexOf("/"));
852:      fileName = fullName.substring(directoryName.length()+1, fullName.length());
853:    }
854:    else
855:    {
856:      fileName = fullName;
857:    }
858:    if (fileName.indexOf(searchForTextField.getText()) > -1)
859:    {
860:      Vector nextRow = new Vector();
861:      nextRow.add(archive.getAbsolutePath());
862:      nextRow.add(fileName);
863:      rowData.add(nextRow);
864:    }

I'd like to print a short message at line 863 which outputs the value of fileName.

How do you specify a breakpoint in JDI? You have to know what you're asking for. Normally you would look for a class, maybe a method, and a line number. My target application is a Swing application with a lot of anonymous inner classes, so rather than figure out which one is the one I want, I'm just going to search on line number. You probably want to call a constructor to create a breakpoint for a line number, but there is no constructor; you'll have to search through a lot of metadata and "find" the description of this line of code, then request a breakpoint using that description and a factory method in the EventRequestManager. To make a long story somewhat shorter:

Get a list of all classes (as ReferenceTypes).

For each class, get all line locations (Location).

At line location corresponding to line 863, break out of the search loop.

Get an instance of the EventRequestManager from the VirtualMachine.

Create a BreakpointRequest in the EventRequestManager, using the Location object for line 863.

Get the EventQueue instance from the VirtualMachine.

Create a while(true) loop on the EventQueue, calling its remove() method.

For each EventSet removed from the queue, process each Event.

For each Event, check to see if it is a BreakpointEvent, and if the line number matches the breakpoint we're interested in, process the Event further.

For a matching Event, get the top element of the StackFrame, get all visible variables on the StackFrame element, find the one whose name matches the variable you are looking for, and if so, dig through the API for the correct chain of method calls to extract its value.

This is probably easier shown with code. Below is an updated version of the first cut of the example code (note: please refactor out of main for a real application!):

import java.util.List;
import java.util.Map;
import com.sun.jdi.AbsentInformationException;
import com.sun.jdi.Bootstrap;
import com.sun.jdi.LocalVariable;
import com.sun.jdi.Location;
import com.sun.jdi.ReferenceType;
import com.sun.jdi.StackFrame;
import com.sun.jdi.StringReference;
import com.sun.jdi.ThreadReference;
import com.sun.jdi.Value;
import com.sun.jdi.VirtualMachine;
import com.sun.jdi.VirtualMachineManager;
import com.sun.jdi.connect.AttachingConnector;
import com.sun.jdi.connect.Connector;
import com.sun.jdi.event.BreakpointEvent;
import com.sun.jdi.event.Event;
import com.sun.jdi.event.EventIterator;
import com.sun.jdi.event.EventQueue;
import com.sun.jdi.event.EventSet;
import com.sun.jdi.request.BreakpointRequest;
import com.sun.jdi.request.EventRequest;
import com.sun.jdi.request.EventRequestManager;

public class JDIDemo
{
public static void main(String[] args) throws Exception
{
  if (args.length != 3)
  {
    System.out.println("Usage:  java JDIDemo debugPortNumber sourceLineNumber variableName");
    System.exit(-1);
  }
  int debugPort = Integer.parseInt(args[0]);
  int lineNumber = Integer.parseInt(args[1]);
  String varName = args[2];

  VirtualMachineManager vmMgr = Bootstrap.virtualMachineManager();
  AttachingConnector socketConnector = null;
  List attachingConnectors = vmMgr.attachingConnectors();
  for (AttachingConnector ac: attachingConnectors)
  {
    if (ac.transport().name().equals("dt_socket"))
    {
      socketConnector = ac;
      break;
    }
  }

  if (socketConnector != null)
  {
    Map paramsMap = socketConnector.defaultArguments();
    Connector.IntegerArgument portArg = (Connector.IntegerArgument)paramsMap.get("port");
    portArg.setValue(debugPort);
    VirtualMachine vm = socketConnector.attach(paramsMap);
    System.out.println("Attached to process '" + vm.name() + "'");

    List refTypes = vm.allClasses();
    Location breakpointLocation = null;
    for (ReferenceType refType: refTypes)
    {
      if (breakpointLocation != null)
      {
        break;
      }
      List locs = refType.allLineLocations();
      for (Location loc: locs)
      {
        if (loc.lineNumber() == lineNumber)
        {
          breakpointLocation = loc;
          break;
        }
      }
    }

    if (breakpointLocation != null)
    {
      EventRequestManager evtReqMgr = vm.eventRequestManager();
      BreakpointRequest bReq = evtReqMgr.createBreakpointRequest(breakpointLocation);
      bReq.setSuspendPolicy(BreakpointRequest.SUSPEND_ALL);
      bReq.enable();
      EventQueue evtQueue = vm.eventQueue();
      while(true)
      {
        EventSet evtSet = evtQueue.remove();
        EventIterator evtIter = evtSet.eventIterator();
        while (evtIter.hasNext())
        {
          try
          {
            Event evt = evtIter.next();
            EventRequest evtReq = evt.request();
            if (evtReq instanceof BreakpointRequest)
            {
              BreakpointRequest bpReq = (BreakpointRequest)evtReq;
              if (bpReq.location().lineNumber() == lineNumber)
              {
                System.out.println("Breakpoint at line " + lineNumber + ": ");
                BreakpointEvent brEvt = (BreakpointEvent)evt;
                ThreadReference threadRef = brEvt.thread();
                StackFrame stackFrame = threadRef.frame(0);
                List visVars = stackFrame.visibleVariables();
                for (LocalVariable visibleVar: visVars)
                {
                  if (visibleVar.name().equals(varName))
                  {
                    Value val = stackFrame.getValue(visibleVar);
                    if (val instanceof StringReference)
                    {
                      String varNameValue = ((StringReference)val).value();
                      System.out.println(varName + " = '" + varNameValue + "'");
                    }
                  }
                }
              }
            }
          }
          catch (AbsentInformationException aie)
          {
            System.out.println("AbsentInformationException: did you compile your target application with -g option?");
          }
          catch (Exception exc)
          {
            System.out.println(exc.getClass().getName() + ": " + exc.getMessage());
          }
          finally
          {
            evtSet.resume();
          }
        }
      }
    }

  }
}
}

When I run this application with a command line like:

     java -cp c:\jdk1.6.0_20\lib\tools.jar;. JDIDemo 56485 863 fileName

I get the following output:

    Attached to process 'Java HotSpot(TM) 64-Bit Server VM'
   Breakpoint at line 863:
   fileName = 'BreakpointEvent.class'
   Breakpoint at line 863:
   fileName = 'EventSetImpl$BreakpointEventImpl.class'

Pointers

You might have noticed the line above referencing the AbsentInformationException. You will get this if your target application has not been compiled with the debug (-g) switch. If you cannot compile the code with the debug switch, you will be able to set a breakpoint, but there won't be any information available on the stack when you get there.

Some JDI operations are more expensive than others. The last time I wrote a JDI application, I noticed that "method-entry" and "method-exit" breakpoints were enormously more expensive than simple line breakpoints. Now that I have a working example, I'll investigate these issues in a later post to see how things are in the current update of Java 6.

VisualVM Extensions

2011-10-04T19:11:00.000-07:00

As I mentioned in an earlier post, I noticed recently that the JDK utility VisualVM is extensible, and it was my goal to create a useful extension. I didn't intend to create a "getting started" guide, as there is plenty of information already available, but I keep running into the "out of date tutorial" issue. Having encapsulated some additional information into my earlier post in the setup category, I now find myself having to clear up issues I'm encountering while using NetBeans 7 to actually create an extension. While some of the differences might be traceable to the new version of NetBeans, I think some of them may be due to expected API changes and similar issues.

I'm working through the Creating a "Hello World" Plugin tutorial. I'll basically follow that tutorial here, with changes and observations/tips as needed.

If you followed my earlier post, you already have a new project, as that step was necessary to "seed" the Platform Manager into the clean-install NetBeans 7 Tools menu. If not, just follow Geertjan's steps in his tutorial. My next step was to create the Module Installer. In NetBeans 7, the selection items are slightly different. You still choose "New", then "Other...", then select "Module Development", but the File Type is now called "Installer / Activator" rather than "Module Installer". Further, the description says this installer is frequently not needed. I don't know if that's the case here, so I created one anyway.

The tutorial says to add the "VisualVM-Core" library as a dependency for the project. As you create your "Hello World" application according to the tutorial, you'll quickly learn that you also need to add a dependency on "VisualVM-Application", also, so you may as well do so at the same time.

If you're not fond of "seeing red" in an IDE, you might want to create the Java classes in the reverse order of the tutorial. Also, source listings in the tutorial don't include imports (the downloadable full sample does, of course). If you don't feel like retrieving the full sample, here are the imports for the HelloWorldProvider class:


 import com.sun.tools.visualvm.application.Application;
 import com.sun.tools.visualvm.core.ui.DataSourceView;
 import com.sun.tools.visualvm.core.ui.DataSourceViewProvider;
 import com.sun.tools.visualvm.core.ui.DataSourceViewsManager;

and, in the HelloWorldView class:


 import javax.swing.ImageIcon;
 import com.sun.tools.visualvm.application.Application;
 import com.sun.tools.visualvm.core.ui.DataSourceView;
 import com.sun.tools.visualvm.core.ui.components.DataViewComponent;
 import javax.swing.BorderFactory;
 import javax.swing.JEditorPane;
 import javax.swing.JPanel;
 import org.openide.util.Utilities;

From here, right-click on the project and select "Run" and you will get an instance of VisualVM. Selecting an application to monitor will expose your new "Hello World" tab. That was actually quite easy!

Pause for Perspective

What have we just done?

With VisualVM extensions, there are two different paths you can take:

Creating a new tool that you want to make available to VisualVM, a tool that will be available to analyze any Java application.

Providing a plugin for a unique type of application and using VisualVM to expose tooling specific to that application type.

What we have done in this "Hello World" is an example of #1 above, provided in the form of a new tab. If you do this, you aren't likely going to be leveraging anything unique about whatever application you are currently thinking of monitoring -- this version of VisualVM will expose the new tab for any and all Java applications to which it attaches.

Which type you choose will probably depend on what project currently interests you, and your general motivations. For example, it seems every project I work on could really use an application-specific VisualVM plugin. I typically use VisualVM on every project I work on, but I have to admit I don't see a lot of other users (unfortunately, because I think it's a great tool). In a production situation, technical support personnel are usually familiar with a small set of tools specific to their environment. For example, a WLS shop has support staff who are familiar with the WLS console; switching to JBoss in such a shop is a lot more complex than simply convincing someone of the technical virtues of one platform over another. There's a training and support issue, too. Additionally, I don't know of any support staff who use VisualVM.

Much as I'd like to provide a unique plugin for the applications I'm currently involved with, it would be a big investment for something that might not get a lot of adoption (and be useless for almost any other application). So, I probably would be more motivated to build a general-purpose module, or build a module that could be highly configurable for any application.

I have an idea in mind for just such a project; I'll share it at a later date. That way, if I get distracted by the guitar or practicing Portuguese and never get around to doing it, there won't be any 404s from my blog. But if I do get around to it, I'll update this entry just as soon as I do! In the interim, I hope that everyone who sees this will be motivated to add some valuable content to VisualVM. More later.

Extending JVisualVM -- More on Getting Started

2011-10-02T20:10:00.000-07:00

I'm not sure when this happened, but at some point the JDK-included VisualVM utility became extensible. This is really great news for me. Writing a profiler is a lot of work (although admittedly very interesting and fun). VisualVM has become such a great tool that I welcome the opportunity to add a plugin to it.

Oracle has provided a lot of information in this area, which I do recommend reading. Look at Getting Started extending VisualVM and a number of links at the bottom of that page. It helps a lot to get NetBeans if you don't already have it, as it has significant support built-in to develop VisualVM plugins.

Rather than repeating "getting started" info, I planned to produce an actual VisualVM plugin in this entry, but I ran into some issues with the "getting started" guide. So, this is a "getting started" update for NetBeans 7.0.1 users, especially those who have just installed it (as I have) and have not yet created any projects.

As usual, some things in the "getting started" guides don't turn out as expected. For example, the guides say to choose "Platform Manager" from the "Tools" menu; in my NetBeans 7.0.1 installation, there was no such menu item. It seems things have changed slightly. If you are using NetBeans 7.0.1, here is how I worked my way around the problem:

"File", "New Project..."

Under "Choose Project", choose "NetBeans Modules"

Under "Projects", choose "Module"

Click "Next"

Click "Standalone Module"; the dropdown mostly likely says "NetBeans IDE 7.0.1...".

Click on the "Manage..." button. This click will bring up the coveted Platform Manager.

Click "Add Platform..."

Now, rather than browse to the root folder of the VisualVM application and select that directory (Platform Manager won't let you use it), instead browse to the root folder of the JDK you are using in NetBeans, drill down into the lib directory, and select "VisualVM". Click "Next".

If you're OK with the default name "VisualVM", click "Finish". I named mine "VisualVM 1.6u20", to identify which version I chose.

The newly-added platform should be selected in the "Platforms:" dialog. Click "Close" and then in the "New Module" dialog, choose the VisualVM platform. Click "Next".

Provide a Code Name Base for the project (an example is given in the dialog) and click "Finish". You will now have a new project that will be built against the registered VisualVM binary.

I noticed that once I created a project and opened it (using the above steps), the "Tools" menu in NetBeans 7.0.1 became much more populated, including a menu item called "NetBeans Platforms". Apparently it was necessary to create and open one project to populate the "Tools" menu (however, it stays populated even if you close the new project). The only problem is that I needed for it to be populated before I created my project, so that I could make it a VisualVM project. The above steps show you how to achieve this goal if you have a fresh NetBeans deployment.

Now, on to that plugin I was going to write... That will have to wait for another day. Before I write that plugin, I have another post you might want to read that clears up some inconsistencies between the currently-published "getting started" tutorials and my actual experience trying to use them with NetBeans 7. It's called VisualVM Extensions. After that post, then I will get around to writing a (hopefully) useful VisualVM extension.

Direct Web Remoting tips

2011-10-01T09:49:00.000-07:00

We use DWR (http://www.directwebremoting.org) a lot where I'm working now. It's a Java library used to integrate JavaScript-based web development with Java middleware. In addition to providing a servlet that creates a nice JavaScript library (that delegates to your server-side Java beans), it also does server push (reverse Ajax) in a variety of flavors and situations. I'm really happy with it; you kind of just put it in your application and it works without a lot of trouble.

I recently had to dig a little into the source to learn how to do something I think might be generally useful (how to change, in real time, parameters which are typically only set via init-params in web.xml), so that's the topic here. If you aren't a DWR user, this might not be very interesting!

In my case, I needed to change DWR's maxWaitAfterWrite parameter, a DWR servlet init-param. Normally you set this once in web.xml, with:

<init-param>
<param-name>maxWaitAfterWrite</param-name>
<param-value>5000</param-value>
</init-param>

(maxWaitAfterWrite refers to the amount of time DWR waits, after output has occurred, before closing the browser connection). DWR has an org.directwebremoting.Container class which is used to inject properties, including this one, but no immediately obvious API for changing it during an active deployment.

As it turns out, DWR's IoC container is accessible from its org.directwebremoting.ServerContext, and this is the path to changing this parameter, at least. The Container holds a number of beans. Some of these are java.lang.Strings (for example, the initial value of maxWaitAfterWrite) and are read-only, but some of these are DWR classes with APIs that you can use to your advantage.

As I mentioned, the value of maxWaitAfterWrite from web.xml is injected into a Container bean of type java.lang.String and named maxWaitAfterWrite. However, this parameter is itself injected at deployment time into any DWR class which extends org.directwebremoting.dwrp.BasePollHandler. You can iterate over the beans in Container and find (on my deployment, at least) two beans which are instances of BasePollHandler. Below is some code which gets a reference to the Container and iterates over its beans, outputting the details on the maxWaitAfterWrite bean and any instances of BasePollHandler:

package com.adamsresearch;

import java.util.*;
import org.directwebremoting.*;
import org.directwebremoting.dwrp.*;

public class JavaEndpoint
{
public JavaEndpoint()
{
  ServerContext sc = ServerContextFactory.get();
  Container cont = sc.getContainer();
  Collection beanNamesColl = cont.getBeanNames();
  Iterator beanNames = beanNamesColl.iterator();
  while (beanNames.hasNext())
  {
    String nextBeanName = beanNames.next();
    if (nextBeanName.equals("maxWaitAfterWrite"))
    {
      System.out.println("Bean name:  '" + nextBeanName + "'");
      String staticBean = (String)cont.getBean(nextBeanName);
      System.out.println("  Class:  java.lang.String");
      System.out.println("  Value:  '" + staticBean + "'");
    }
    else
    {
      Object nextBean = cont.getBean(nextBeanName);
      if (nextBean instanceof BasePollHandler)
      {
        System.out.println("Bean name: '" + nextBeanName + "'");
        System.out.println("  Class:  " + nextBean.getClass().getName());
      }
    }
  }
}
}

If you add a method to this Java bean and invoke it from DWR, when the constructor is called, you should see output like the following:


Bean name:  'maxWaitAfterWrite'
  Class:  java.lang.String
  Value:  '500'
Bean name: 'url:/call/htmlpoll/'
  Class:  org.directwebremoting.dwrp.HtmlPollHandler
Bean name: 'url:/call/plainpoll/'
  Class:  org.directwebremoting.dwrp.PlainPollHandler

As subclasses of BasePollHandler, HtmlPollHandler and PlainPollHandler inherit the method setMaxWaitAfterWrite(). For my particular situation, then, I just grab all BasePollHandlers from the DWR Container (via their bean names, as output above) and set the parameter as needed.

This isn't rocket science, but it took me 2 or 3 hours to learn this, and there aren't any hits on the subject AFAIK. So, hopefully this will be useful to someone out there!

Day 7 -- Haskell

2011-04-03T11:24:00.001-07:00

I've reached the 7th and final language of Bruce Tate's Seven Languages in Seven Weeks. While some of the previous languages were functional with some imperative support, Haskell is a purely functional, and statically typed, language. I'm looking forward to seeing how I will perform tasks with no (or little -- more on that below) state.

Bruce starts with the usual investigation of types, including looking at Haskell's type inference, then moves on to functions. Functions are composed of a type declaration and the function declaration. Because Haskell is statically typed, a function must have a type, but the declaration isn't strictly necessary because of Haskell's type inferencing. He covers some examples using Erlang-familiar pattern matching and guards. As with many of the other languages in this book, Haskell uses tail-recursion optimization. Next we look at programming with tuples, and using tuples to combine functions. We follow with a discussion of Haskell lists, including generating them with recursion, ranges, and list comprehension (a feature shared with Erlang). By the way, Bruce's explanation of list comprehension here is one of the many reasons I like this book. I didn't quite catch it the first time, but his elaboration here is quite articulate.

Day 1 of Haskell finishes with an interview with Philip Wadler, a member of the committee which designed Haskell. One thing I took away from this interview was the variety of organizations in which Haskell is being used, which is very encouraging. I've been doing OO for 20 years now and I suppose I'm just looking around for evidence that it can be safe and effective to give up mutable state. I may actually decide to learn Haskell first, then pick up more Scala, as a way of disciplining myself to stick with the paradigm.

Day 2 starts with higher-order functions, including anonymous functions and partially-applied functions. Bruce then discusses some common functions, like filter and the Haskell versions of fold-left and fold-right.

Bruce spends a little more time discussing function composition. Every function in Haskell has only one parameter, which, as you can guess, leads to some interesting functional composition requirements to perform any useful computation. You can define a function with multiple parameters, but Haskell will split the function into multiple functions, one with each argument, a procedure known as partial application, or currying. Next he discusses lazy evaluation and how, as in Clojure, it can be used to create infinite lists, where only the parts necessary to perform a task are actually calculated. He finishes this section by describing how problem solutions can be constructed by chaining/nesting functions.

Haskell was created by committee. One side effect of this process is that a number of talented individuals not only contributed to the language, but are potentially available to share their thoughts on the genesis of the language. Bruce next interviews Simon Peyton-Jones, another member of the Haskell creation committee, and the lead designer of the GHC compiler. Peyton-Jones makes one observation that really sticks with me -- Haskell is a useful, successful language, designed by committee! How often does that happen?

Day 3 begins with a deeper dive into Haskell's type system, which Bruce calls one of its strongest features. As I mentioned earlier, the typing is static, but Haskell is smart enough to infer types most of the time, sparing some boilerplate. Developers can define their own data types with the data keyword. He then discusses defining polymorphic data types and recursive data types (the latter of which could be used to define a tree, for example).

Next is a discussion of Haskell classes, with a side note that these are not object-oriented classes. There is no data involved in a Haskell class! So a class, as Bruce puts it, "defines which operations can work on which inputs." So a type is an instance of a class if it supports all the functions signatures which make up the definition of the class.

Monads are discussed next. Bruce describes some example problems which are awkward to formulate in a functional language as a motivation for the use of monads, which are basically a technique of function composition. When I referred earlier to absence of state, but with a qualification, I was looking ahead to monads. Monads allow you to simulate state in a Haskell program. He goes on to outline the components of a monad, which are a container-based type constructor, a return function which wraps a function into a monad, and a bind function that unwraps the function. He then gives a few non-trivial monad-construction examples, including the use of the Haskell do notation to add some syntactic sugar. Bruce ends his Day 3 of Haskell with a discussion of the language's core strengths and weaknesses.

My Personal Wrap-Up

I started reading this book with some understanding and hands-on experience with Scala, and a slight feeling of lost opportunity as a result (Was this a good idea? Will I really learn functional programming with a "compromise" language?). I still feel that way a little, but I still expect to move forward with Scala as my bridge to the FP world, although with a better understanding of what compromises I am making in the process. I'm considering learning Haskell, at least minimally, to see what it's like for me to work with a language that doesn't have the Java/JVM escape hatch in it, so that I don't automatically reach for that pill just because it's available. Hopefully that experience will encourage me to use Scala's functional side first, before reaching for the Java escape hatch.

Concerning this book: Bruce Tate is a very articulate, clear and engaging writer. I see a couple of books a year that speak very directly to me. Maybe I think these books are superior simply because they happen to have been written in a style that is especially suited to the way I think and learn. With that acknowledgment of likely bias, I have to say Seven Languages in Seven Weeks is definitely one of those "once or twice a year" books for me. I really recommend you pick up a copy if you have a desire for a shallow-dive into a variety of languages.

Day 6 -- Clojure

2011-04-02T20:35:00.000-07:00

Today I'm reviewing the discussion of Clojure from Bruce Tate's Seven Languages in Seven Weeks. Clojure is Lisp on the Java virtual machine. Lisp is another language that, despite being around a long time, I have yet to investigate, so this is another new experience. Bruce starts with a brief review of Lisp itself, then a quick discussion of the link between Clojure and the JVM. A discussion of Lisp functions and datatypes is followed by a view of Clojure lists, sets, maps and vectors (in Clojure, lists are typically used for data, while vectors are used for data).

Still on Day 1 of Clojure, we discuss functions, including how to pass parameters, deconstruction using bindings, and then anonymous functions. The day ends with an interview with Rich Hickey, the creator of Clojure. One interesting revelation from this interview is that a concerted effort was made to reduce the use of parentheses, to make the language more accessible to the rest of us!

Day 2 discusses recursion, including interesting syntax Clojure uses to support tail recursion (required because of the fact that Clojure runs on the JVM). Next, Bruce discusses Clojure sequences, which can be used to wrap all the Clojure collection classes (plus a few more), and functions available to operate on sequences. He discusses lazy evaluation, used to create infinite sequences. A little discussion on Java interoperability is followed by an overview of macros, used to extend the Clojure language.

Bruce's Day 3 discusses concurrency. Unlike any of the other languages discussed so far in the book, Clojure supports concurrency with an approach that is called software transactional memory, a strategy that uses multiple versions of data to avoid issues surrounding simultaneous mutation. To use this strategy, Clojure requires that all references be changed within a transaction. Following are discussions of working with atoms, agents, and futures. Bruce ends Day 3 with a list of some features of Clojure which were not covered in the book, then ends with the expected discussion of Clojure's strengths and weaknesses.

Day 5 -- Erlang

2011-04-01T20:04:00.000-07:00

If you are just dropping in on me, I'm reviewing Bruce Tate's Seven Languages in Seven Weeks, with the slightly lazy (or aggressive, depending on your view) twist of reviewing one language per day. As you can imagine, shrinking an already-tight one-week study schedule to one day means glossing over some things and skipping the extra exercises. The idea, though, is to still get a good feel for the topography of the language, and as I've already pointed out, this is a great little book and will be my first contact for the languages I choose to get to know a little better.

The 5th language of this book is Erlang ("Ericsson Language"), first developed by Joe Armstrong in 1986. A functional, dynamically-typed language geared toward telecommunications and its high availability requirements, Erlang relies on lightweight processes (rather than threads) as its base model. It is based on and quite similar to Prolog, something brought out by Bruce's interview with Joe Armstrong. Have I said I really like the interview portions of this book? After enough years in this industry, I had started to say "one more language; let me take a look at it and just give me the API docs", but after reviewing these languages I see I've overlooked the huge amount of variety in the development-language world.

Erlang is a compiled language, but it provides an interpreter which Bruce uses for the earliest, shortest examples. In his Day 1, Bruce discusses atoms, variables, lists and tuples, and gives a short discussion of pattern matching (including bit matching). Day 2 starts by discussing control structures, mostly case statements but also including the if statement.

Bruce then moves on to higher-order functions, starting with anonymous functions, and discusses using functions to manage lists. Highlights include the foldl and foldr functions, as well as a brief review of Erlang's list comprehension.

On Day 3 (using Bruce's timeline), he talks about concurrency. I said earlier that Erlang looked more to lightweight process than lightweight threads as a base model. The concurrency primitives in Erlang are sending messages, spawning processes, and receiving messages. He covers pattern matching in receive loops, which have a syntax similar to case statements, then discusses the use of spawn to spawn a process. Finally, the returned process ID of the spawned process is used as the destination for messages. He walks us through an asynchronous example, followed by a simple asynchronous messaging example. He then discusses linking processes and how to handle death (exit) of a process by its linked process (a technique used to create monitoring and keep-alive services).

The end of the Erlang section covers strengths and weaknesses of the language. One weakness, if you can call it that (at least from an adoption point of view) is not being on the Java Virtual Machine. I thought of this frequently while going through the Erlang section of this book. Reading about Erlang's support for concurrency is refreshing and quite inspiring for me. However, I know that I will most likely eventually use Scala. Why? That JVM is just going to make it a lot easier to sell to management. That might be a shame, but it's kind of a fact of life. I remember really liking Smalltalk, but realizing that summer was coming to an end and going with C++ anyway. But it doesn't mean I don't still think fondly of my first Smalltalk application. :)

Day 4 -- Scala

2011-03-31T21:20:00.000-07:00

Scala is the 4th language in Bruce Tate's Seven Languages in Seven Weeks. It is the only language in this book with which I am already familiar, although I've only been learning it for a month or so. Scala is interesting in that it runs on the Java virtual machine, a feature which may give it a leg up in the language race. I remember when I and my co-workers "discovered" object orientation. We studied Coad and Yourdon and prototyped in Smalltalk and C++. C++ was a great bridge to OO for C programmers, and it wasn't hard to tell from the beginning which language was going to win that race.

Of course, Clojure runs on the JVM also, so maybe this time the race will be a little more interesting. From my own point of view, the JVM connection definitely influenced my decision to start learning Scala rather than some other functional language.

Bruce covers a lot of material on Day 1, really too much to list here. As he points out, to explore Scala is to explore both a functional language and an object-oriented language. So Day 1 is a very dense race through many features of Scala.

Day 2 begins with a discussion of function definition, and a description of vals and vars, the general concept of mutability, and how important it is to pay attention to mutability if you are going to be a successful functional developer. As Bruce puts it, "mutable state limits concurrency." Think of all those IllegalStateExceptions and NullPointerExceptions in your multi-threaded Java code. If your application had no mutable state, it wouldn't matter how many times you called a method, or what you called before you called that method, or how many threads are calling the method. After years of being sent on-site to diagnose such problems, this is the reason I'm interested in functional programming.

Next, Bruce introduces Scala's collections, emphasizing Lists, Sets, and Maps. He then discusses higher-order functions (that is, functions which take functions as arguments and return functions), code blocks and anonymous functions. Following is a detailed enumeration of list operations. This is all preparation for Day 3, of course, which primarily covers XML processing and concurrent programming.

Scala's support for XML involves treating XML as a first-level language construct and providing query support similar to XPath. A couple of brief query examples are followed by a discussion of Scala pattern matching, including guard conditions and regular-expression support, and then a discussion of using pattern matching in XML processing. Day 3 ends with a quick discussion of Scala concurrency and a couple of examples.

As with all the languages, Bruce ends with a discussion of Scala's strengths and weaknesses. His discussion of the advantages and disadvantages of Scala's Java connection is worth reading even if you decide not to dive deep into Scala, or even dive deep into Bruce's "week with Scala." As with VHS and Betamax, it probably doesn't make sense to make any predictions at this point concerning what development paradigm and/or representative language will "win" in the developer marketplace. That's another reason why I like this book, or course -- it's a very thoughtful and non-trivial investigation of a variety of languages. And I didn't even have to choose which languages to investigate!

Day 3 -- Prolog

2011-03-30T07:30:00.000-07:00

Prolog is the 3rd language covered in Bruce Tate's Seven Languages in Seven Weeks, and is a declarative, rather than imperative, language. Prolog is not new, of course (1972), but I have to admit this is the first time I've taken a look at it.

Like Bruce, I used GNU Prolog (1.3.1). As he describes the language, the building blocks are facts and rules (which together form a knowledge base), and queries. He describes some simple queries, then moves on to substituting variables in queries. He finishes the first day of the Prolog segment with a discussion of Prolog unification, then an interview with Brian Tarbox, a dolphin researcher who used Prolog both to simulate dolphin behavior and to schedule the personnel needed for each dolphin experiment. According to Tarbox, Prolog was very well suited for solving both types of problems.

The first day looked at declaring facts and rules and making simple queries against them. Day 2 adds some more tools, including recursion, math functions, and lists and tuples (and operations on them). Day 2 is the day where my experiment in covering one language per day seems the least feasible; I'm not familiar with declarative (programming) languages and although I get the idea, a lot of it went over my head. Bruce is extremely profuse with examples, deconstructed ones at that. If I wanted to learn Prolog, I would start with this book first and spend a few more days on it.

For Day 3, Bruce goes through two detailed examples -- a solution for a small Sudoku puzzle, and the Eight Queens problem. He then finishes with a discussion of Prolog's strengths and weaknesses, including listing problem spaces for which Prolog is particularly well-suited. Those areas? Scheduling constrained resources, natural-language processing, games, semantic web representation and artificial intelligence.

Day 2 -- Io

2011-03-29T10:09:00.000-07:00

Io is the 2nd language in Bruce Tate's Seven Languages in Seven Weeks. Io is a prototyping language, where most of the mass exists in the libraries. The syntax itself is refreshingly simple, and he is not exaggerating to say you can grasp it in about 15 minutes.

As with the other languages he discusses, there's no download/install information save for the Io site URL. I did need a little help getting past this stage; if you click the only "download" link at http://iolanguage.com, you'll get a zipped distribution of the Io source. If you're smart enough to append "/downloads" to the URL (there's no link to this from the home page!), you'll find (in my case, Windows) binaries to spare you the build process (I wasn't smart enough to figure this out; I found another website which pointed me to http://iolanguage.com/downloads.)

Day 1 of Bruce's discussion of Io gets you familiar with the Io interpreter and the process of cloning. You learn about the "slots" of an object, which are basically the attributes and behavior. He then covers Io's lists and collections (specifically, List and Map). I was a little surprised to get different results than Bruce did (e.g. sending the asObject message to a Map clone returns Exception: Map does not respond to 'asObject', whereas in Bruce's output it returns an Object , though the version of Io I'm using appears to be old enough that it certainly should be the same as Bruce's). He then discusses true, false, nil and singletons, the last of which you create by overriding an Object's clone method. The ability to override even core methods is a key to Io's potential for creating domain-specific languages.

He finishes the "day" with an interview with Steve Dekorte, the inventor of Io, asking him about his motivations to create the language. I really like these interviews, as they give a (usually unavailable) view into the mind behind and motivations for the existence of a language. Language developers tend to be quite articulate when describing their motivations for creating a language; how often does anyone ask them?

Day 2 covers conditionals and loops, operators and messages, as well as message reflection and object reflection. Day 3 starts with a discussion of a non-trivial example of a domain-specific language, then discusses changing the language's behavior, by modifying Io's forward method (analogous to the discuss of Ruby's method_missing yesterday).

Bruce spends the rest of the day discussing Io's support for concurrency (as a longtime Java developer, alternative-language concurrency support is the number-one reason I am reading this book). He describes coroutines and contrasts them with pre-emptive multi-tasking in Java and C, then describes Io actors and contrasts them with threads, and shows how easy it is to make an object an actor. Concluding the concurrency segment is a discussion of Io futures.

As with Ruby on Day 1, Bruce ends the discussion of Io with a list of strengths and weaknesses, and as with Ruby, he puts a lot of thought into this discussion. I'll repeat that I am only scratching the surface by covering this "week" in one day (while recognizing that even one week would be scratching the surface), but I really appreciate the compactness of these segments and the density of information contained in them.

Day 1 -- Ruby

2011-03-28T19:49:00.000-07:00

When I decided to blow through Seven Languages in Seven Weeks in only 7 days, I had yet to read even the introduction to the book. I can tell now that Bruce might prefer that I not do it this way, and I have to admit it would truly be a focused experience to do all the suggested exercises on each language, spending only one day on each. But... I'm on a schedule, and I'm not pretending to come out of this with the level of understanding that he aims to cultivate. At least not on this first run through the book.

First, some notes about his introduction. Bruce describes up-front how he'll contrast the different languages: the typing model, programming model, language constructs, data structures, and any other features that distinguish a language. I really like this set of abstractions, and the simple fact that he lists and describes them before launching into the first language. He also says he's not going to burn space describing how to download and set up each language's environment, thus allowing him to focus on the language itself. It's nice to see a book with this much thought put into it.

The first language, Ruby, appears to be the language (of these 7) in which he is strongest. I left scientific development for business development quite a few years ago, at which point I was rather proficient with Perl (version 4, to tell you how long ago that was). Since then I've been immersed mostly in Java and C# on Windows . I've always thought that it would be cool to pick up Ruby, but have never got around to it. Finally, I'm spending one day on it. :)

Bruce starts by sharing his interview with Yukihiro Matsumoto, the creator of Ruby (nice touch), then talks about Ruby's pure-object-oriented typing model. After using Java for 14 years, it's refreshing to be able to call a method on what appears to be the numeral 4! ;-> Ruby is (mostly) strongly-typed, but also dynamically typed. He then covers function definition, collections, code blocks, classes, and mixins. And this is just by the end of the 2nd day of the week he allots to studying Ruby. This is just my opinion, of course, but it's not often I've seen someone so proficient at introducing relevant features of a language in an order that is so intuitive -- and useful -- to an experienced developer looking to evaluate a new language.

Day 3 of the Ruby chapter talks about Ruby support for metaprogramming, including open classes and modules. Here is where you can start to see Bruce's depth of knowledge about Ruby. He concludes the Ruby discussion with sections on the strengths and weaknesses of Ruby vs other languages.

In the interest of finishing in one day, I sped over most of the hands-on training. Following along and doing the suggested exercises would indeed fill a whole week, comfortably. I like this chapter so much that when the day comes for me to finally really pick up Ruby, I'm sure I'll actually start with this chapter.

7 Languages in 7... Days???

2011-03-28T10:06:00.000-07:00

I recently started learning Scala (you know when the New York Times refers to Java as an "older" language, it's time to update!). As I've started trying to shift my thinking from object orientation to functional programming, I remembered the book Seven Languages in Seven Weeks by Bruce Tate. I'm about halfway through Odersky et al.'s (excellent!) Scala book and thought I'd benefit from a look at Bruce's.

So here I go. Nothing unusual except that my goal is to go through a language a day and finish the book in a week. Why a week? I don't really have seven weeks, and besides, I'm going in to my office for one of my 3-4 times-per-year office visits, and I want to finish before I go in. Unless I totally flame out, I'll publish an entry each day, for each language.

One of my motivations is the effort involved to think in terms of functional programming. I never completed reading the first Scala book I picked up (and have already donated it to my local library book drive). Why? The author obviously no longer remembers what it was like to make the switch from Java to Scala. This makes it difficult for him to relate to someone who hasn't yet made the switch, and the material is presented in an order which is wildly inappropriate for someone who hasn't already learned Scala. Before >I< learn everything I need to learn, I would like to capture that learning process myself, so I can remember what it was like and possibly be more helpful to someone else (in case you're wondering, Odersky, Spoon and Venners do a great job connecting with object-oriented developers in their Scala book).

That's it for now. I see the first language in Bruce's book is Ruby, a language I once downloaded and never got around to using. Hopefully I'll have a meaningful post on his Ruby section by my late-night deadline. If not (either it's not there or it's not meaningful!), I'll try to catch up the next day.

Java SE Endpoints and Providers

2010-12-21T09:31:00.000-08:00

Recently I had a repeat of a problem I was unable to solve the first time I encountered it. When this first occurred, I was happily serving up content from a Java SE-based web service (if you're not familiar with this, it involves specially annotating a class and exposing it by publishing it to a javax.xml.ws.Endpoint). I ran into a problem when I tried accessing the web service (I believe from a Flex app), because the Flex runtime really "wanted" to find a crossdomain.xml file at the HTTP server root. Since it was unable to do so, it obligingly refused to expose my web service to possible attack and would not access the web service.

Leaving aside for the moment the philosophy of the client's enforcing security upon itself (this doesn't protect my web service from other unscrupulous clients), I was left with a problem. This approach is a little naive because the client is assuming the web service has something like a root document directory, where you can just drop the crossdomain.xml file and get on with your life. As you know, when you expose a Java bean as an Endpoint, there is no file system; just an internal HTTP server servicing the web-service requests. So where do you put the file?

The answer is: wherever you want, as long as you figure out how to serve it up! I ran into this same problem recently as I was building a Silverlight-based application. Silverlight enforces the same constraints on itself, and indeed will actually look for a crossdomain.xml file (after searching first for a clientaccesspolicy.xml file). This time I worked a little harder to find an answer, and got a lucky break from an online posting.

Using tcpmon, I noticed an interesting point. While my web service was hosted at a URL like http://www.example.com:8080/WebManager, the Silverlight runtime actually looks for the policy file at http://www.example.com:8080/crossdomain.xml. In other words, not at the context at which my web service was deployed, but at the root of the HTTP server itself. This distinction proves to greatly facilitate the solution.

The solution is to publish two endpoints -- one for the web service, and one for a document server. The javax.xml.ws.Endpoint class allows you to publish multiple beans at the same host and port (a crucial point, which makes this solution work). You just need to ensure the context, or full URL, is different for each. So, while I may publish my web service with the following statement:

Endpoint.publish("http://www.example.com:8080/WebManager", new WebManager());

I am able to publish a different endpoint to satisfy Flex/Silverlight with:

Endpoint.publish("http://www.example.com:8080/", new DocumentProvider());

When my Silverlight control attempts to access the web service, it will first look for a crossdomain.xml file at the web server root; if it determines it is allowed to access the service, it will then do so.

What is a DocumentProvider? It is a class which I've published to an Endpoint much like my actual business logic, but it intercepts the HTTP requests and only processes requests for documents. In other words, instead of letting the Java SE implementation handle all the requests, delegating them to @WebMethod-annotated methods, it handles all the work itself. The API for this can be found at javax.xml.ws.Provider, which defines a single method:

public T invoke(T request);

where T must be either a Source, a SOAPSource, or a DataSource. This looks a little generic, and it is. But this is where all the work occurs. In my case, I'm going to return a DataSource whose input stream is the requested document.

Following is a complete class which implements only the document server (exposing methods as web service operations is fairly well documented):

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.IOException;
import java.io.OutputStream;
import javax.activation.DataSource;
import javax.annotation.Resource;
import javax.xml.ws.BindingType;
import javax.xml.ws.Endpoint;
import javax.xml.ws.Provider;
import javax.xml.ws.Service;
import javax.xml.ws.ServiceMode;
import javax.xml.ws.WebServiceContext;
import javax.xml.ws.WebServiceException;
import javax.xml.ws.WebServiceProvider;
import javax.xml.ws.handler.MessageContext;
import javax.xml.ws.http.HTTPBinding;

@WebServiceProvider
@BindingType(HTTPBinding.HTTP_BINDING)
@ServiceMode(value=Service.Mode.MESSAGE)
public class DocumentServer implements Provider
{

protected static final String CROSSDOMAIN_XML = "<?xml version=\"1.0\" ?><cross-domain-policy><allow-access-from domain=\"*\" /></cross-domain-policy>";

@Resource
protected WebServiceContext wsContext;

public static void main(String [] args)
{
DocumentServer docServer = new DocumentServer();
}

public DocumentServer()
{
Endpoint.publish("http://localhost:10486/", this);
}

public DataSource invoke(DataSource ds) throws WebServiceException
{
MessageContext msgCtx = wsContext.getMessageContext();

String method = (String)msgCtx .get(MessageContext.HTTP_REQUEST_METHOD);
String pathInfo = (String)msgCtx.get(MessageContext.PATH_INFO);
if (method.equals("GET") && pathInfo.equals("crossdomain.xml"))
{
  return new DataSource()
  {
    public InputStream getInputStream()
    {
      return new ByteArrayInputStream(CROSSDOMAIN_XML.getBytes());
    }
    public OutputStream getOutputStream()
    {
      return null;
    }
    public String getContentType()
    {
      return "text/xml";
    }
    public String getName()
    {
      return "";
    }
  };
}

throw new WebServiceException();
}
}

Some things to note about the above code:

There is a lot of flexibility in the Provider interface.
How your implementation of Provider is interpreted by the Java compiler depends in great part on the annotations you have added to your implementation. The WebServiceProvider, BindingType, and ServiceMode annotations are tightly coupled with the specifics of your implementation. Try compiling the example without them, to see what I mean.
The WebContext is vital but can be obtained by simple injection.
I only process "GET" requests for "crosssdomain.xml", and return a very liberal policy; of course, this is just an example. You could just as easily create a web-serviceable management interface on your crossdomain policy and create it dynamically, as well as serve it -- and other documents -- from a file system, rather than the way I've done it here.

Finally, I'm no expert in this area but am happy to pass the information along, as it is somewhat difficult to find examples of Provider implementations. In that spirit, I acknowledge the folks who posted this example: http://acm2010.cct.lsu.edu/localdoc/java/6-jdk/sample/webservices/EbayClient/. It was greatly appreciated!

Gorilla Logic releases the MonkeyWrench Java profiler

2010-02-17T13:04:00.000-08:00

Hello, everyone:

I've posted a number of entries in the last year about profiling Java applications. Some of this effort went in to a standalone Java profiler with a Swing interface, called the MonkeyWrench. My company (Gorilla Logic) maintains and has released MonkeyWrench, and you can find it at http://www.gorillalogic.com/monkeywrench. Source code is also available, so if there's some interesting data you want from the java.lang.instrument package (or the JMX beans that I query) that you don't see, feel free to experiment to suit your needs. If you're curious but not sure if you need a profiler right now, take a look at the Getting Started Guide, which contains screenshots. Happy Profiling!

Java profiling with DTrace

2010-01-15T08:44:00.000-08:00

DTrace, a dynamic tracing framework on Solaris (see dtrace(1m)) is a valuable and extremely easy-to-use tool if you find yourself analyzing Java performance issues on Solaris. While there is a lot of information on DTrace available (see the DTrace Wiki, which includes an extensive user guide), less information is available on using DTrace specifically with Java applications. My motivation with this post is to focus on profiling Java applications with DTrace.

Quick Tour

If you're not at all familiar with DTrace, look at the Getting Started section of the user guide. In short, DTrace allows you to specify a set of actions you would like performed at a location of interest (in the kernel or an application). A location of interest is referred to as a probe. In addition to a numerical ID, each probe is identified by the following four pieces of information:

The probe Provider name
The probe Module
The name of the Function where the probe is located
A Name for the probe, designed for human readability

(By the way, if you don't have a Solaris environment but would still like to follow along, you can install VirtualBox, then retrieve the ISO image for OpenSolaris, then follow the VirtualBox guide to get up and running on OpenSolaris).

What probes are available? Well, a lot. On my OpenSolaris (2009.06) environment, a count (using pfexec dtrace -l | wc -l), yields 60068 probes! (Note: this includes some Java-6-specific probes; more on that later). Most of these are fairly low-level probes from the point of view of a Java profiler, but let's take one and see what we can do with it.

Using DTrace, we can perform actions when the read system call is entered, using the following:

     pfexec dtrace -n syscall::read:entry

If you enter this, you'll begin to see output such as the following:

  0  57922                       read:entry 
  0  57922                       read:entry 
  0  57922                       read:entry 
  0  57922                       read:entry 
  0  57922                       read:entry

Note this doesn't tell you a lot; you don't know the PIDs of the processes making the calls, and so on. DTrace has a number of built-in functions and variables that you can use to provide meaningful output at these locations. I won't go into this in any detail because there is a large body of information on these topics at the links I mentioned above. This is just the very quick tour to get to, and focus on, Java-specific probes.

At this point, before going any farther, you would begin placing your declarations and logic in a file. DTrace understands the D programming language, a language that will be familiar to most developers (especially C developers). The above syscall::read:entry example, placed in a file and enhanced to 1) apply only to a specified PID, and 2) to print out the name of the process, follows:


syscall::read:entry
/pid == $1/
{
printf("'%s' (PID: %d): entered syscall:read", execname, pid);
}

Place the following text in a file called blogExample.d, then find a process to monitor (I picked Firefox) and enter the following, substituting the PID of the process of your choice:

     pfexec dtrace -s blogExample.d 697

and you should see output similar to the following:

0  57922                       read:entry 'firefox-bin' (PID: 697): entered syscall:read
0  57922                       read:entry 'firefox-bin' (PID: 697): entered syscall:read
0  57922                       read:entry 'firefox-bin' (PID: 697): entered syscall:read
0  57922                       read:entry 'firefox-bin' (PID: 697): entered syscall:read

Note the predicate (/pid == $1/) in the example which restricts actions to be performed only when pid (a built-in DTrace variable) is equal to the first argument supplied on the command line ($1). To suppress all output except for that specified in your printf() call, add the -q ("quiet") switch (you'll need to add a newline to the end of your printf() specification, also, as "quiet" mode does not provide one):

     pfexec dtrace -q -s blogExample.d 697

and you should see output similar to the following:

'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read

That wraps up the quick tour of DTrace. Now, on to our main topic!

DTrace Support in Java

Java SE 6 now supports DTrace, in the form of a couple of providers (specific to the HotSpot JVM):

hotspot
hotspot_jni

For more information, see DTrace Probes in HotSpot VM. We'll be focusing on the hotspot provider in this post. Normally, you would expect to find a provider with the dtrace -l command, looking for "hotspot". This approach gets you most of the way there; actually, providers are on a per-JVM basis, and they are identified by appending the PID to the name "hotspot." So, for example, if you have a Java process running and its PID is 1234, then there will be a DTrace provider called "hotspot1234". This naming convention is the reason that you will see references to "hotspot$1" in example D-language scripts -- the author is expecting a PID on the command line and is using it to create the provider name. This convention also spares us from having to restrict our probe actions to a specific PID, as the restriction is built in to the choice of provider.

Back to the Java-specific probes. Enter the following command in a shell:

pfexec dtrace -l | grep hotspot

and you should see a list of all the hotspot and hotspot_jni probes. On my system, I got 499 probes with one Java process running; note there will be a complete set for each JVM. I have 26 "hotspot" probes, and 473 "hotspot_jni" probes. I will only be looking at "hotspot" probes in this post. You can also get a list of these probes, along with their arguments, in the Reference section of DTrace Probes in HotSpot VM.

Some Examples

Thread Monitoring

Note the following two probes related to thread activity:

  probe thread-start(char* thread_name, uintptr_t thread_name_length,
uintptr_t java_thread_id, uintptr_t native_thread_id, bool is_daemon);
probe thread-stop(char* thread_name, uintptr_t thread_name_length,
uintptr_t java_thread_id, uintptr_t native_thread_id, bool is_daemon);

One interesting detail to note with string arguments in DTrace is that they come in pairs, where the first item is a pointer to the un-terminated string and the second item is the length of the string. Because of this convention, in example code you'll frequently see the following construct, which builds a null-terminated string from the provided information:

self->str_ptr = (char*) copyin(arg0, arg1+1);
self->str_ptr[arg1] = '\0';
self->stringVariable = (string) self->str_ptr;

Note that the above fragment could be used to construct the name of the thread from the above thread-start and thread-stop probes.

To see how easy it is to use DTrace to create a simple thread profiler on Java 6, open a file in a text editor and enter the following:

     hotspot$1:::thread-start
{
self->ptr = (char*) copyin(arg0, arg1+1);
self->ptr[arg1] = '\0';
self->threadname = (string) self->ptr;

printf("thread-start: '%s' (thread ID = %d; native thread ID = %d; is_daemon? %d)\n",
 self->threadname, arg2, arg3, arg4);
}

hotspot$1:::thread-stop
{
self->ptr = (char*) copyin(arg0, arg1+1);
self->ptr[arg1] = '\0';
self->threadname = (string) self->ptr;

printf("thread-stop: '%s' (thread ID = %d; native thread ID = %d; is_daemon? %d)\n",
 self->threadname, arg2, arg3, arg4);
}

Note as mentioned earlier the $1 macro variable, representing the JVM process ID you will be passing in on the command line. In this script, we output everything we get from these two probes, which is the thread name, ID, native ID, and a boolean "is_daemon". Save this file as threadWatcher.d and enter the following:

pfexec dtrace -q -s threadWatcher.d pid

substituting the process ID of a running JVM. In my test Java application, I then opened a file chooser and got the following output from DTrace:

thread-start: 'Basic L&F File Loading Thread' (thread ID = 42; native thread ID = 42; is_daemon? 0)
thread-stop: 'Basic L&F File Loading Thread' (thread ID = 42; native thread ID = 42; is_daemon? 0)
thread-start: 'Basic L&F File Loading Thread' (thread ID = 43; native thread ID = 43; is_daemon? 0)
thread-start: 'Basic L&F File Loading Thread' (thread ID = 41; native thread ID = 41; is_daemon? 0)
thread-stop: 'Basic L&F File Loading Thread' (thread ID = 41; native thread ID = 41; is_daemon? 0)
thread-stop: 'Basic L&F File Loading Thread' (thread ID = 43; native thread ID = 43; is_daemon? 0)

Garbage Collection

Note the following probes relevant to garbage collection:

probe gc-begin(bool is_full);
probe gc-end();
probe mem-pool-gc-begin(
 char* mgr_name, uintptr_t mgr_name_len, char* pool_name, uintptr_t pool_name_len,
 uintptr_t initial_size, uintptr_t used, uintptr_t committed, uintptr_t max_size);
probe mem-pool-gc-end(
 char* mgr_name, uintptr_t mgr_name_len, char* pool_name, uintptr_t pool_name_len,
 uintptr_t initial_size, uintptr_t used, uintptr_t committed, uintptr_t max_size);

The first two probes provide generic information about the start and stop of a garbage collection, while the latter two provide more detailed information. We'll output information from both to compare. Note that the manager name and pool name are provided with the same pointer/length data pairs as we saw earlier.

Open a file called gcWatcher.d in a text editor and enter the following:


hotspot$1:::gc-begin
{
printf("gc-begin: is_full = %d\n", arg1);
}

hotspot$1:::gc-end
{
printf("gc-end:\n");
}

hotspot$1:::mem-pool-gc-begin
{
self->ptr1 = (char*) copyin(arg0, arg1+1);
self->ptr1[arg1] = '\0';
self->mgr_name = (string) self->ptr1;

self->ptr2 = (char*) copyin(arg2, arg3+1);
self->ptr2[arg3] = '\0';
self->pool_name = (string) self->ptr2;

printf("mem-pool-gc-begin: mgr: '%s'; pool: '%s'; initial: %d; used: %d; committed: %d; max: %d\n",
    self->mgr_name, self->pool_name, arg4, arg5, arg6, arg7);
}

hotspot$1:::mem-pool-gc-end
{
self->ptr1 = (char*) copyin(arg0, arg1+1);
self->ptr1[arg1] = '\0';
self->mgr_name = (string) self->ptr1;

self->ptr2 = (char*) copyin(arg2, arg3+1);
self->ptr2[arg3] = '\0';
self->pool_name = (string) self->ptr2;

printf("mem-pool-gc-end: mgr: '%s'; pool: '%s'; initial: %d; used: %d; committed: %d; max: %d\n",
    self->mgr_name, self->pool_name, arg4, arg5, arg6, arg7);
}

Save the file, then trace your Java application with the following:

     pfexec dtrace -q -s gcWatcher.d pid

For my example, after triggering a little activity in my application, I saw a number of minor collections; here is the output from one of them:


gc-begin: is_full = 0
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Code Cache'; initial: 163840; used: 1751360; committed: 1769472; max: 33554432
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Eden Space'; initial: 917504; used: 917504; committed: 917504; max: 4194304
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Survivor Space'; initial: 65536; used: 58904; committed: 65536; max: 458752
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Tenured Gen'; initial: 4194304; used: 1406096; committed: 4194304; max: 61997056
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Perm Gen'; initial: 12582912; used: 1144824; committed: 12582912; max: 67108864
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Perm Gen [shared-ro]'; initial: 8388608; used: 6733248; committed: 8388608; max: 8388608
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Perm Gen [shared-rw]'; initial: 12582912; used: 7676960; committed: 12582912; max: 12582912
mem-pool-gc-end: mgr: 'Copy'; pool: 'Code Cache'; initial: 163840; used: 1751360; committed: 1769472; max: 33554432
mem-pool-gc-end: mgr: 'Copy'; pool: 'Eden Space'; initial: 917504; used: 0; committed: 917504; max: 4194304
mem-pool-gc-end: mgr: 'Copy'; pool: 'Survivor Space'; initial: 65536; used: 29648; committed: 65536; max: 458752
mem-pool-gc-end: mgr: 'Copy'; pool: 'Tenured Gen'; initial: 4194304; used: 1458336; committed: 4194304; max: 61997056
mem-pool-gc-end: mgr: 'Copy'; pool: 'Perm Gen'; initial: 12582912; used: 1144824; committed: 12582912; max: 67108864
mem-pool-gc-end: mgr: 'Copy'; pool: 'Perm Gen [shared-ro]'; initial: 8388608; used: 6733248; committed: 8388608; max: 8388608
mem-pool-gc-end: mgr: 'Copy'; pool: 'Perm Gen [shared-rw]'; initial: 12582912; used: 7676960; committed: 12582912; max: 12582912
gc-end:

Note that not a great deal of information is available from the generic garbage-collection begin and end probes, while much more information is available from the mem-pool probes.

You can do much more than simply print the values of probe arguments, of course; see the reference documents for more information. As an example, you can count and output the number of threads created in a time period, gather statistics on the average lifetime of a thread, output the average memory recovered during a garbage collection, and so on.

Other Considerations

One question you might have is "How expensive are these operations?" It depends on the operation. The probes we have looked at above are relatively inexpensive. Other probes, such as method-entry and method-exit probes, or those monitoring object allocation, are much more expensive and indeed have a significant cost when enabled, even when they are not being used.

For this reason, there is a JVM option -- -XX:+ExtendedDTraceProbes -- which when present enables these and some other expensive probes. For more information, see the "Effect of using DTrace with Java Programs" section of the Observability Using Java Platform, Standard Edition 6, and Solaris OS article. If this option is not enabled, DTrace will ignore any of the above-mentioned probes in your profiling application, sparing you from inadvertently slowing down someone's production server.

That's the end of this quick tour. As I mentioned earlier, there are a lot of resources available on DTrace, specifically the user guide on the DTrace wiki (see the "Documentation" link, or go directly to this link).

FlexMonkey 1.0 Released

2009-07-15T04:54:00.000-07:00

My entries here are usually about Java, but today I want to call attention to the 1.0 release of FlexMonkey, an Adobe AIR app for testing Flex and Adobe AIR applications (I work for Gorilla Logic, the creator of FlexMonkey). If you have a need to run automated tests of Flex and AIR apps, please give it a try. If you are familiar with "old" versions of FlexMonkey (as in, 2008), take another look at it; it has evolved quite a lot during its short lifetime!

To learn about FlexMonkey and to download, go to http://flexmonkey.gorillalogic.com/. Don't let the "Beta 1" fool you; I've been using FlexMonkey since its birth and it is significantly more polished than a certain operating system I use every day. There is also a site describing Gorilla Logic's Flex service offerings at http://www.gorillalogic.com/what.development.services.flex.html.

Finally, I should mention that you are not alone should you become a FlexMonkey user. I've been on a lot of support mailing lists in my time, and I am very impressed by the dedication of the user community (including a number of Gorillas!) who respond to questions. So by all means give it a try with the confidence that you'll have all the help you need for both "newbie" and more advanced questions!

Instrumenting running Java processes with the com.sun.tools.attach API

2009-07-04T15:37:00.000-07:00

In recent posts, I've investigated using the java.lang.instrument package to instrument Java classes for a simple profiler. The package summary documentation describes a standard method for instrumenting a target Java application at launch time (using the static premain() method of the profiling agent). It also describes the outline of a procedure to attach to, and instrument, an already-running application (using a static agentmain() method). As the package documentation says, the details of how to initiate this process attachment are implementation specific. In this entry I'll describe how this process works for Sun Hotspot JVMs.

This subject has been discussed in a few other blogs (search on the package name); I'm sharing my experience because some of the pitfalls haven't been covered too well in other postings, and the more hits you can get on a particular stack trace, the better.

If you've used JDK tools like Java VisualVM or JConsole, you've seen the attach API at work. If you're going to attach to a running process, you need to get a handle for it (a VirtualMachineDescriptor, specifically), which implies a method for listing processes (namely, VirtualMachine.list()). These are the process descriptors you see in the above tools' connect dialogs at launch time. The best place to start is to look at the API documentation for com.sun.tools.attach. This information can be found in the tools documentation of your JDK documentation download, or online. From there, take a look at the documentation for the com.sun.tools.attach.VirtualMachine class -- it provides an outline for getting started. I'll start with this, pointing out some of the issues you can run into along the way.

The basic process is outlined below:

Create a public static void agentmain() method in your profiling agent (if you have already created a launch-time profiling agent, this can contain the same logic as your premain() method).
Add an Agent-Class: attribute to your profiler's manifest file, specifying the fully-qualified name of your profiler class
In your launch application, perform the following steps:

Obtain a descriptor for the target JVM.
Obtain a VirtualMachine instance with the static VirtualMachine.attach() method.
Load your profiling agent into the target JVM with the VirtualMachine.loadAgent() method.
Detach from the target JVM

At this point, your profiler's agentmain() method will be called and you are in business.

Note that on Windows, a String version of the target JVM's Windows process ID can be passed as the id for the VirtualMachine.attach() method.

A number of issues can arise during this process. First, note that you need the full JDK, not just the JRE, to do this. The reason is that you need the JDK's tools.jar file.

One of the first issues you might see is the following:

java.lang.UnsatisfiedLinkError: no attach in java.library.path

AttachNotSupportedException: no providers installed

com.sun.tools.attach.AttachNotSupportedException: no providers installed

My diagnosis of this problem is that I was executing java on the command line, which as a result of my JDK and JRE installation choices was pointing to the java.exe in the %JAVA_HOME%/jre/bin directory. While this is technically the same executable, you will find that when using the instrumentation packages, the java executable expects to find tools.jar using a relative path, and that path is relative to the java.exe in the %JAVA_HOME%/bin directory. So ensure at launch time that you are running the java executable in the JDK bin, not the JRE bin, directory.

Another more interesting problem you can run into when adding your agent jar to the target JVM is that the target JVM doesn't necessarily know about the dependencies of your agent. For example, all agents will have a dependency on tools.jar. The one I have been building has an additional dependency on the Javassist javassist.jar file. If your target application does not have these files on its classpath, it will fail to start your agent. For the record, in the console window (if you have one) for your target JVM (not the console from which you are trying to attach), you'll see something like

Exception in thread "Attach Listener" java.lang.NoClassDefFoundError: com/sun/tools/attach/AgentInitializationException

But you can't very well require that all potential target applications be launched with your dependencies on their classpath. So what to do?

The answer lies in the Boot-Class-Path: attribute in the profiling agent manifest file (see the package summary documentation for java.lang.instrument). Entries in this attribute are space-separated jar files. So to get past this issue, I added the paths to tools.jar and javassist.jar to this attribute in my profiler's manifest file.

At this point, you should be able to launch your agent. When you launch your profiler agent at target JVM launch time, your class transformer will be called as each class is loaded. When you launch your profiler by attaching to an already-running JVM, you will need to transform classes which have already been loaded. In my profiler, I do this by calling the getAllLoadedClasses() method of the java.lang.instrument.Instrumentation instance, then calling retransformClasses() on them. As you retrieve this set of classes, you might notice that you can check to see if the class is modifiable (using another method in the Instrumentation instance).

Note that according to the java.lang.instrument package summary documentation, you need the manifest attribute Can-Retransform-Classes: to be able to retransform a class. This attribute is not related to instrumenting at launch time versus after launch time (in either case, you could instrument the target application by transforming its classes only one time), but rather is related to the issue of transforming the same files multiple times (for example, to remove your profiling code). I'll discuss retransforming classes a little later.

Another important point to keep in mind is the call to add your transformer to your Instrumentation instance: this should set the canRetransform attribute to true (use the 2-argument signature addTransformer(ClassFileTransformer, boolean), since the default value for the attribute is false). Although the class files had not yet actually been transformed at attach time (when I launch my profiler by attaching to an already-running JVM), I noticed in my own case that if the canRetransform attribute is false, none of my methods get instrumented. I believe the reason for this behavior is simply that I am calling retransformClasses() when I attach to an already-running JVM (I see no other available method to use), and that method will not invoke the transformer's transform() method if the canRetransform attribute is false.

I specifically ignore the classes of my own profiler when I perform this operation -- in my case I get a stack overflow error if I try to method-entry and method-exit instrument the very methods which process these events! You may have similar issues if you are building a similar type of profiler. I also sometimes get odd and unrepeatable (including core-dumping) behavior if I attempt to instrument core Java classes, so I am more inclined to exempt those from profiling (at least of the method-entry/exit kind).

An interesting question to consider is what happens when you terminate your profiling agent. Mine is included in a Swing application. Note that since your agent is launched in the same JVM as the target, you might not want to call System.exit() anywhere -- your profiler users (including yourself) may not want to kill their target application just to close your profiler. But thinking about this brings up another interesting question -- when you close your profiler, what happens to the instrumented classes in your target application? The short answer is that they continue to be instrumented, possibly at significant cost to performance. You don't want to force your profiler users to bounce their application to rid themselves of your bytecodes. So the question is: how to get the target application back to its pre-instrumented state?

Some clues to this question can be found (again!) in the package documentation, specifically the retransformClasses() method of java.lang.instrument.Instrumentation. The description of this method states that this method starts with the intial class bytes of the class, before it was transformed (some differences may exist with the original class bytes, but they are considered cosmetic in the classfile-structure sense). Further reading of this description suggests that we should be able to revert to the original class bytes by (1) removing our current bytecode-manipulating transformer from our Instrumentation instance, and (2) calling retransformClasses() on all classes. Doing this should result in reversion to uninstrumented code, something that we should politely do before closing our profiler. Besides, not doing so results in strange behavior if you launch another profiler against the same running JVM, since you will be re-instrumenting classes you had already instrumented with your previous profiler instance.

Looking at what we need to do to make this work, probably the safest thing to do is to always outfit the Instrumentation class with no ClassFileTransformer and call retransformClasses() first, before then attempting to instrument the target JVM's classes. This behavior will ensure that you are always starting with the original class bytes (at target JVM launch time) before you transform. Then, at profiler-close time, perform the same instrumentation-clearing operation. In other words, the code would look something like:

// initialization code:

MyTransformer transformer = new MyTransformer();

instrumentationInstance.addTransformer(transformer);

...

// to instrument, first revert all added bytecode:

instrumentationInstance.removeTransformer(transformer);

// call retransformClasses() on all modifiable classes...

instrumentationInstance.addTransformer(transformer);

// call retransformClasses() on all modifiable classes...

...

// before exiting application:

instrumentationInstance.removeTransformer(transformer);

// call retransformClasses() on all modifiable classes...

Next, I will verify that this approach works by implementing the above code and running a debug version of my profiler to see if my instrumentation code has been removed.

My first concession to efficiency was to skip the initial revert-to-uninstrumented step, and only revert when I know that the classes have actually been instrumented. The reason is that this is a fairly expensive operation and it isn't necessary to perform when the target application is being instrumented for the first time.

The second issue I had was specific to Javassist. Once certain operations have been performed on a Javassist class model, that class is "frozen" and cannot be retransformed without first "defrosting" it (see Getting Started with Javassist for details). I noticed that, in spite of the fact that I was creating a new Javassist class instance from classfile bytecodes, I get a "class is frozen" RuntimeException when I attempt to transform classes an additional time, so I got around the issue with code like the following:

CtClass cc = cp.get(javassistClassName);

     // for repeated instrument/uninstrument cycles, 
     // make sure the class is unfrozen before we
     // attempt to do anything to it:
     if (cc.isFrozen())
     {
  cc.defrost();
     }

Looking at the above code reminds me of a minor point you should know -- the fully-qualified class names that the Java runtime passes to your class file transformer's transform() method are slash delimited, while Javassist expects dot (".") delimiters. So to match names, you need to something like the following:

     String javassistClassName = className.replace('/', '.');

where className is the class name as passed in to transform().

The above procedure should allow you to stop and start profiling (by un-instrumenting and instrumenting classes) and also to clean up before you close your profiler application, so you don't leave expensive profiling code in your target application at profiler disconnect time. I tested this procedure in my profiler with a pair of stop/start buttons and using Javassist (and the defrost() method shown above) and verified that the target application does indeed revert to pre-instrumented bytecodes.

I keep saying I'm going to do something really interesting with this profiling data (and I will!), but in the process of investigating this API I decided it merited its own entry. Soon, and maybe in my next post, I'll discuss generating events from this data and running them through an event processor.

Java Profiling, continued

2009-06-29T10:32:00.000-07:00

I've been building a Java profiler lately, partly to address what I think are shortcomings in the current set of free/shareware profilers, and also to enjoy the flexibility of obtaining only the profiling information I need. If you've spent much time with profilers, you already know they can have quite an impact on the application they're profiling! My goal with my own profiler is to obtain a narrow set of information, having more control over the size and frequency of the data collection.

One feature I really like about hprof (see http://java.sun.com/developer/technicalArticles/Programming/HPROF.html) is the heap=sites option. If you're not familiar with this option, it is described in the above link, but in short, hprof can provide a profile of the allocation of objects per site, where a site is a unique Java stack trace of a fixed depth. This can be very handy information when you are investigating the heap usage of an allocation, but it is extremely costly to get with hprof (in one recent case I was investigating, starting an application with the hprof agent slowed the application down by several orders of magnitue).

Today I'll discuss how I get this same information without the overhead of hprof. I'll build on my last blog entry about the java.lang.instrument package and combine bytecode injection with the ThreadMXBean to get this same information.

The basic outline is:

Create a java.lang.instrument profiling agent.
Intercept all constructors as they are loaded, instrumenting them on constructor entry and constructor exit. Information passed back to the profiler application will include the thread ID and the size of the object created.
In the profiling agent, keep track of the total memory allocated for each object type, as well as a list of stack trace signatures at object instantiation, with frequency of each. The stack trace is obtained via the ThreadMXBean of the JVM.
Rank objects by total amount of memory allocated, the set of unique stack trace signatures for each constructor, and the number of times the object was instantiated via each stack trace.

This information will approximate the heap=sites output of hprof, and can be obtained more economically for a number of reasons:

We can restrict the classes (constructors) instrumented. Note that if we do not instrument every class, we may not see every instance of a particular object's instantiation, but in practice what you usually care about are objects instantiated by your application. Learning in hprof that 20 MB of java.lang.Objects were created as a side effects of a java.* method call usually doesn't help diagnose performance problems.
The custom profiler can get this information in a very small fraction of the same time that hprof would require to get it, as it is a much more targeted application.
With the custom profiler, we have a lot more control over the interval during which the data is collected. For example, we can clear our profiler's data, initiate an operation in the profiled application, and see just the allocation history for the time period of interest. I actually do not know how to obtain this information with hprof; the heap=sites option dumps its summary at application termination (as a snapshot of the heap at that point in time), which may not be what you want.

As I mentioned in my last entry, I use Javassist (see http://sourceforge.net/project/showfiles.php?group_id=22866&package_id=80766). Javassist will create a Java class representation from the bytecode array, and from there it is very easy to obtain the constructors, using CtClass.getConstructors(). The CtConstructor.insertBefore() and insertAfter() methods round out all you will need from Javassist to instrument your code.

Object size is a pesky issue in Java, especially if you have experienced the convenience of the sizeof() operator. I use the java.lang.instrument.Instrumentation instance, supplied by the JVM in the call to the profiler's premain() method, to get the size of the objects (using getObjectSize(Object)). It isn't exactly clear what size this method returns; the javadocs are a little cagey and it's safe to safe this value isn't a deep copy. While it is possible to drill down through object references in a class and attempt to accumulate a deep-copy value, this operation itself can be very time-consuming, incomplete (due to inaccessible attributes) and slightly misleading (static references, anyone?). Not to mention you can easily get a stack overflow error in the process. It may not be perfect, but I have chosen to simply use the value returned by Instrumentation.getObjectSize(Object), and to keep in mind that what I'm seeing is useful in a rough, rather than a precise, sense.

Since the profiler runs in the same JVM as the profiled application, it is fairly simple to get a reference to the JVM ThreadMXBean, using java.lang.management.ManagementFactory.getThreadMXBean(). Given a thread ID (which you can arrange for your instrumented constructor to pass back to your profiler), you can get the stack trace for the thread with ThreadMXBean.getThreadInfo(long,int). Note that there are several signatures for this method call; remember to pick the one where you specify the maximum depth of the stack trace, as the default value is zero!

The value of collecting information like this in a profiler should be self-evident; however, there are greater opportunities available for this data. For example, a variety of profiling information can be used to generate events which can be used to monitor the health of the profiled system. These don't all need to be just events generated by the profiler; they could include SCM and system-maintenance events, with the result being that maybe you could see that your ASN.1 parser suddenly takes three times as long to parse a switch file since the last change was checked in and rolled out to production. In a later posting, I'll look at how such a system could be implemented, looking at the open-source tools available to analyze this kind of data and make associations between events that can provide much faster resolution of operational and performance issues.

Java Profiling with the java.lang.instrument Package

2009-05-15T12:59:00.000-07:00

Lately I've been looking at profilers and wondering how easy it would be to write one, mostly to be able to provide a little more sophisticated guidance to performance investigations. For example, while it is useful to pop a profiler and see what methods are consuming the most time in an application, wouldn't it be even more interesting if your profiler could generate an event any time your application spent more than a specified amount of time in a method, like 25% more than the average time (or fell more than 2 sigma outside the mean)?

In general, getting events when something interesting happens is more helpful than watching a display and waiting for anomalies (how many monkeys does that employ?!). That's why I'm also a little disappointed in the JVM's built-in JMX beans, only two of which (MemoryMXBean and MBeanServerDelegateMBean) implement the NotificationEmitter interface. I would much prefer registering for notifications from the garbage collector JMX bean than triggering verbose gc output (with command-line options) and then following and parsing log output!

In this posting I'm going to show you how quickly you can instrument a Java application, and hopefully it will look easy enough that you will give it a try. First, I'd recommend reading the package documentation for java.lang.instrument. It isn't a large package and the time spent is well worth it.

As usual, let's get the setup out of the way. For pre-requisites (such as Java SDK, Maven, etc.), please see one of my earlier posts. Assuming you have the Java SDK and Maven 2 (along with the usual Maven environment variables created), you're ready to go. In a directory containing your development projects, enter the following:

mvn archetype:generate -DgroupId=com.example.profiler -DartifactId=JavaProfiler

Accept all the defaults and wait for your project directory to be generated.

Change into the new project directory, open the pom.xml file and add the following build plugins:

<build>

<artifactId>maven-compiler-plugin</artifactId>

</configuration>

</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-jar-plugin</artifactId>

<manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>

</archive>

</configuration>

</plugin>

</plugins>

</build>

The first plugin specifies Java SE6 (rather than the default 1.3!) and the second one says we're going to be supplying our own manifest file. More on that below.

Our basic profiler is responsible for doing a couple of things:

It must supply a premain method, per the requirements discussed in the package documentation;
It must implement the ClassFileTransformer interface (or delegate to a class which does; we're keeping it simple here).

Rename the App.java class to SimpleProfiler.java. For our first cut, we will not bother parsing agent arguments (again, see the package documentation), and our interface-specified transform class will simply print the name of the class and return null. In other words, as the runtime passes us the bytecodes of each class, rather than manipulate them and return an instrumented version, we'll just note the class name and size and let it go. To save you some time, here is what a simple implementation would look like:

package com.example.profiler;

import java.lang.instrument.ClassFileTransformer;

import java.lang.instrument.IllegalClassFormatException;

import java.lang.instrument.Instrumentation;

import java.security.ProtectionDomain;

/**

* Simple java.lang.instrument profiler

public class SimpleProfiler implements ClassFileTransformer

{

protected String agentArgString = "";

protected Instrumentation instrumentation;

public static void premain(String agentArgs, Instrumentation inst)

{

SimpleProfiler profiler = new SimpleProfiler(agentArgs, inst);

}

public SimpleProfiler(String agentArgs, Instrumentation inst)

{

agentArgString = agentArgs;

instrumentation = inst;

instrumentation.addTransformer(this);

}

public byte[] transform(ClassLoader loader,

String className,

Class<?> classBeingRedefined,

ProtectionDomain protectionDomain,

byte[] classfileBuffer)

throws IllegalClassFormatException

{

System.out.println("SimpleProfiler.transform(): class: " + className + " (" + classfileBuffer.length + " bytes)");

return null;

}

This class

provides the premain method that the runtime needs to launch this profiling agent when the target application launches
adds itself as a class file transformer using the Instrumentation object passed to it by the runtime
implements a very simple transform method (returning null means the transformer does not modify the class file)

Now, the runtime will need to know how to launch the profiler, and that is why you need a manifest file. Create the directory structure resources/META-INF under the src/main directory, and add the following single-line MANIFEST.MF file:

Premain-Class: com.example.profiler.SimpleProfiler

Now you should be able to build with

mvn clean install

To see our profiler in action, modify the command line of any Java application you have with the following:

-javaagent:<path to your profiler project directory>/target/JavaProfiler-1.0-SNAPSHOT.jar

This option will cause the premain method of the profiler to be called before the target application's classfiles are loaded, and will give the profiler an opportunity to modify their bytecodes as they are loaded.

I ran a simple Java-based JDBC query tool with this agent and my output from the transform method looked like the following:

SimpleProfiler.transform(): class: Query (21580 bytes)

SimpleProfiler.transform(): class: java/sql/SQLException (2836 bytes)

SimpleProfiler.transform(): class: java/awt/Component$2 (854 bytes)

SimpleProfiler.transform(): class: java/awt/Component$4 (777 bytes)

SimpleProfiler.transform(): class: java/sql/ResultSet (8938 bytes)

SimpleProfiler.transform(): class: java/sql/Wrapper (337 bytes)

SimpleProfiler.transform(): class: java/sql/ResultSetMetaData (1045 bytes)

SimpleProfiler.transform(): class: java/awt/AWTEvent$1 (539 bytes)

SimpleProfiler.transform(): class: sun/awt/AWTAccessor$AWTEventAccessor (270 bytes)

SimpleProfiler.transform(): class: sun/reflect/misc/Trampoline (482 bytes)

SimpleProfiler.transform(): class: sun/reflect/GeneratedConstructorAccessor1 (1277 bytes)

SimpleProfiler.transform(): class: sun/reflect/GeneratedMethodAccessor1 (918 bytes)

SimpleProfiler.transform(): class: javax/swing/text/WrappedPlainView (7040 bytes)

SimpleProfiler.transform(): class: javax/swing/text/BoxView (8593 bytes)

SimpleProfiler.transform(): class: javax/swing/text/CompositeView (6079 bytes)

SimpleProfiler.transform(): class: javax/swing/text/WrappedPlainView$WrappedLine (5054 bytes)'

and so on. You are probably thinking it would be nice to restrict the set of classes that are inspected (and possibly instrumented). This is where the agent argument string (passed to premain) comes in handy. This is a single java.lang.String, rather than the usual array as in the main method, but it's a simple matter to parse one. For example, you could pass in a comma-delimited list of regular expressions to determine what classes get inspected and which do not. If we don't do a little filtering, our profiler's performance will suffer and any output we create on the instrumentation process will be overwhelming, so I will provide sample argument-parsing code in the example without spending time discussing it.

I should say at this point that I've seen a number of blog postings on this subject, all of which stop here and say "Next, we'll use an open-source package to insert bytecodes etc.!". I have yet to see a follow-on where this actually occurs. I was thinking of putting this in a follow-on posting, but I got here (Starbucks) by bus, and on the off chance I get run over catching one home, let's go ahead and do something meaningful with the bytecodes before we end this post.

I looked at ASM, BCEL and Javassist, and decided on Javassist (Why? Not too much -- and not too little -- 'getting started' documentation.) Having tried it, it was so easy to use I haven't tried any of the others yet.

As you would expect, Javassist builds a representation of the class file by inspecting the bytecode array. It also includes some very handy methods for inserting blocks of code (which you specify in String form) at the beginning and ending of methods (among many other possibilities, of course). "Insert before" and "insert after" are exactly what I want for this profiler. I want to insert code to print out an "entry" or "exit" message in every method.

A simple way to do this with Javassist is to modify the transform method as below (preceded by some import statements). Note the addition of the regular-expression-processing code:

import java.util.regex.Matcher;

import java.util.regex.Pattern;

import java.util.regex.PatternSyntaxException;

import javassist.CannotCompileException;

import javassist.ClassPool;

import javassist.CtClass;

import javassist.CtConstructor;

import javassist.ByteArrayClassPath;

import javassist.CtMethod;

There is a new attribute for the profiler:

protected Pattern[] classRegexPatterns = null;

which gets initialized at the end of the constructor:

// agentArgString is list of regexes of classes to instrument:

if (agentArgString != null && agentArgString.length() > 0)

{

String[] classRegexes = agentArgString.split(",");

classRegexPatterns = new Pattern[classRegexes.length];

for (int i=0; i<classRegexes.length; i++)

{

classRegexPatterns[i] = Pattern.compile(classRegexes[i]);

}

Now, for the new transform method:

public byte[] transform(ClassLoader loader,

String className,

Class<?> classBeingRedefined,

ProtectionDomain protectionDomain,

byte[] classfileBuffer)

throws IllegalClassFormatException

{

for (int i=0; i<classRegexPatterns.length; i++)

{

Matcher matcher = classRegexPatterns[i].matcher(className);

if (matcher.matches())

{

try

{

// first modify the class name for javassist. convert slashes to dots:

String javassistClassName = className.replace('/', '.');

// turn bytecodes into a Javassist CtClass

ClassPool cp = ClassPool.getDefault();

cp.insertClassPath(new ByteArrayClassPath(javassistClassName, classfileBuffer));

CtClass cc = cp.get(javassistClassName);

// add output at the beginning and end of each method

CtMethod[] methods = cc.getMethods();

for (int k=0; k<methods.length; k++)

{

// do not instrument inherited methods:

if (methods[k].getLongName().startsWith(javassistClassName))

{

methods[k].insertBefore("System.out.println(\"Entering " + methods[k].getLongName() + "\");");

methods[k].insertAfter("System.out.println(\"Exiting " + methods[k].getLongName() + "\");");

}

// return the new bytecode array:

byte[] newClassfileBuffer = cc.toBytecode();

return newClassfileBuffer;

}

catch (Exception exc)

{

System.err.println(exc.getClass().getName() + ": " + exc.getMessage());

if (!(exc instanceof CannotCompileException))

{

// don't care about the stack trace if there was no method body, etc.:

exc.printStackTrace();

}

return null;

}

Note that the Java runtime returns class names slash-delimited, while Javassist expects them to be dot-delimited. Other than that detail, Javassist has no trouble creating and manipulating class file objects from our byte arrays. Also note that I have chosen not to instrument all methods. The Javassist full method name starts with the Javassist class name. To avoid instrumenting methods in parent classes (which normally results in Javassist's throwing an exception because it does not have access to the method body), I have chosen to restrict instrumentation to the methods defined for the class in question. You can remove the if statement and see what happens...

Note the insertBefore() and insertAfter() methods. These are also available for your target application's constructors (retrieved by CtClass.getConstructors()), but I left constructor instrumentation out of this profiler to keep it as short as possible.

To build the profiler, add the following dependency to your pom.xml file:

<groupId>javassist</groupId>

<artifactId>javassist</artifactId>

</dependency>

and rebuild with

mvn install

Your target application launch will now need to include the Javassist jar file in its classpath. If you have Javassist installed, just point to the jar file there. If not, you can find it in your local Maven repository, as Maven downloaded it to perform the updated build.

Now, relaunch your target application. Note from the package documentation that you pass in your agent argument string at the end of the -javaagent option, following an equals sign. For example, I used:

-javaagent:<path to your profiler project directory>/target/JavaProfiler-1.0-SNAPSHOT.jar=.*Query.*

Note as it runs that you are now seeing console output when methods are entered and exited. In my query-tool example, I see the following:

Entering Query.main(java.lang.String[])

Entering Query.layOutTool()

Exiting Query.layOutTool()

Entering Query.initializeTool()

Entering Query$DatabaseConnection.toString()

Exiting Query$DatabaseConnection.toString()

Entering Query.loadQueries()

Exiting Query.loadQueries()

Exiting Query.initializeTool()

Entering Query.addListeners()

Exiting Query.addListeners()

Entering Query.setInitialDividerLocations()

Exiting Query.setInitialDividerLocations()

Exiting Query.main(java.lang.String[])

Entering Query$DatabaseConnection.toString()

Exiting Query$DatabaseConnection.toString()

Entering Query$DatabaseConnection.toString()

Exiting Query$DatabaseConnection.toString()

Entering Query$DatabaseConnection.toString()

Exiting Query$DatabaseConnection.toString()

and so on.

Following the log output from this profiler isn't that exciting, either! But note what is now available to you. First of all, the profiler is loaded by the same classloader as your target application. This means that if you have methods in your profiler to process method-entry and method-exit events, you can insert bytecodes into the target application to call your profiler (I used a singleton pattern, with a static getInstance() method, etc., in my in-house profiler application). You can insert code to send along the system time stamp, and your profiler can keep a running total of the execution time, number of invocations, percentage of total time, and so on, of the target application. If you do something like this, be sure to include the full method signature in your output, as well as the calling thread's ID, to correctly match your entry and exit events.

Once you are collecting sufficient information, you can start monitoring it for anomalies and generate events (e.g. JMS messages) for incidents you find interesting, and customize your profiler for specific problems and customer situations.

Tapping into the GlassFishESB Worklist Manager

2009-03-24T14:03:00.000-07:00

I recently investigated the Worklist Manager in GlassFish ESB and want to share some of the details here. GlassFish is based on the Java Business Integration (JBI) specification and includes service engines for Java EE, BPEL and so on. The Worklist Manager is one of these service engines. It is not based on BPEL4People as far as I know but basically meets the same need -- providing a way to integrate human/workflow tasks into a business-process environment.

Note that there is a Wiki for this service engine at http://wiki.open-esb.java.net/Wiki.jsp?page=WLMSE. My goal when investigating the WLMSE was to demonstrate accessing it from a web service client external to the GlassFish environment. The documentation in this area is rather spotty, and hence this post. I started at the above Wiki and took things from there.

An important consideration here is that there are a number of versions of GlassFishESB, OpenESB, NetBeans, and so on, and the combination makes a difference. If you peruse enough online tutorials, you'll finally begin wondering what version, exactly, you need to download to see the same functionality as you are seeing in the tutorial. For example, the latest release, recommended on the WorkList Manager Service Engine (WLMSE) Wiki, is GlassFish ESB V2.1 Milestone 1. The download for this release includes NetBeans 6.5. It does not include the WLMSE, however, which the Wiki recommends obtaining as a separate download. The only problem with the provided download is that it is only compatible with the latest patch of NetBeans 6.1, and lower. So you will need an earlier version of NetBeans in order to install the WLMSE!

For reference, I used GlassFish V2 UR2 and NetBeans 6.1 Patch 4. It does not appear feasible at this time to use V2.1M1 with the WLMSE based on the above issue, so I recommend trying something a little older. My pre-requisites below send you to the older version that I used to develop this post.

Pre-requisites:

Get NetBeans 6.1 at http://www.netbeans.org/downloads/6.1/index.html, selecting the "All" column, which includes SOA support.
Java SE 6 installed: http://java.sun.com/javase/downloads/index.jsp. Be sure to get the JDK, not the JRE, as there are utilities you will need from the JDK.
Maven 2 installed: http://maven.apache.org/download.html
The following environment variables:

JAVA_HOME: top-level Java SE 6 directory
M2_HOME: top-level Maven 2 directory
M2: $M2_HOME/bin or %M2_HOME%\bin
MAVEN_OPTS: -Xms256m -Xms512m (optional)
PATH: $JAVA_HOME/bin:$M2:$PATH or %JAVA_HOME%\bin;%M2%;%PATH%

The Java SE JDK, Maven 2 and the environment-variable settings are for the Java-based web-service client we'll be building after we deploy an instance of the Worklist Manager.

Create a Worklist Application

Start NetBeans. Once the IDE is up, select the Services tab, expand "Servers", then right-click on "GlassFish V2" and select "Start". Once the GlassFish server has started, select the Projects tab.

From the "File" menu, select "New Project...", then select category "SOA", then select "Worklist Module". Enter "TutorialWorklistApp" for the project name and click "Finish".

Our worklist application will start with a single operation. Just as an example, assume this is an ETL system and we want to create a trouble ticket on failure of a file to load. The creation of the application will be a lot smoother if we first create the XML Schema for the request and response messages, then create the WSDL definition to define the operation, then create the worklist Task itself.

The XML Schema

Right-click on the "WorklistFiles" directory under "TutorialWorklistApp", select "New" -> "XML Schema...". If "XML Schema..." is not listed, select "Other...", then "XML", then "XML Schema", then "Next". Name the schema tutorial_worklist (NetBeans will supply an .xsd extension) and click "Finish". The schema will open in a graphical schema editor. Provide elements for the request and response messages and base them on complexTypes containing information that might be needed for this application. An example schema is below:

<?xml version="1.0" encoding="UTF-8"?>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://xml.netbeans.org/schema/tutorial_worklist" xmlns:tw="http://xml.netbeans.org/schema/tutorial_worklist" elementFormDefault="qualified"> <xsd:element name="loadFailureActionRequest" type="tw:LoadFailureActionRequestType"/> <xsd:element name="loadFailureActionResponse" type="tw:LoadFailureActionResponseType"/> <xsd:complexType name="LoadFailureActionRequestType"> <xsd:sequence> <xsd:element name="feedName" type="xsd:string"/> <xsd:element name="fileName" type="xsd:string"/> <xsd:element name="timestamp" type="xsd:date"/> </xsd:sequence> </xsd:complexType> <xsd:complexType name="LoadFailureActionResponseType"> <xsd:sequence> <xsd:element name="troubleTicketID" type="xsd:string"/> </xsd:sequence> </xsd:complexType>

</xsd:schema>

Save the schema and check it with the "Validate XML" button.

The WSDL definition

Right-click on the "WorklistFiles" directory under "TutorialWorklistApp", select "New" -> "WSDL Document...". If "WSDL Document..." is not listed, select "Other...", then "XML", then "WSDL Document", then "Next". Name the WSDL document "tutorial_worklist" (NetBeans will supply a .wsdl extension) and click "Finish". Select "Concrete WSDL Document", then "SOAP" Binding and "Document Literal" for the Type. Click "Next".

In the "Abstract Configuration" panel, clean up some of the names a little. Change the port type name to worklistPortType, the operation name to processLoadFailure, then click the ellipses ("...") for the Input and Output messages to select the input and output messages we defined in the XML Schema above (namely, loadFailureActionRequest and loadFailureActionResponse; you will need to scroll up in the dialog to find the definition, and expand the "Elements" node). Leave the "Generate partnerlinktype automatically" box checked. Click "Next".

On the "Concrete Configuration" panel, clean up the names a little, using worklistBinding, worklistService, and worklistPort for Binding Name, Service Name, and Port Name, respectively. Click "Finish".

The Task definition

Right-click on the "WorklistFiles" directory under "TutorialWorklistApp", select "New" -> "Worklist Task Definition...". If "Worklist Task Definition..." is not listed, select "Other...", then "SOA", then "Worklist Task Definition", then "Next". Name the file processLoadFailure (no need for an extension), click the ellipsis by the empty Operation name, expand the WSDL definition down to the processLoadFailure operation, and click "OK". Click "Finish". The Task definition will now open in a graphical editor.

Build

Right click on the name of the Worklist Application ("TutorialWorklistApp") and select "Build". You should get a successful build. This application is now a JBI module which can be deployed to a composite application, which is our next step.

Create a Composite Application

From the "File" menu, select "New Project...", then project category "SOA", then "Composite Application". Click "Next". Name the project TutorialCompositeApp. Click Finish. The Service Assembly file TutorialCompositeApp.casa will open in a graphical editor containing three sections:

WSDL Ports
JBI Modules
External Modules

Drag the Worklist Application (TutorialWorklistApp) over to the "JBI Modules" section of the service assembly and drop it. At this point, your service assembly will look as follows:

Right-click on the project name and select "Build". You will see the service-assembly view reload, with the following results:

Note that the processLoadFailure operation has a configured SOAP/HTTP binding, as we expected (we just configured it in some detail), but notice that we also have a TaskCommonPort, with no binding. This is the Worklist Manager. We will add a SOAP/HTTP binding for this port so that we will be able to access the items in the Worklist from a simple demonstration Java client.

Create the Worklist Manager binding

From the "WSDL Bindings" section of the Palette, drag and drop a "soap" binding on the "WSDL Ports" section of the service assembly. You will see an additional binding similar to the one already configured for the Worklist application. Select the purple (right-pointing) arrow in the binding and drag over to and connect to the TaskCommonPort port. Now, right-click on this binding and select "Properties". Change casaPort1 to worklistManagerPort, then click on the ellipsis ("...") by the soap:address Location. You should see the following:

http://localhost:${HttpDefaultPort}/casaService1/worklistManagerPort

Simplify this a little by changing it to

http://localhost:${HttpDefaultPort}/worklistManager

Note that the URL for the Worklist application is similarly wordy. If you want to shorten it, you can in similar manner, but you'll have to first right-click on the binding and select "Clone WSDL Port to edit...". Confirm on the dialog which follows, then right-click and select "Properties" to edit, and change the soap:address Location from

http://localhost:${HttpDefaultPort}/tutorial_worklistService/tutorial_worklistPort

http://localhost:${HttpDefaultPort}/worklistApp

then click "Close" to save. Your service assembly should now look as follows:

Deploy the Composite Application

Before we deploy the application, you're probably wondering what ${HttpDefaultPort} means. You might guess that it is the default port of the GlassFish web services, but actually each of the binding components in GlassFish has their own unique default port. This can be a problem if you are trying to access a binding component from outside of GlassFish and don't know the port number, but these can be found in the Admin console. For more information, see "Understanding the ${HttpDefaultPort} Token" at http://wiki.open-esb.java.net/Wiki.jsp?page=ClusteringSupportForHTTPBindingComponent. But I will walk you through the process here.

To determine what the default port is for a binding component (in this case, the HTTP BC):

Open the Admin console (in Netbeans, open Services tab, expand "Servers", right click on Glassfish, select "View Admin Console". Unless you changed values, the username/password pair is "admin"/"adminadmin".
Under "Common Tasks", expand "JBI", "Components", "sun-http-binding".
Select the "Configuration" tab.
Second entry is "Default HTTP Port Number". If you did not change this on installation, it will probably be 9080.

If you choose to, you can always select an unused port number and hard-code it into the bindings before you deploy. If you're like me, all you care about is being able to find out where the service is deployed!

Now, right-click on the composite application name in the "Projects" tab, select "Build", then again right-click on the application name and select "Deploy". The project should deploy successfully.

To verify that the two services are correctly deployed, and assuming you used the soap:address Locations I used above, navigate to the following:

http://localhost:9080/worklistManager?wsdl

http://localhost:9080/worklistApp?wsdl

where I have substituted the HTTP default port for my deployment. Note you may have to view the page source to see the WSDL definition, but you should see both. Note the worklistApp has a single operation available (processLoadFailure), while the worklistManager has a number of operations related to the processing of tasks, as we might expect. For example, GetTaskList, RevokeTask, CompleteTask, and so on. We will be calling these operations from the simple Java web-service client which we will create next.

Create a simple Java client

All we'll be doing here is generating the minimal Java client necessary to exercise some operations on the Worklist Manager. The steps involved are almost identical to those under the heading "Create a client for the web service" at my earlier post Java-SE-Based Web Services . The only difference will be the URL of the web service, of course. Because the WSDL definition of the Worklist Manager contains a URL to the TaskCommon.wsdl file (the latter is the abstract WSDL, and the former imports it and contains the necessary information to make the WSDL definition concrete), we will copy both files to a wsdl directory in our client build.

Expand the NetBeans project directory for the composite application and retrieve the following files:

src/JbiServiceUnits/TutorialCompositeApp.wsdl

src/JbiServiceUnits/TutorialWorklistApp/TaskCommon.wsdl

Copy these files into the wsdl directory (under your Maven project src directory). Open the TutorialCompositeApp.wsdl file and change the import path for TaskCommon.wsdl to reflect the fact that it is now in the same directory as TutorialCompositeApp.wsdl:

<import namespace="http://jbi.com.sun/wfse/wsdl/TaskCommon" location="TaskCommon.wsdl">

Also, edit the soap:address, replacing ${HttpDefaultPort} with the actual port number (defaults to 9080, or use whatever value you hard-coded, if any).

Then, instead of configuring a WSDL URL in your pom.xml file, configure a WSDL file as follows:


           <configuration>
             <wsdlFiles>
               <wsdlFile>TutorialCompositeApp.wsdl</wsdlFile>
             </wsdlFiles>
             <packageName>com.example.webservice.service</packageName>
           </configuration>

The client of course instantiates a different service and invokes different operations than in my previous post. Here is an example Java client that exercises a couple of operations on the Worklist Manager:


package com.example.webservice.client;

import com.example.webservice.service.QueryType;
import com.example.webservice.service.CasaService1;
import com.example.webservice.service.TaskCommonPortType;
import com.example.webservice.service.TaskFaultMsg;
import com.example.webservice.service.TaskListType;
import com.example.webservice.service.TaskType;

public class WLMClient
{
 public static void main(String[] args) throws TaskFaultMsg
 {
   CasaService1 service = new CasaService1();
   TaskCommonPortType port = service.getWorklistManagerPort();
   System.out.println("Getting Task List:");
   QueryType qt = new QueryType();
   QueryType.Users users = new QueryType.Users();
   users.getUser().add("john");
   qt.setUsers(users);
   QueryType.Groups groups = new QueryType.Groups();
   groups.getGroup().add("CustomerServiceRep");
   qt.setGroups(groups);
   TaskListType tasks = port.getTaskList(qt, "john", 0, 100);
   System.out.println("  Total record count: " + tasks.getTotalRecords());
   System.out.println("  Returned record count: " + tasks.getReturnedRecords());
   for (TaskType nextTask: tasks.getTask())
   {
     System.out.println("Task ID: " + nextTask.getTaskId() + "; title: " + nextTask.getTitle() + "; " +
                        "Claimed by: " + nextTask.getClaimedBy() + "; Assigned to: " + nextTask.getAssignedTo());
   }
 }
}

Build and run this application and you should see output like the following:


C:\Users\wayne.adams\Blogger\GlassfishWLMJavaClient>java -cp target\glassfishWLMJavaClient.jar com.example.webservice.client.WLMClient
Getting Task List:
 Total record count: 0
 Returned record count: 0

In this case, there are no tasks, but we can create some by exercising the other service, the worklistApp service. More on that in a subsequent post!

Java-SE-based Web Services

2009-03-24T10:34:00.000-07:00

This isn't exactly late-breaking news, as Java SE6 has been out for a while, but the Standard Edition now includes support for non-trivial web-service development and deployment, no Java-EE application server necessary. The purpose of this very quick introduction is to produce a working web service and a web service client, in as few lines of Java as possible, without relying on heavy infrastructure (namely, an IDE and/or an application server). The output of this post will re-appear in subsequent posts, for example when I need a quick client to kick off a BPEL (Business Process Execution Language) process, or simply to exercise a web service to verify deployment.

Pre-requisites:

Java SE 6 installed: http://java.sun.com/javase/downloads/index.jsp. Be sure to get the JDK, not the JRE, as there are utilities you will need from the JDK.
Maven 2 installed: http://maven.apache.org/download.html
The following environment variables:

JAVA_HOME: top-level Java SE 6 directory
M2_HOME: top-level Maven 2 directory
M2: $M2_HOME/bin or %M2_HOME%\bin
MAVEN_OPTS: -Xms256m -Xms512m (optional)
PATH: $JAVA_HOME/bin:$M2:$PATH or %JAVA_HOME%\bin;%M2%;%PATH%

Create a Web Service project and launch the service

As you've guessed, I'm using Maven to build this service. Maven will generate the project directory for you, so there's no need to create it first. Just open a command prompt or shell in a development directory and enter the following:

mvn archetype:generate -DgroupId=com.example.webservice -DartifactId=JavaSEWebService

substituting your desired package name for groupId. The artifactId will be the name of the project directory Maven will create in your current working directory. You will be asked for an archetype, where the default is 15, or maven-archetype-quickstart. Select defaults by hitting "Enter" until Maven generates the project.

Next, create a simple web service. Note: I prefer to start with a WSDL definition first, then generate the server and client-side artifacts from it, but for this quick example we'll go in the opposite direction, taking a small piece of business logic and exposing it as a service.

cd to the the groupId-defined package under the source branch (./{artifactId}/src/main/java/...) of the newly-created project. There will be a class called App.java. We will ignore this class and create our own.

Create a directory called service and cd into it.
Create a Java class called Tutorial.java.
Add a couple of methods; one called getHelloWorld that takes no arguments and returns the String "Hello, World!" and a second one called getPersonalizedHelloWorld that takes a String and returns a personalized greeting.
Add a main method, and in it, instantiate a Tutorial and publish it, using the javax.xml.ws.Endpoint class, on your machine. Annotate the class as a WebService and the methods as WebMethods.

An example class is shown here:

package com.example.webservice.service;

import javax.jws.WebMethod;
import javax.jws.WebService;
import javax.xml.ws.Endpoint;

@WebService
public class Tutorial
{
public static void main( String[] args )
{
  Tutorial wsInstance = new Tutorial();
  Endpoint.publish("http://localhost:8282/tutorial", wsInstance);
}
@WebMethod
public String getHelloWorld()
{
  return "Hello, World!";
}
@WebMethod
public String getPersonalizedHelloWorld(String yourName)
{
  return "Hello, " + yourName + "!";
}
}

Next we'll modify the Maven "pom" (Project Object Model) file, first to use Java SE 6 (Maven defaults to 1.3) and secondly to specify the wsgen Maven plugin. wsgen creates all the artifacts necessary to build and deploy the web service. cd into the project directory and open the pom.xml file. Add the following to your pom:


<build>
  <finalName>tutorialWebService</finalName>
  <plugins>
    <plugin>
      <artifactId>maven-compiler-plugin</artifactId>
      <configuration>
        <source>1.6</source>
        <target>1.6</target>
      </configuration>
    </plugin>
    <plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>jaxws-maven-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>wsgen</goal>
            </goals>
            <configuration>
            <sei>com.example.webservice.service.Tutorial
            </sei>
            <genWsdl>true</genWsdl>         
       <keep>true</keep>
            </configuration>
          </execution>
        </executions>
      </plugin>
  </plugins>
</build>

Save and close the file. From the project directory, enter mvn install. You should get a clean build. Note that the name of the produced .jar file is obtained from the finalName element in the pom.xml file. Start the web service from the project-level directory with:

java -cp target/tutorialWebService.jar com.example.webservice.service.Tutorial

and verify that the service is available by retrieving the WSDL description in a browser. Append ?wsdl to the endpoint URL you defined in your Java class, in this case

http://localhost:8282/tutorial?wsdl

Note that depending on the browser you are using, you may need to view the page source to see the WSDL description.

Create a client for the web service

We'll create a new Maven project for the client. Open another command prompt or shell in your development directory (again, no need to create the project directory, as Maven will do this for you), and enter the following:

mvn archetype:generate -DgroupId=com.example.webservice -DartifactId=JavaSEWebServiceClient

substituting your desired package name for groupId. The artifactId will be the name of the project directory Maven will create in your current working directory. As for the web service example, you will be asked for an archetype, where the default is 15, or maven-archetype-quickstart. Select defaults by hitting "Enter" until Maven generates the project.

cd into the project directory, given by the value you supplied for artifactId (for this example, JavaSEWebServiceClient). Open the pom.xml (Maven Project Object Model) file. Add the following to the POM; wsimport is used to generate client stubs for your web service:

  <build>
   <finalName>tutorialWebServiceClient</finalName>
   <plugins>
     <plugin>
       <artifactId>maven-compiler-plugin</artifactId>
       <configuration>
         <source>1.6</source>
         <target>1.6</target>
       </configuration>
     </plugin>
     <plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>jaxws-maven-plugin</artifactId>
         <executions>
            <execution>
              <goals>
                <goal>wsimport</goal>
              </goals>
              <configuration>
                <wsdlUrls>
                  <wsdlUrl>http://localhost:8282/tutorial?wsdl</wsdlUrl>
                </wsdlUrls>
            <packageName>com.example.webservice.service
            </packageName>
              </configuration>
            </execution>
         </executions>
       </plugin>
   </plugins>
 </build>

Save and close the file. Note that you are directing Maven to obtain the WSDL description via the URL we used earlier to verify that the web service was correctly deployed.

Create a simple web service client:

cd to the groupId-defined package under the source branch (./{artifactId}/src/main/java/...) and create a client directory; cd into this directory.
Create a Java class called TutorialWebServiceClient.java.
Create a main method which gets a reference to the tutorial web service and exercises its two operations. For example:


package com.example.webservice.client;

import com.example.webservice.service.Tutorial;
import com.example.webservice.service.TutorialService;

public class TutorialWebServiceClient
{
  public static void main(String[] args)
  {
    TutorialService service = new TutorialService();
    Tutorial port = service.getTutorialPort();
    System.out.println("Calling getHelloWorld(); returns '" + port.getHelloWorld() + "'");
    System.out.println("Calling getPersonalizedHelloWorld(\"Bob\"); returns '" +
                       port.getPersonalizedHelloWorld("Bob") + "'");
  }
}

Note that the wsimport utility will create an interface named TutorialService.java and a class named Tutorial.java. It will do this before the client is compiled, but if you want to see the stub classes before you build the client, just execute mvn install before creating the client.

The web service interface name is obtained by prepending the package name from the POM to the name of the web service portType from the WSDL description, and the name of the service itself is the serviceName from the WSDL description, if you want to reference the classes before they are generated by wsimport.

Also note that the endpoint URL of the web service was embedded into the generated client-side classes by the setup in the Maven POM file, as I mention above.

From the project directory, enter

mvn install

This will generate the client-side stubs you need in your client, followed by the client itself. You can safely ignore the "[INFO] Nothing to do, no WSDL found!" log message; it is misleading and if you get a "BUILD SUCCESSFUL" message, then all is well.

Test the client and the web service with:

java -cp target/tutorialWebServiceClient.jar com.example.webservice.client.TutorialWebServiceClient

You should see output like the following:

Calling getHelloWorld(); returns 'Hello, World!'

Calling getPersonalizedHelloWorld("Bob"); returns 'Hello, Bob!'

You now have a working web-service/client pair, which will come in handy in future entries as we explore Flex web-service clients, BPEL and JBI.

Introductions

2009-03-24T09:48:00.000-07:00

Hello, everyone!

Before I start posting, I should tell you a little about where I'm going.

My main software interest is enterprise Java, and more specifically business process modeling, either on top of Java or a slightly higher level of abstraction. So I'm really interested in Web Services, Business Process Execution Language (BPEL), Java Business Integration (JBI), and (at this time, to a lesser extent) the Service Component Architecture (SCA).

I'm much more practical than theoretical. I like to get things working first and ruminate later. I like tutorials! I like finding good ones, and I like creating them, especially when it seems there's a hole in the tutorial offerings. Most of what you see posted here will be in the form of tutorials. Some of them will be really simple, as they will serve as building blocks for subsequent posts that you'll hopefully find more interesting. For example, sometimes it's handy to have a 25-line Java-SE-based web service, or an even briefer web service client, to test an application in development (in fact, this will be the topic of my next post).

With that, I hope you enjoy what's forthcoming and that you also find it useful! Ciao!

Wayne