For nearly two years, I've been trying to branch out and add another programming language to my brain.  I read and blogged about Seven Languages in Seven Weeks, by Brian Tate, an excellent book that I blasted through in seven days to save a little time.
I've been experimenting with using Pig on some Fannie-Mae MBS data lately.  While I don't mind writing MapReduce programs to process data (especially the fairly simple tasks I'm doing now), I really do appreciate the "magic" Pig does under the blanket, you might say.
I've recently been writing JMS clients for an application I'm building and keep finding myself having to re-learn some basic configuration.
A few years ago, I posted a how-to on Java-SE-based Web Services. More recently, I've become interested in asynchronous web-service invocation, and, as it turns out, Java SE supports that, too. This post, then, is the asynchronous version of that older post.
In an earlier post, we stepped through the building of an asynchronous web service, deployed in Java SE. I saved my comments for this post to keep things a little cleaner.
As I have mentioned in earlier posts, I am using the Java Debug Interface (JDI) to create a Java process-monitoring tool.
In my Part-1 post on this topic, we actually did all the I/O I'm going to do here. We lazily read in the entire sample data file, a file containing data describing events generated by a process monitor. My next goal was to re-hydrate my Events from the Strings serialized to the file.
I'm about halfway through Real World Haskell, and I've spent a week trying to decide when to write this post. As the authors point out, Haskell I/O is easy to work with.
For the last few weeks, I have been building a Java process monitoring tool based on the Java Debug Interface. Although I've done much of this work before, it has been a few years, and so now I'm retracing my steps.
After my last post scrolled off the bottom of the page, I realized I missed a couple of opportunities: one related to some additional code optimization, and one related to the topic of lazy (or nonstrict) evaluation.

First, let me review what I was doing.
Today I'm going to process a set of structured data using Haskell, tainted by years of Smalltalk, C++, Java and C# experience.
I've been working on a JDI (Java Debug Interface) project lately and have been posting helpful tips as I go along. It has been a few years since I've worked with this API, but although I know there have been a few enhancements, the API is quite consistent with what I remember.
Today I'm looking at Haskell type definition and the use of pattern-matching in functions. Pattern-matching is much more an integral feature of FP, as opposed to OO. But first...
I'm learning Haskell by following O'Sullivan, Goerzen and Stewart's Real World Haskell. I've been writing object-oriented code for well over half my career as a developer, and there are things about functional programming that really stand out to me specifically because of my OO background.
For the last year or so, I've been trying to come up to speed on functional programming, studying bits and pieces here and there. One interesting source was Bruce Tate's Seven Languages in Seven Weeks, which included a number of FP languages.
If you've looked at my recent posts, you know I'm working on a plugin for VisualVM, a very useful tool supplied with the JDK. In one example, I showed how to attach to a waiting Java application using a socket-based AttachingConnector.
Say you've got a good-sized chunk of code, in production, that doesn't always act as expected but it does so often enough that everyone's willing to keep using it (including your customers).

Say you've got a good-sized chunk of code, in production, that doesn't always act as expected but it does so often enough that everyone's willing to keep using it (including your customers). You have a lot on your to-do list, and you keep busy enough just handling the serious meltdowns, so much so that you don't really have time to investigate that occasional failed parse-and-load, or that mysterious stack trace that's supposed to be harmless. Besides: your application does so much; statistically, it can't get everything right all the time, can it?

For an issue that isn't easily repeatable but is considered to be a significant problem, running in the debugger can be demoralizing. What if you could produce the equivalent of a "robo-debugger", a process that would run the debugger for you, continuously, and wait with infinite patience for that rare occurrence? And then have the common sense to collect information off the stack and -- gasp -- even tell you about it? If this does not sound revolutionary, then good for you. Why do humans ever sit in front of a monitor, stepping through a debugger manually anyway? We've "manualized" an operation that should be automated.

Of course, logging could do the same thing for you. My interest in this idea arose when I was supporting an application for which I had the source code, but I was not permitted to modify it. I was allowed to recompile the application with the debug switch on, however, and I was allowed to attach with a debugger. After I wrote a JDI-based monitor for this application, I realized it had one additional advantage -- you don't have to add a lot of logging statements for an issue that might only need to be debugged once. Also note that code like this could be embedded into another application (for example, it could be a VisualVM extension) and be used to generate events on demand, another reason to skip the embedded logging statements.

Here's the general approach:
  1. Ensure your target application is compiled with the -g switch.

  2. Start the targeted application as usual, but listening on a port for a debugger connection.

  3. Start your robo-debugger and attach to the target JVM.

  4. Read a list of breakpoint specifications, each of which contains the following information:

    • Class name and source line number.

    • List of variables on the stack that you want to inspect.

    • Optional message in the form of a formatted String with placeholders for said variables retrieved from the stack.

    • Optional list of key-value pairs, values again being retrieved from the stack.


  5. At each defined breakpoint, halt execution (briefly!) and generate an event, realized as a log (to file, JDBC, etc) message, JMS message, etc.

  6. Mine the event stream from your application to solve all the issues that have been nagging you since you went to production.

This post will cover everything except the last item, which is the hard part. I also won't write logging or JMS code, as that's not relevant to the discussion. My example will generate some output to stdout.

To get started, pick a target application. I'll be using an application I wrote called "JarView" (just a simple Swing application to search through a directory of .jar files to find a missing class file).

Start the target application

There are two primary transports in JPDA (Java Platform Debugger Architecture): socket-based, and shared-memory-based. I'll start my application using socket-based JPDA and (transport=dt_socket), instruct it to wait for a debugger to attach to it (server=y) and do not suspend while waiting for a connection (suspend=n):
     c:\JarView>java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n -cp jarview.jar JarView
You'll see a launch message like the following:
     Listening for transport dt_socket at address: 50069
Attach to the target

Write a program to
  1. Use the JDI Bootstrap class to get an instance of a VirtualMachineManager.

  2. Iterate over the VirtualMachineManager's list of AttachingConnectors until you find a connector supporting transport dt_socket.

  3. Get the port Connector.Argument of the AttachingConnector and set it to the port on which your target application is listening.

  4. Attach to the AttachingConnector and get an instance of a VirtualMachine.
An example piece of code (bare minimum, with no special exception handling) that will perform the above steps follows. You will need to compile and run with the JDK's lib/tools.jar on the classpath (this is not found in the JRE, by the way).
import java.util.List;
import java.util.Map;
import com.sun.jdi.Bootstrap;
import com.sun.jdi.VirtualMachine;
import com.sun.jdi.VirtualMachineManager;
import com.sun.jdi.connect.AttachingConnector;
import com.sun.jdi.connect.Connector;

public class JDIDemo
{
public static void main(String[] args) throws Exception
{
VirtualMachineManager vmMgr = Bootstrap.virtualMachineManager();
AttachingConnector socketConnector = null;
List attachingConnectors = vmMgr.attachingConnectors();
for (AttachingConnector ac: attachingConnectors)
{
if (ac.transport().name().equals("dt_socket"))
{
socketConnector = ac;
break;
}
}
if (socketConnector != null)
{
Map paramsMap = socketConnector.defaultArguments();
Connector.IntegerArgument portArg = (Connector.IntegerArgument)paramsMap.get("port");
portArg.setValue(Integer.parseInt(args[0]));
VirtualMachine vm = socketConnector.attach(paramsMap);
System.out.println("Attached to process '" + vm.name() + "'");
}
}
}
It is a lot easier to get a Connector.Argument from an existing data structure (as above) than it is to create one from scratch. Also note, there are very few (if any) constructors in this API; just about every reference you get is retrieved ultimately by going through the Bootstrap class and working your way into the API. In my example, there were 3 AttachingConnectors, representing transports dt_socket, dt_shmem, and local. When I run the above example, I see the following output:
    Attached to process 'Java HotSpot(TM) 64-Bit Server VM'
Note that when this program exits, the target VM changes the port on which it is listening, something you should remember if you run again. I don't remember this behavior on Java 5, but it has been a while since I've written a JDI application.

Pause at a breakpoint and generate an event

To conclude this post in a reasonable length, I will just pick a line of code in my target that I know well and give some example code that will pick a variable off the stack and output it to stdout. The details of logging or sending a JMS message aren't really relevant to this topic.

For this example, I want to break at line 863, where I'm about to add the name of a file to my Swing table. This is a file whose name at least partially matches an input class name. Below is a segment of the source:
849:    if (fullName.lastIndexOf("/") > -1)
850: {
851: directoryName = fullName.substring(0, fullName.lastIndexOf("/"));
852: fileName = fullName.substring(directoryName.length()+1, fullName.length());
853: }
854: else
855: {
856: fileName = fullName;
857: }
858: if (fileName.indexOf(searchForTextField.getText()) > -1)
859: {
860: Vector nextRow = new Vector();
861: nextRow.add(archive.getAbsolutePath());
862: nextRow.add(fileName);
863: rowData.add(nextRow);
864: }

I'd like to print a short message at line 863 which outputs the value of fileName.

How do you specify a breakpoint in JDI? You have to know what you're asking for. Normally you would look for a class, maybe a method, and a line number. My target application is a Swing application with a lot of anonymous inner classes, so rather than figure out which one is the one I want, I'm just going to search on line number. You probably want to call a constructor to create a breakpoint for a line number, but there is no constructor; you'll have to search through a lot of metadata and "find" the description of this line of code, then request a breakpoint using that description and a factory method in the EventRequestManager. To make a long story somewhat shorter:
  1. Get a list of all classes (as ReferenceTypes).

  2. For each class, get all line locations (Location).

  3. At line location corresponding to line 863, break out of the search loop.

  4. Get an instance of the EventRequestManager from the VirtualMachine.

  5. Create a BreakpointRequest in the EventRequestManager, using the Location object for line 863.

  6. Get the EventQueue instance from the VirtualMachine.

  7. Create a while(true) loop on the EventQueue, calling its remove() method.

  8. For each EventSet removed from the queue, process each Event.

  9. For each Event, check to see if it is a BreakpointEvent, and if the line number matches the breakpoint we're interested in, process the Event further.

  10. For a matching Event, get the top element of the StackFrame, get all visible variables on the StackFrame element, find the one whose name matches the variable you are looking for, and if so, dig through the API for the correct chain of method calls to extract its value.
This is probably easier shown with code. Below is an updated version of the first cut of the example code (note: please refactor out of main for a real application!):
import java.util.List;
import java.util.Map;
import com.sun.jdi.AbsentInformationException;
import com.sun.jdi.Bootstrap;
import com.sun.jdi.LocalVariable;
import com.sun.jdi.Location;
import com.sun.jdi.ReferenceType;
import com.sun.jdi.StackFrame;
import com.sun.jdi.StringReference;
import com.sun.jdi.ThreadReference;
import com.sun.jdi.Value;
import com.sun.jdi.VirtualMachine;
import com.sun.jdi.VirtualMachineManager;
import com.sun.jdi.connect.AttachingConnector;
import com.sun.jdi.connect.Connector;
import com.sun.jdi.event.BreakpointEvent;
import com.sun.jdi.event.Event;
import com.sun.jdi.event.EventIterator;
import com.sun.jdi.event.EventQueue;
import com.sun.jdi.event.EventSet;
import com.sun.jdi.request.BreakpointRequest;
import com.sun.jdi.request.EventRequest;
import com.sun.jdi.request.EventRequestManager;

public class JDIDemo
{
public static void main(String[] args) throws Exception
{
if (args.length != 3)
{
System.out.println("Usage: java JDIDemo debugPortNumber sourceLineNumber variableName");
System.exit(-1);
}
int debugPort = Integer.parseInt(args[0]);
int lineNumber = Integer.parseInt(args[1]);
String varName = args[2];

VirtualMachineManager vmMgr = Bootstrap.virtualMachineManager();
AttachingConnector socketConnector = null;
List attachingConnectors = vmMgr.attachingConnectors();
for (AttachingConnector ac: attachingConnectors)
{
if (ac.transport().name().equals("dt_socket"))
{
socketConnector = ac;
break;
}
}

if (socketConnector != null)
{
Map paramsMap = socketConnector.defaultArguments();
Connector.IntegerArgument portArg = (Connector.IntegerArgument)paramsMap.get("port");
portArg.setValue(debugPort);
VirtualMachine vm = socketConnector.attach(paramsMap);
System.out.println("Attached to process '" + vm.name() + "'");

List refTypes = vm.allClasses();
Location breakpointLocation = null;
for (ReferenceType refType: refTypes)
{
if (breakpointLocation != null)
{
break;
}
List locs = refType.allLineLocations();
for (Location loc: locs)
{
if (loc.lineNumber() == lineNumber)
{
breakpointLocation = loc;
break;
}
}
}

if (breakpointLocation != null)
{
EventRequestManager evtReqMgr = vm.eventRequestManager();
BreakpointRequest bReq = evtReqMgr.createBreakpointRequest(breakpointLocation);
bReq.setSuspendPolicy(BreakpointRequest.SUSPEND_ALL);
bReq.enable();
EventQueue evtQueue = vm.eventQueue();
while(true)
{
EventSet evtSet = evtQueue.remove();
EventIterator evtIter = evtSet.eventIterator();
while (evtIter.hasNext())
{
try
{
Event evt = evtIter.next();
EventRequest evtReq = evt.request();
if (evtReq instanceof BreakpointRequest)
{
BreakpointRequest bpReq = (BreakpointRequest)evtReq;
if (bpReq.location().lineNumber() == lineNumber)
{
System.out.println("Breakpoint at line " + lineNumber + ": ");
BreakpointEvent brEvt = (BreakpointEvent)evt;
ThreadReference threadRef = brEvt.thread();
StackFrame stackFrame = threadRef.frame(0);
List visVars = stackFrame.visibleVariables();
for (LocalVariable visibleVar: visVars)
{
if (visibleVar.name().equals(varName))
{
Value val = stackFrame.getValue(visibleVar);
if (val instanceof StringReference)
{
String varNameValue = ((StringReference)val).value();
System.out.println(varName + " = '" + varNameValue + "'");
}
}
}
}
}
}
catch (AbsentInformationException aie)
{
System.out.println("AbsentInformationException: did you compile your target application with -g option?");
}
catch (Exception exc)
{
System.out.println(exc.getClass().getName() + ": " + exc.getMessage());
}
finally
{
evtSet.resume();
}
}
}
}

}
}
}
When I run this application with a command line like:
     java -cp c:\jdk1.6.0_20\lib\tools.jar;. JDIDemo 56485 863 fileName
I get the following output:
    Attached to process 'Java HotSpot(TM) 64-Bit Server VM'
Breakpoint at line 863:
fileName = 'BreakpointEvent.class'
Breakpoint at line 863:
fileName = 'EventSetImpl$BreakpointEventImpl.class'
Pointers

You might have noticed the line above referencing the AbsentInformationException. You will get this if your target application has not been compiled with the debug (-g) switch. If you cannot compile the code with the debug switch, you will be able to set a breakpoint, but there won't be any information available on the stack when you get there.

Some JDI operations are more expensive than others. The last time I wrote a JDI application, I noticed that "method-entry" and "method-exit" breakpoints were enormously more expensive than simple line breakpoints. Now that I have a working example, I'll investigate these issues in a later post to see how things are in the current update of Java 6.
As I mentioned in an earlier post, I noticed recently that the JDK utility VisualVM is extensible, and it was my goal to create a useful extension.
I'm not sure when this happened, but at some point the JDK-included VisualVM utility became extensible. This is really great news for me. Writing a profiler is a lot of work (although admittedly very interesting and fun).
We use DWR (http://www.directwebremoting.org) a lot where I'm working now. It's a Java library used to integrate JavaScript-based web development with Java middleware.
I've reached the 7th and final language of Bruce Tate's Seven Languages in Seven Weeks. While some of the previous languages were functional with some imperative support, Haskell is a purely functional, and statically typed, language.
Today I'm reviewing the discussion of Clojure from Bruce Tate's Seven Languages in Seven Weeks. Clojure is Lisp on the Java virtual machine. Lisp is another language that, despite being around a long time, I have yet to investigate, so this is another new experience.
If you are just dropping in on me, I'm reviewing Bruce Tate's Seven Languages in Seven Weeks, with the slightly lazy (or aggressive, depending on your view) twist of reviewing one language per day.
Scala is the 4th language in Bruce Tate's Seven Languages in Seven Weeks. It is the only language in this book with which I am already familiar, although I've only been learning it for a month or so.
Prolog is the 3rd language covered in Bruce Tate's Seven Languages in Seven Weeks, and is a declarative, rather than imperative, language. Prolog is not new, of course (1972), but I have to admit this is the first time I've taken a look at it.

Like Bruce, I used GNU Prolog (1.3.1).
Io is the 2nd language in Bruce Tate's Seven Languages in Seven Weeks. Io is a prototyping language, where most of the mass exists in the libraries. The syntax itself is refreshingly simple, and he is not exaggerating to say you can grasp it in about 15 minutes.
When I decided to blow through Seven Languages in Seven Weeks in only 7 days, I had yet to read even the introduction to the book.
I recently started learning Scala (you know when the New York Times refers to Java as an "older" language, it's time to update!). As I've started trying to shift my thinking from object orientation to functional programming, I remembered the book Seven Languages in Seven Weeks by Bruce Tate.
Recently I had a repeat of a problem I was unable to solve the first time I encountered it.
Hello, everyone:

I've posted a number of entries in the last year about profiling Java applications. Some of this effort went in to a standalone Java profiler with a Swing interface, called the MonkeyWrench.
DTrace, a dynamic tracing framework on Solaris (see dtrace(1m)) is a valuable and extremely easy-to-use tool if you find yourself analyzing Java performance issues on Solaris.
My entries here are usually about Java, but today I want to call attention to the 1.0 release of FlexMonkey, an Adobe AIR app for testing Flex and Adobe AIR applications (I work for Gorilla Logic, the creator of FlexMonkey).
In recent posts, I've investigated using the java.lang.instrument package to instrument Java classes for a simple profiler.
I've been building a Java profiler lately, partly to address what I think are shortcomings in the current set of free/shareware profilers, and also to enjoy the flexibility of obtaining only the profiling information I need.
Lately I've been looking at profilers and wondering how easy it would be to write one, mostly to be able to provide a little more sophisticated guidance to performance investigations.
About Me
About Me
My Photo
I'm a software architect/consultant in Boulder, Colorado.
Picture
Picture
Blog Archive
Loading
Dynamic Views theme. Powered by Blogger. Report Abuse.