HP (Hewlett-Packard) 5992-1918 manual

Ja v a™ T r ou bleshoo ting Guide f or HP -UX S y st ems HP P art Number: 5 99 2 - 1 9 1 8 P ublished: July 200 7 E dition: 3[...]

© Copyright 2007 Hewlett-Packard Dev elopment Company Confidential computer softw are. V alid license from HP required for possession, use or copying. Consistent with F AR 12.211 and 12.212, Commercial Computer Softw are, Computer Softw are Documentation, and T echnical Data for Commercial Items are licensed to the U.S. Gov ernment under vendor&ap[...]

T able of C ont ents Abou t T his Doc ument ....................................................................................................... 11 1 Diagno stic and Mo nitor ing T ools and Optio ns ........................................................ 13 1 .1 H P- UX J av a T oo ls a nd O pt io ns T ab le s...................................[...]

1 .2 7. 1 Ot he r He ap Du mp O pt io ns ....................................................................................................48 1 .2 7. 2 -X X: +H e ap Du mp On Ct rl Br e ak ..............................................................................................4 8 1 .2 7. 3 -X X: +H ea pD u mp On Ou tO fM em or y Er ro r ...[...]

Glos sar y ............................................................................................................................ 75 Inde x ................................................................................................................................. 77 T able of Contents 5[...]

6[...]

L ist o f F i gur es 1 -1 H Pj co nf ig - S ys te m T ab ..................................................................................................................22 1 -2 H Pj co nf ig - A pp li c at io n T a b..........................................................................................................2 2 1 -3 H Pj co nf ig - P[...]

8[...]

L ist o f T able s 1 -1 T o ol s an d Op t io ns f or C ra sh A na l ys is ...........................................................................................13 1-2 T ools and Options for Debugging Hung and Deadlocked Processes.. . .. .. . .. .. . .. .. .. . .. .. . .. .. . .. .. . .. .. . .. .14 1 -3 O pt io ns f or F at al E r ro r Ha nd [...]

10[...]

Abo ut T his Doc ument The information in this document will help application developers and support engineers debug their Jav a applications on HP-UX systems. Inte nded Audi ence This document is intended for application developers and support engineers who are debugging Jav a applications on HP-UX systems. Note that some features described in thi[...]

{} The contents are required in syntax. If the contents are a list separated by |, you must choose one of the items. ... The previous element can be repeated an arbitrary number of times.  Indicates the continuation of a code example. | Separates items in a list of choices. W ARNING A w arning calls attention to important information that if not[...]

1 Di agnos tic and Monit or ing T ools and Optio ns This chapter describes the tools and options available for postmortem diagnostics, analysis of hung/deadlocked processes, monitoring memory usage, and performance monitoring. The tools and options are listed in tables by their respectiv e functions in the first section of this chapter . Many of th[...]

T able 1 - 2 T ools and Options for Debugging Hung and Deadlocked Pr ocesses Description and Usage T ool or Option An HP-supported implementation of the gdb debugger that has Jav a support. For simplicity , this document refers to wdb / gdb as gdb from this point forward. gdb can be used to attach to a running process. wdb / gdb Used to identify an[...]

T able 1 - 4 T ools and Options for Monitoring Memory Use (continued) Description and Usage T ool Used to enable logging of garbage collection information. The HP-only -Xverbosegc option generates additional GC information that is used by HPjtune . It is preferable to use -Xverbosegc instead of -verbose:gc . -Xverbosegc (HP only) and -v erbose:gc T[...]

1 . 1 .7 JDK T ools Not A v ailable on HP -UX Some JDK tools are not av ailable on HP-UX, so they are not described in this document. They are provided in JavaSoft JDK as unsupported tools. Equiv alent functionality is available via gdb Jav a support, HPjmeter , and the HeapDump options. T able 1 - 7 JDK T ools Not Av ailable on HP -UX Description [...]

at java.lang.Thread.run(Thread.java:595) "Low Memory Detector" daemon prio=10 tid=00778b80 nid=19 lwp_id=2669774 runnable [00000000..00000000] "CompilerThread1" daemon prio=10 tid=00772c30 nid=17 lwp_id=2669772 waiting on condition [00000000..0a7ff728] "CompilerThread0" daemon prio=10 tid=007703f0 nid=16 lwp_id=2669771[...]

1 .4 gcore The gcore command creates a core image of a running process. By default, the name of the core file for a process-id will be core. pr ocess-id . The process information in the core image can be obtained by using gdb or other debuggers. When gcore creates a core image of each specified process, the process is temporarily stopped. When the [...]

Additionally , this is how you set the environment v ariable on an Integrity machine for a 32–bit Jav a application: export GDB_JAVA_UNWINDLIB=/opt/java1.4/jre/lib/IA64N/server/libjunwind.so If the SDK is installed in a location other than the default, substitute the non-default location for /opt/java1.4 in the previous commands. 1 . 5 . 1 Jav a [...]

Type "help java" followed by java subcommand name for full documentation. Command name abbreviations are allowed if unambiguous. The following two tables list Java VM debugging commands and Jav a subcommands: T able 1 -9 Java VM Debugging Commands Print backtrace of mixed Java and nativ e frames backtrace Print Jav a frame specific inform[...]

If the Jav a and system libraries used by the failed application reside in non-standard locations, then the GDB_SHLIB_PATH environment v ariable must be set to specify the location of the libraries. The following example illustrate how to invoke gdb on a hung process: • Determine the process id: $ ps -u user1 | grep java 23989 pts/9 8:52 java •[...]

HPjconfig [ options ] <object> <action> objects: -patches &| -tunables actions: -listreq | -listmis | -listpres | -apply options: -patches operate on java-specific patches -tunables operate on java-specific tunables -listreq list all java required patches or tunables that are applicable to this system -listmis list missing java-spec[...]

Figur e 1 -3 HPjconfig - P atches T ab Figur e 1 -4 HPjconfig - T unables T ab Following are the commands for invoking HPjconfig in non-GUI mode. The -help option lists options you can use in this mode. $ cd <hpjconfig_installation_dir> $ java -jar ./HPjconfig.jar -nogui -help Following is an example using HPjconfig in non-GUI mode to list mi[...]

maxfiles 2*1024 maxfiles_lim 2*1024 maxdsiz 2000*1024*1024 Following is an example of using HPjconfig to display tunables that are set to values less than those recommended: $ java -jar HPjconfig.jar -nogui -tunables -listmis Log written to HPjconfig_mutant_20060915_040955.log List of tunables whose values are less than the recommended values: Name[...]

T able 1 - 1 1 HPjmeter 3. 0 Featur es Drill down into application profile metrics • Graphic display of profiling data • Call graphs with call count, or with CPU or clock time • Per thread display of time spent • Per thread or per process display Integrated HPjtune functions with concurrent improvements in tool and help usability Ability to[...]

1 .7 . 1 S tati c D ata A naly sis 1 .7 . 1 . 1 Using HPjmeter to A nalyz e Pr ofiling Data The following steps summarize how to use HPjmeter to save and view profiling information from your applications. 1. Change the command line of your Jav a application to use -Xeprof , -agentlib:hprof , or -Xrunhprof options to capture profiling data. For exam[...]

Figur e 1 -5 HPjmeter - Pro file Data 6. Click among the tabs to view av ailable metrics. Use the Metrics or Estimate menus to select additional metrics to view . Each metric you select opens in a new tab. Mousing ov er each category in the cascading menu will rev eal the relev ant metrics for that category . The following screen shows the av ailab[...]

1 .7 . 1 .2 Using HPjmeter t o Anal yz e G ar bage Collecti on Data The following steps summarize how to use HPjmeter to save and view garbage collection information from your applications: 1. Change the command line of your Jav a application to use -Xverbosegc or -Xloggc to capture garbage collection data. 2. Run the application to create a data f[...]

(csh) setenv SHLIB_PATH /opt/hpjmeter/lib/IA64N (ksh) export SHLIB_PATH=/opt/hpjmeter/lib/IA64N T o select the Integrity 64-bit library: (csh) setenv SHLIB_PATH /opt/hpjmeter/lib/IA64W (ksh) export SHLIB_PATH=/opt/hpjmeter/lib/IA64W 2. Confirm that the node agent is running. With a standard installation, the node agent should be running as a daemon[...]

Figur e 1 -8 HPjmeter - Connecting to Ser ver 2. In the Connect to Serv er dialog box, type the host name where the Jav a application and corresponding node agent are running. 3. If the node agent w as started on a nonstandard port, specify the port number in the Optional P ort box. 4. Select Connect . The running Jav a VM for each application shou[...]

Figur e 1 -9 HPjmeter - Setting Session Pref erences 2. Check the default settings for metrics, filters, and alerts, and enable the settings you w ant to activ ate. 3. Click OK. The Session Preferences window will close and the newly Open Session will be visible, marked by the icon. Refer to the following screen for an example: Figur e 1 - 1 0 HPjm[...]

1 .7 .2 .4 V ie w ing Monitor ing Metr ics Dur ing Y our Open S essi on 1. Click the open session or time slice to highlight the data to be viewed. 2. Use the Monitor menu on the console main window to select the desired metrics. Refer to the following screen for an example: Figur e 1 - 1 1 HPjmeter - Choosing M etr ics to Monitor 3. Select a metri[...]

1 .7 .2 . 5. 1 Sample Memo ry Leak Appli cation This application demonstrates how memory leak alerts work in HPjmeter . It uses a simple program which allocates some objects. The program uses a java.util.Vector object to retain references to some of the objects. This application is configured to leak memory at the rate of about 10 MB per hour . It [...]

Figur e 1 - 1 3 HPjmeter - H eap Monitor Displa y 1 .7 .2 . 5.2 Sample T hread Deadloc k Appli cation This application demonstrates how HPjmeter detects deadlocked threads. It creates pairs of threads every 30 seconds, stopping at 50 threads, which synchronize w ork using shared locks. Occasionally , the program reverses the order on which locks ar[...]

Figur e 1 - 1 4 HPjmeter - Thread Hist ogram 1 .8 HPjtune NO TE: The HPjtune product has reached end of life. HP has integrated HPjtune functionality into HPjmeter 3.0 and recommends migrating to HPjmeter for the latest in bug fixes, enhancements, and support. HPjtune is a garbage collection visualization tool for analyzing garbage collection activ[...]

Following is an example screen shot to illustrate HPjtune 's output: Figur e 1 - 1 5 HPjtune Sc reen 1 .9 hat NO TE: Beginning with JDK 6.0, hat is replaced with jhat . For information on jhat , refer to the jhat section in this document. The hat tool is a third-party tool that can be used for heap analysis. It starts a web server on a binary-[...]

$ java -agentlib:hprof[=options] appl_to_profile (JDK 1.5+) $ java -Xrunhprof[:options] appl_to_profile (SDK 1.4.2.0+) hprof supports a number of profiling options. Use java -Xrunhprof:help to display the av ailable options. Following is an example hprof command to capture object data: $ java -Xrunhprof:heap=dump Hello Load the resulting text file [...]

environment v ariable may be useful to add options to the command line when the application is run. This environment v ariable is primarily intended to support the initialization of tools, specifically the launching of native or Jav a agents using the -agentlib or -javaagent options. The JAVA_TOOL_OPTIONS environment v ariable is processed at the t[...]

$ /opt/java1.5/bin/jconsole 13028 The following figure shows a jconsole screen shot: Figur e 1 - 1 6 jconsole Screen jconsole can also be run remotely . T o learn more about jconsole , including remote invocation, refer to the following website: http://jav a.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html 1 . 1 4 jdb The SDK includes a comma[...]

http://jav a.sun.com/javase/6/docs/technotes/tools/share/jhat.html 1 . 1 6 jps ( 1 .5+ o nl y) The jps tool lists the Jav a VMs on the target system. The tool is limited to reporting information on Jav a VMs that the user has access rights to, as determined by HP-UX specific access control mechanisms. For example, if a non-root user executes the jp[...]

T able 1 - 1 3 Options to the jstat Command (continued) Prints statistics of the sizes of the old generation -gcoldcapacity Prints statistics of the sizes of the permanent generation -gcpermcapacity Prints a summary of garbage collection statistics -gcutil Prints Java compilation method statistics -printcompilation A complete description of the jst[...]

As of JDK 1.5, the following subset of jvmstat tools is included with the JDK: jps (formerly jvmps ) , jstat (formerly jvmstat ), and jstatd (formerly perfagent ). The visualgc tool is not included with JDK 1.5+, but is instead provided in the unbundled jvmstat 3.0 distribution. For more details, refer to the following website: http://jav a.sun.com[...]

1. Application Information window 2. Graph window 3. Survivor Age Histogram window (optional) The Survivor Age Histogram window is only av ailable when P arallel Scavenge is in use ( -XX:+UseParallelGC or -XX:+AggressiveHeap options). Following is an example visualgc Application Information window and a description of the different window areas: Fi[...]

where each sample occupies two pixels of screen area. The height of each display depends on the metric being plotted. Following is an example Graph window: Figur e 1 - 1 8 visualgc Graph Windo w Each of the GC space graphs can be displayed in one of tw o modes: reserved mode or committed mode; committed mode is the default. In reserved mode, the da[...]

Figur e 1 - 1 9 visualgc Surviv or Ag e Hist ogram Windo w When the Jav a VM is started with the Parallel Y oung GC option ( -XX:+UseParallelGC ), the Survivor Age Histogram window is not displayed because the P arallel Y oung collector does not maintain a survivor age histogram since it applies a different policy for maintaining objects in the sur[...]

In general, all errors detected by -Xcheck:jni are fatal; the error is printed and the Java VM is aborted. One exception to this is a non-fatal w arning that is printed when a JNI call is made within a JNI critical region. This is the w arning that is displayed when this happens: Warning: Calling other JNI functions in the scope of Get/ReleasePrimi[...]

T able 1 - 1 5 Garbage Collec tion F ield Information (continued) Information in F ield Field Eden Sub-space (within the New Generation) occupied after GC. 8 Eden Sub-space (within the New Generation) current capacity . 9 Survivor Sub-space (within the New Generation) occupied before GC. 10 Survivor Sub-space (within the New Generation) occupied af[...]

1 .2 7 -XX:+HeapDump and _JAVA_HEAPDUMP En v ir onment V ar ia ble The -XX:+HeapDump option can be used to observe memory allocation in a running Jav a application by taking snapshots of the heap over time. Another w ay to get heap dumps is to use the _JAVA_HEAPDUMP environment v ariable; setting this environment variable allows memory snapshots to[...]

example, the following file names are created: java_27298.hprof.1152743593943 and java_27298_060712_153313_heapDump.hprof.txt . If JAVA_BINARY_HEAPDUMP is set and the -Xrunhprof:heap=dump command is giv en, then both hprof ASCII and binary files are produced for this option. 1 .2 7 . 3 -XX:+HeapDumpOnOutOfMemoryError The -XX:+HeapDumpOnOutOfMemoryE[...]

where <string> is a single command or a list of commands each separated by a semicolon. W ithin <string> all occurrences of “%p” are replaced with the current process id (pid), and all occurrences of “%%” are replaced by a single “%”. Following is an example showing how the fatal error report can be mailed to a support alias[...]

2 Us ef ul S y st em T ools f or Jav a T r ouble shooting This chapter contains information about some system tools available on HP-UX that are useful when troubleshooting Jav a application problems. The tools discussed include: GlancePlus , tusc , Prospect , HP Caliper , sar , vmstat , iostat , sw apinfo , top , netstat , and others. 2 . 1 GlanceP[...]

2 .8 swapinfo The swapinfo command displays information about device and file system paging space. For more information on this command, refer to the following w ebsite: http://docs.hp.com/en/B2355-60127/sw apinfo.1M.html 2 .9 top The top command displays the top processes on the system, periodically updating the information; raw CPU percentage is [...]

3 Getting H elp f r om H e wle t t -P ack ar d Sometimes you need help troubleshooting y our Jav a application problems. Before opening a support call, search for information that may help y ou by referring to the Go Jav a! website: http://www .hp.com/go/jav a This site contains much information about Java, including known issues, release notes, pa[...]

a. What is the version of the Jav a VM that is having the problem? Run the command java -version to retrieve this information. b. What are the v alues of the environment variables used by Jav a? c. What libraries are being loaded? This information is best collected with gdb 's packcore command. 5. Contact Information a. Contact name b. Company[...]

memory(kbytes) unlimited coredump(blocks) 4194303 If coredump is not set to unlimited, set it to unlimited using the ulimit -c command: $ ulimit -c unlimited $ ulimit -a time(seconds) unlimited file(blocks) unlimited data(kbytes) 4292870144 stack(kbytes) 8192 memory(kbytes) unlimited coredump(blocks) unlimited 3 .2 . 1 . 1 .3 V erify Amo unt of Dis[...]

3 .2 . 1 . 1 .5 Ensure P ermis sions Allo w Cor e F iles Some Jav a processes run setuid; that is, a process where the effective uid or gid differs from the real uid or gid. On HP–UX 11.11 and later v ersions a kernel security feature prev ents core file creation for these processes. Use the following command when y ou are logged in as the root u[...]

early Jav a versions generate less information in the fatal error log). Following is a summary of the type of information contained in this file: 1. The error causing the Jav a VM to abort, including the pc, process id, and thread id at which the error occurred. For example: # An unexpected error has been detected by HotSpot Virtual Machine: # # SI[...]

3 . 3 Co llecting S y stem Inf or mation Along with HP-UX version information and information about which window manager is being used, it is also useful to know which patches are installed on the system. This information can be gathered either with swlist or HPjconfig . Following is an example using the swlist command to retrieve this list: $ /usr[...]

runtime environment v ariable values, run the following command under the same environment (that is, the same user) that the Jav a application was executed: (ksh)$ env > app_environment.txt (csh)$ getenv > app_environment.txt Include the app_environment.txt file when you send in your collected data files to Hewlett-P ackard. 3 .4.2 L ibrar ie[...]

T able 3- 1 Libjunwind Libr ary Location for P A-RISC Sy stems libjunwind Location Application T ype /opt/<java_vers>/jre/lib/PA_RISC/server/libjunwind.sl P A1.1 applications ( java -pa11 ) /opt/<java_vers>/jre/lib/PA_RISC2.0/server/libjunwind.sl P A2.0 32–bit applications (default P A-RISC) /opt/<java_vers>/jre/lib/PA_RISC2.0W/[...]

4 C or e F ile Anal y sis The previous chapter described how to collect necessary information before opening a call to HP Support to get help troubleshooting Jav a applications. Sometimes it is possible to at least attempt the core file analysis on your own. This chapter w alks through an example core file analysis step by step. By studying this ex[...]

/usr/bin/echo "Create header file" javah -verbose -jni StackTrace # compile jni code /usr/bin/echo "Compile c code" /usr/bin/cc +z -c -I $JAVA_HOME/include -I $JAVA_HOME/include/hp-ux stacktrace.c # create shared library /usr/bin/echo "Create shared library" /usr/bin/ld -b -o libstacktrace.sl stacktrace.o /usr/bin/ec[...]

System.out.println("Call dumpCore to convert " + ci + " to a binary string!"); System.out.println("The binary String: " + StackTrace.dumpCore(ci)); } // end methodMakeCall //************************************************** public static void main(String args[]) { int convertInt; System.out.println(); if(args.length =[...]

"(I)Ljava/lang/String;"); (*env)->ExceptionDescribe(env); (*env)->ExceptionClear(env); if(methodid == NULL) { return (*env)->NewStringUTF(env, "JNI GetStaticMethodID failed!"); } return (*env)->CallStaticObjectMethod(env, classid, methodid, intarg); } 4.2 Building the A pplicati on The StackTraceJob script can be use[...]

• libstacktrace.sl —the runtime library • stacktrace.o —the object file 4. 3 V er ify Cor e F ile Before you proceed further , verify that the core file, core , is complete and v alid. Y ou can do this in two steps. First, open the file in gdb and check the error and w arning messages. $ gdb /opt/java1.4/bin/PA_RISC2.0/java core HP gdb 5.5.[...]

• All source files ( .java , .h , .c ). • All .class files. • The command line used to run the application program. In this example, the StackTraceJob script. Now , use gdb to pack the core file. $ gdb /opt/java1.4/bin/PA_RISC2.0/java core HP gdb 5.5.7 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00 and target hppa1.1-hp-hpux11.00. Copyright 198[...]

-rw-rw-r-- 1 user1 lang 12323556 Mar 30 14:37 packcore.tar.Z $ Uncompress and extract the files from the packcore.tar.Z file: $ uncompress packcore.tar.Z $ tar -xvf packcore.tar packcore/ packcore/modules.tar packcore/progname.txt packcore/core Delete the packcore.tar file since you hav e extracted the files. Next, move the modules.tar and core fil[...]

4.7 Ex ample gdb Ses sion Before beginning core file analysis, examine the fatal error log file, hs_err_pid<pid>.log . This file contains useful information that will help y ou troubleshoot the problem. For more information about the contents of the hs_err_pid<pid>.log file, refer to Section 3.2.2., Collecting Fatal Error Log Informatio[...]

Program terminated with signal 6, Aborted. #0 0xc0214db0 in kill+0x10 () from ./libc.2 The first step is to look at the stack trace of the failing thread. Do this by issuing gdb 's backtrace command. Following is the backtrace which has been annotated with the comment “F AILED HERE” at the point of failure: (gdb) backtrace #0 0xc0214db0 in[...]

Y ou w ant to display 75 instructions from the beginning of the get_method_id() function to the point of failure for frame 9. Since this is a substantial number of instructions, redirect the output to a file: (gdb) set redirect-file frame9instrs (gdb) set redirect on Redirecting output to frame9instrs. (gdb) x /75i 0xc3ed2da0 (gdb) set redirect off[...]

0xc3ed2e38 <get_method_id()+0x98>: ldw,o 0(%r20),%r31 0xc3ed2e3c <get_method_id()+0x9c>: ldb -0xf1(%sp),%r7 0xc3ed2e40 <get_method_id()+0xa0>: extrw,u %r31,28,1,%r23 0xc3ed2e44 <get_method_id()+0xa4>: cmpb,od,n %r7,%r23,0xc3ed30dc <get_method_id()+0x33c> 0xc3ed2e48 <get_method_id()+0xa8>: call 0xc400c248 <meth[...]

(gdb) x /61i 0xc3ed33e0 (gdb) set redirect off Following is the annotated listing of the redirected output file, frame10instrs . Note that the parameters to the jni_GetStaticMethodID() function have been remov ed to simplify the listing. The parameters to this function are: (JNIEnv_ *, _jclass *, char const *, char const *) 0xc3ed33e0 <jni_GetSt[...]

T race through the instructions to see where R25 was loaded with a value. The first place this happens is at offset 0xcc: 0xc3ed34ac <jni_GetStaticMethodID()+0xcc>: ldw -0xb0(%sp),%r25 In this instruction, R25 is loaded with the value at sp-0xb0. Use gdb to examine the memory at sp-0xb0, displaying the output as an address: (gdb) x /2x $sp-0x[...]

74[...]

Glos sary GC Garbage collection. gid Group id. HotSpot VM The JDK comes with a virtual machine implementation called the Jav a HotSpot VM. Java VM On HP implementations this is the same as the HotSpot VM. JDK The Jav a Developer's Kit is the set of Jav a development tools consisting of the API classes, a Jav a compiler , and the Java virtual m[...]

76[...]

Inde x S ymbo ls -verbose:class , 42 -verbose:gc , 42 -verbose:jni , 42 -Xcheck:jni , 45 -Xverbosegc , 46 -XX:+HeapDump , 48 -XX:+HeapDumpOnCtrlBreak , 48 -XX:+HeapDumpOnly , 49 -XX:+HeapDumpOnOutOfMemoryError , 49 -XX:+ShowMessageBoxOnError , 50 -XX:ErrorFile , 47 -XX:OnError , 49 _JAVA_HEAPDUMP environment v ariable, 48 C core file checklist, 54 [...]

problem report checklist, 53 Prospect, 51 S sar , 51 Serviceability Agent, 16 stack trace information, 57 swapinfo , 52 system information, 58 system tools, 51 T top , 52 tusc , 51 V visualgc , 42 vmstat , 51 78 Index[...]