This representation below is similar to the graph (hddoc.png) in Definitions & Algorithms. Notice the indentation corresponds to depth in the tree and the left-hand column represents the depth of each object. That is <0> for roots K and A.
<0> [48,000] 0x200000b0 [40,000] K <1> [8,000] 0x200000c0 [8,000] L <0> [26,600] 0x50001010 [6,000] A <1> [20,600] 0x50002020 [4,000] B <2> [13,672] 0x50004050 [1,104] C <3> [12,568] 0x50005060 [1,032] D <4> [11,536] 0x50003090 [504] E <5> [11,000] 0x50003070 [1,000] F <6> [10,000] 0x30000080 [10,000] G <5> [32] 0xe00090a0 [32] H <2> [1,904] 0x50006040 [1,904] I <2> [1,024] 0x10007030 [1,024] J
n.b. A real dump would have names like 'java/lang/String', 'array of Object' instead of 'A', 'B' etc.
Now for an example of actually running the command:
The threshold value filters small objects. A threshold of 1mb (the default) will only show objects with a total/subtree size of 1mb or greater. See the definitions.
The max-depth parameter prevents HeapRoots from going too deep into the graph. A max-depth of 1 will show objects at most 1 level deeper than the starting point. The root objects appear at level 0.
The address value can be supplied to only dump from a single object instead of all of the roots (the default). This can be any object and can be used to 'walk down' the tree and explore.
> d Enter total-size threshold [1048576] > 0 Enter max depth or -ve for unlimited [8] > Enter 0xto dump from one address or any value for all roots [-] > Dumping object(s) : 2 roots, sorted by total-size Filter : depth at most 8 Prune : N <0> [48,000] 0x200000b0 [40,000] K <1> [8,000] 0x200000c0 [8,000] L <2> {48,000} 0x200000b0 [40,000] K <1> <20,600 root:0x50001010> 0x50002020 [4,000] B <0> [26,600] 0x50001010 [6,000] A <1> [20,600] 0x50002020 [4,000] B <2> [13,672] 0x50004050 [1,104] C <3> [12,568] 0x50005060 [1,032] D <4> [11,536] 0x50003090 [504] E <5> [11,000] 0x50003070 [1,000] F <6> [10,000] 0x30000080 [10,000] G <6> {13,672} 0x50004050 [1,104] C <5> [32] 0xe00090a0 [32] H <4> {12,568} 0x50005060 [1,032] D <2> [1,904] 0x50006040 [1,904] I <3> <1,024 parent:0x50002020> 0x10007030 [1,024] J <2> [1,024] 0x10007030 [1,024] J 2/2 roots were dumped. There were 12 objects expanded.
The line for object A shows that it has address 0x50001010, a subtree/total size of [26,600] bytes. Note that B is a child of, but not owned by K.(since B's root-owner is in fact A).
For any node, the biggest child objects (in total size) are shown first. B has children C, I & J - this is in order of total-size and also in order of display. This also applies at the root level, e.g. K is shown before A at level <0>. See Sorting below.
It is intended to be able to add up the sizes in square '[]' brackets to form the total-sizes:
The total-size for K [48,000] is the size of K [40,000] plus the total-size of L (the only object it is the parent-owner for ) [8,000]. The total-size of L is the same as the size here since L doesn't own anything.
When B is listed under K its subtree size is shown in angle-brackets <20,600 root:0x50001010> to reflect that its size does not count towards the total-size of K.
The 2nd line for 'K' is shown in curly '{}' brackets because it has already been shown in the tree descending from 'K'. At this point, the object K is not explored any further to avoid carrying on in the K -> L -> K cycle for ever.
Each object only has its size counted once (against its parent-owner's total-size). Object J's parent is B and so its size is listed in []'s when shown as a child of B. But when it's shown under I, it is listed as <1,024 parent:0x50002020>. This is so that you don't think the size of J [1,024] counts towards the total-size of I.
In the output below, I have filtered on objects at least 10,000 bytes in size and only down to 4 levels deep.
Note that objects H, I, J, & L are too small in subtree size to be shown. Objects F, G & H would be too deep in the output to display. Object H is both too small and too deep down the tree. Compare this output to that above.
> d Enter total-size threshold [0] > 10000 # only objects with total-size at least 10,000 Enter max depth or -ve for unlimited [8] > 4 # only objects with a depth up to <4> Enter 0xto dump from one address or any value for all roots [-] > Dumping object(s) : 2 roots, sorted by total-size Filter : total-size >= 10,000 and depth at most 4 Prune : N <0> [48,000] 0x200000b0 [40,000] K <1> <20,600 root:0x50001010> 0x50002020 [4,000] B - 1 child of 0x200000b0 filtered. # object L is too small <0> [26,600] 0x50001010 [6,000] A <1> [20,600] 0x50002020 [4,000] B <2> [13,672] 0x50004050 [1,104] C <3> [12,568] 0x50005060 [1,032] D <4> [11,536] 0x50003090 [504] E - 2 children of 0x50003090 too deep. # object E is at the maximum-depth <4> {12,568} 0x50005060 [1,032] D - 2 children of 0x50002020 filtered. # I and J are too small, B is not 2/2 roots were dumped. There were 6 objects expanded.
You can remove the explanations of filtering to unclutter the output, see Settings.
I expect you realised I've made up the above example graph. Now onto a real example...
After loading this (large) dump, I processed it with 'p' and then used the 'Objects' display to show just roots (both artificial and pure) in order of size. The commands used are in the end of the objects table documentation.
Addr. Size Root-Owner Total-Size Descend. Name --------------------------------------------------------------------------------- R 0x00555a30 248 - 70,907,808 837,793 class java/lang/System A 0x101963d0 32 - 8,380,272 43,501 java/lang/ref/Finalizer A 0x1d521820 112 - 3,708,912 5,975 org/apache/xpath/XPathContext
The table shows the 3 biggest objects in terms of their total sizes. The total sizes of the roots are calculated by adding up the sizes of every object owned by each root. Every object which isn't a root is assigned to be owned by exactly one root object. The descendants of a root is the number of objects owned by it. See Definitions.
It will normally be the case that most objects are referenced by many other objects indirectly and so HeapRoots presents a gross simplification of the relationships within the heap. In practice however, this seems to help because it reduces the vast complexity of even an average application's heap.
Taking the biggest root 'class java/lang/System' (which itself only takes 248 bytes), it 'owns' 837,793 objects with a total size of around 70mb. It is a 'pure root' because nothing else references it in the heap. Many other objects (some roots, some not) could probably reach 70mb of objects in the heap, some probably could reach more. The reason this root has been chosen to own so many of the objects is probably because of a first-come first-served basis due to its low address 0x00555a30. This is quite a good strategy since classes and objects like java/lang/System often end up with low addresses (not guaranteed by any means) and are good starting points for graph traversal.
It is also quite common to see large amounts of space used by 'java/lang/ref/Finalizer' in heapdumps. This could be normal behaviour or could indicate some kind of build up of resource which cannot be freed. You can use the types table to count finalizers. Large use of finalizers (even when resources are freed correctly in Java code) will cause severe performance problems. This can be in terms of using heap space or slowing down the VM by causing a disproportionate amount of work for GC since they require special handling. See Links page for resources on finalizers.
The third line lists some Xalan/Xerces objects (XSL&XML handling stuff) which turned out to be causing the performance problem in a WebSphere environment. It was in fact a finalizer related issue.
Now that I've seen the top few root objects are all over 3mb, I'll now do a 'dump from roots'. Picking a threshold value is tricky because if you set it to small, then you get so much data you just have to quit and start again. It's generally better to start conservative and iterate towards a finer grain. Also see the Auto total-size Threshold section below.
So I now run 'd'.
> d Enter total-size threshold [1048576] > 3000000 3mb Enter max depth or -ve for unlimited [-1] > 4 easy tiger, don't want too much data Enter 0xto dump from one address or any value for all roots [-] > Dumping object(s) : 1055309 roots, sorted by total-size Filter : total-size >= 3,000,000 and depth at most 4 Prune : N <0> [70,907,808] 0x00555a30 [248] class java/lang/System <1> [70,864,592] 0x10919f58 [24] com/ibm/ejs/security/SecurityManager <2> [70,864,568] 0x101819d8 [72] java/lang/Thread <3> [70,863,592] 0x1018d2b8 [56] java/lang/ThreadGroup <4> [70,862,816] 0x1018d360 [56] java/lang/ThreadGroup - 3 children of 0x1018d360 too deep. - 3 children of 0x1018d2b8 filtered. <3> <70,287,480 parent:0x1017d728> 0x101816a0 [112] com/ibm/ws/bootstrap/ExtClassLoader - 4 children of 0x101819d8 filtered. - 4 children of 0x00555a30 filtered. <0> [8,380,272] 0x101963d0 [32] java/lang/ref/Finalizer <1> [8,380,176] 0x101ba2e8 [32] java/lang/ref/Finalizer <2> [8,380,144] 0x101b8988 [32] java/lang/ref/Finalizer <3> [8,380,112] 0x101b7ab8 [32] java/lang/ref/Finalizer <4> [8,380,080] 0x101c0a40 [32] java/lang/ref/Finalizer - 4 children of 0x101c0a40 too deep. <4> {8,380,144} 0x101b8988 [32] java/lang/ref/Finalizer - 2 children of 0x101b7ab8 filtered. <3> {8,380,176} 0x101ba2e8 [32] java/lang/ref/Finalizer - 2 children of 0x101b8988 filtered. <2> {8,380,272} 0x101963d0 [32] java/lang/ref/Finalizer - 2 children of 0x101ba2e8 filtered. - 3 children of 0x101963d0 filtered. <0> [3,708,912] 0x1d521820 [112] org/apache/xpath/XPathContext <1> [3,561,040] 0x1d539908 [32] org/apache/xpath/VariableStack <2> [3,544,608] 0x1d54f560 [32,784] array of org/apache/xpath/objects/XObject <3> [3,494,736] 0x1ddc4c10 [24] org/apache/xpath/objects/XNodeSet <4> [3,494,712] 0x1ddc4e08 [120] org/apache/xpath/axes/WalkingIteratorSorted - 9 children of 0x1ddc4e08 too deep. <4> <3,390,048 parent:0x1d542318> 0x1d53afc8 [24] org/apache/xml/dtm/ref/DTMManagerDefault - 10 children of 0x1d54f560 filtered. - 1 child of 0x1d539908 filtered. - 15 children of 0x1d521820 filtered. <0> [3,670,304] 0x129d2930 [24] org/apache/xpath/objects/XNodeSet <1> [3,670,280] 0x129d2ac8 [120] org/apache/xpath/axes/WalkingIteratorSorted <2> [3,669,776] 0x128606c8 [136] org/apache/xml/dtm/ref/dom2dtm/DOM2DTM <3> [3,565,784] 0x1285b490 [24] org/apache/xml/dtm/ref/DTMManagerDefault <4> [3,565,760] 0x1379fdb8 [16,400] array of org/apache/xml/dtm/DTM - 145 children of 0x1379fdb8 too deep. - 2 children of 0x1285b490 filtered. - 15 children of 0x128606c8 filtered. <2> {3,670,280} 0x129d2ac8 [120] org/apache/xpath/axes/WalkingIteratorSorted - 7 children of 0x129d2ac8 filtered. <1> {3,565,784} 0x1285b490 [24] org/apache/xml/dtm/ref/DTMManagerDefault 4/1055309 roots were dumped. There were 20 objects expanded.
Now, lets see what's further down the finalizer chain (as if we couldn't guess).
I'll use the value in bold 0x101c0a40 above and dump from this object.
> d 0x101c0a40 Specify 0x101c0a40 as a starting point. This is using an argument/short-cut, see 'help d'. Dumping object(s) : 0x101c0a40 (parent 0x101b7ab8 root 0x101963d0), sorted by total-size Filter : total-size >= 3,000,000 and depth at most 4 Prune : N <0> [8,380,080] 0x101c0a40 [32] java/lang/ref/Finalizer <1> [8,380,048] 0x101c0898 [32] java/lang/ref/Finalizer <2> [8,380,016] 0x101cfde0 [32] java/lang/ref/Finalizer <3> [8,379,984] 0x101db9a0 [32] java/lang/ref/Finalizer <4> [8,379,952] 0x101ddcd0 [32] java/lang/ref/Finalizer - 4 children of 0x101ddcd0 too deep. <4> {8,380,016} 0x101cfde0 [32] java/lang/ref/Finalizer - 2 children of 0x101db9a0 filtered. <3> {8,380,048} 0x101c0898 [32] java/lang/ref/Finalizer - 2 children of 0x101cfde0 filtered. <2> {8,380,080} 0x101c0a40 [32] java/lang/ref/Finalizer - 2 children of 0x101c0898 filtered. <1> <8,380,112 parent:0x101b8988> 0x101b7ab8 [32] java/lang/ref/Finalizer - 2 children of 0x101c0a40 filtered. There were 5 objects expanded.
We just see more finalizers!. See that the 'parent' and 'root' values given in bold correspond to the objects above 0x101c0a40 at levels <3> and <0> respectively in the original output.
I then ran 'd 0x101ddcd0' to see deeper down the tree, but just reveal more finalizers. I'm thinking they'll never end! I try setting max-depth to 300 and redirect output to a file (so that I don't flood my display):
> dt > deeptree.txt Redirecting command output to file 'deeptree.txt'. Enter total-size threshold [3000000] > Enter max depth or -ve for unlimited [300] > 300 # Increase the depth limit. Enter 0xto dump from one address or any value for all roots [0x101c0a40] > 0x101c0a40 # Same object, (4 levels from root).
Looking at the output:
D:\>type deeptree.txt In a cmd window. Dumping object(s) : 0x101c0a40 (parent 0x101b7ab8 root 0x101963d0), sorted by total-size Filter : total-size >= 3,000,000 and depth at most 300 Prune : N <0> [8,380,080] 0x101c0a40 [32] java/lang/ref/Finalizer <1> [8,380,048] 0x101c0898 [32] java/lang/ref/Finalizer <2> [8,380,016] 0x101cfde0 [32] java/lang/ref/Finalizer <3> [8,379,984] 0x101db9a0 [32] java/lang/ref/Finalizer ................................................. <299> [8,370,304] 0x142549d8 [32] java/lang/ref/Finalizer <300> [8,370,272] 0x1452c7b8 [32] java/lang/ref/Finalizer - 4 children of 0x1452c7b8 too deep. <300> {8,370,336} 0x14105c90 [32] java/lang/ref/Finalizer - 2 children of 0x142549d8 filtered. ................................................. There were 301 objects expanded.
So, this chain just seems to keep on going. Luckily, HeapRoots has a 'max-depth' calculation for roots which will tell us how deep this goes.
I change the process settings to calculate 'max-depth' against each object.
> ? p .......................... Select data to store (more data increaes memory usage) [T] total/subtree sizes(T), maximum depth(M), # descendants(D) > MT # Calculate max-depths & total-sizes
Then I recalculate with 'p'. Now running 'dm' to sort by (and show) the max-depth values, you get:
Dumping object(s) : 0x101c0a40 (parent 0x101b7ab8 root 0x101963d0), sorted by max-depth Filter : total-size >= 3,000,000 and depth at most 300 Prune : N <0> [8,380,080 m:3,066] 0x101c0a40 [32] java/lang/ref/Finalizer ................................................. <299> [8,370,304 m:2,767] 0x142549d8 [32] java/lang/ref/Finalizer <300> [8,370,272 m:2,766] 0x1452c7b8 [32] java/lang/ref/Finalizer - 4 children of 0x1452c7b8 too deep. .................................................
We can see that even, at <300> levels deep. The chain has another 2766 levels to go.
We could have run 'om' to see the deepest objects. Here's some sample output:
Showing objects whose type is in 'AR', sorted by max-depth. Addr Size Root-owner Parent Total-size Depth Name ------------------------------------------------------------------------------- A 0x101963d0 32 - - 8,380,272 3,070 java/lang/ref/Finalizer R 0x00555a30 248 - - 70,907,808 418 class java/lang/System
Note that the root has a max-depth of 3,070 and the object 0x101c0a40 (4 levels deep) that we chose to 'dump' from has a max-depth of m:3,066. Then 300 levels deeper, the object's max-depth is reduced by 300 to m:2,766. This shows that 'max-depth' is a measure of how much further there is to go in a tree.
Running 'dt > deepertree.txt' and searching beyond just the Finalizer objects reveals that they start to reference a lot of 'org/apache' type objects. I entered '-1' for the 'max depth' prompt to give unlimited depth and reduced the size threshold to 1000000.
Dumping object(s) : 1055309 roots, sorted by max-depth Filter : total-size >= 1,000,000 # unlimited depth ........................... <1231> [1,357,768 m:1,835] 0x2268fb70 [32] java/lang/ref/Finalizer <1232> [1,140,160 m:174] 0x2268fb90 [40] org/apache/xpath/objects/XRTreeFrag
I can then use the single object dump to get information on what exactly references any of these objects and how they are used within the application. e.g. by using 'i 0x2268fb70' to see what it references/references it from outside this 'finalizer' chain from the root.
I confirm what exactly is in this very deep tree by running 'ts', but only showing objects owned by this root. The following table shows that a lot of the objects in this tree are 'org/apache'.
> ts 0x101963d0 # objects with this root-owner ........................... Tabulating objects whose root-owner equals '0x101963d0', sorted by total-size Count Total-size Type Name ------------------------------------------------------------------------------- 6,966 5,067,504 primitive array 26 426,400 array of org/apache/xml/dtm/DTM 13 426,192 array of org/apache/xpath/objects/XObject 2,689 390,912 array of [I 13 213,200 array of org/apache/xml/utils/PrefixResolver 13 213,200 array of org/apache/xalan/templates/ElemTemplateElement 13 213,200 array of javax/xml/transform/SourceLocator
This by no means shows what the problem was, just how to have a look around in the heap. The actual problem in this scenario was a WebSphere environment problem relating to the use of 'org/apache' code and 'finalizers'. I want to write a full tutorial in future.....
You can sort by 'max-depth' or number of 'descendants' if you have this data available by changing processing settings.
Now running 'dt' ('d' defaults to 'dt') and setting the depth limit to 1 gives :
Dumping object(s) : 2 roots, sorted by total-size Filter : depth at most 1 Prune : Y <0> [48,000] 0x200000b0 [40,000] K # K is biggest <1> [8,000] 0x200000c0 [8,000] L <0> [26,600] 0x50001010 [6,000] A <1> [20,600] 0x50002020 [4,000] B
Use 'dm' to sort by 'max-depth':
> dm # 'm' for max-depth Enter total-size threshold [0] > Enter max depth or -ve for unlimited [1] > Enter 0xto dump from one address or any value for all roots [-] > Dumping object(s) : 2 roots, sorted by max-depth Filter : depth at most 1 Prune : Y <0> [26,600 m:6] 0x50001010 [6,000] A # A has deepest chain (6 deep) from it <1> [20,600 m:5] 0x50002020 [4,000] B # B's max-depth is 5,one less than A's <0> [48,000 m:1] 0x200000b0 [40,000] K <1> [8,000 m:0] 0x200000c0 [8,000] L
Above, A (with the deepest tree) is now shown first rather than K (the biggest). Note that the max-depth values are shown [26,600 m:6], here 6.
Now running 'dd', we sort by # of 'descendants'. Here the order is the same as 'dm', but different data is shown.
Dumping object(s) : 2 roots, sorted by descendants Filter : depth at most 1 Prune : Y <0> [26,600 d:9] 0x50001010 [6,000] A # A has more descendants than K <1> [20,600 d:8] 0x50002020 [4,000] B 9 1 <0> [48,000 d:1] 0x200000b0 [40,000] K <1> [8,000 d:0] 0x200000c0 [8,000] LThere are 12 objects in this dump: A & K (the roots) and their descendants (9 and 1 in quantity respectively).
The 'd auto NNN' command automatically calculates a threshold to show about NNN objects when starting from the roots. In most cases, the number you actually see will be very close to the number specified.
Note that 'd auto' uses a default value of 100.
> d auto 5 # Want to show about 5 objects Setting threshold automatically to show about 5 objects. Dumping object(s) : 2 roots, sorted by total-size Filter : total-size >= 12,568 Prune : N <0> [48,000] 0x200000b0 [40,000] K <1> <20,600 root:0x50001010> 0x50002020 [4,000] B - 1 child of 0x200000b0 filtered. <0> [26,600] 0x50001010 [6,000] A <1> [20,600] 0x50002020 [4,000] B <2> [13,672] 0x50004050 [1,104] C <3> [12,568] 0x50005060 [1,032] D <4> {12,568} 0x50005060 [1,032] D - 1 child of 0x50005060 filtered. - 2 children of 0x50002020 filtered. 2/2 roots were dumped. There were 5 objects expanded.
This is used by the batch option.
The first two settings let you unclutter the output.
See the overview for an example of changing these values from their defaults. See the threshold section for an example of the default 'cluttered' display.
> ? d # Modify settings for 'd' command # Defaults in []'s : Settings for 'Dump from Roots' : Prune indirect references/children ? [N] > Inform about filtered objects ? [Y] > Maximum indent [32] # If indentation was unlimited, you would get > # some very wide lines.