Friday, October 6, 2006

CEC: Dtrace Approaches to Real-World Problems

Jim Fiori

Apparently the presentation
is normally a 3+ hour presentation and Jim requested 3 hours, but all
the slots are 60min.

This is a presentation on approach not learning D.

This entry is a horrible hack job on content, too much and I am
learning more than I

can easily condense/digest usefully on the fly.


Everyone needs
dtrace...but it is not the first thing to run.

Identify a possible issue use Dtrace to figure out what is going on.

Advice: Practice, Practice, Practice


use the manual all
examples are in /usr/demo/dtrace

use quantize(), min()/max()/avg() can hide data

Be careful using the PID provider it can impose load on a highly active

Normal system tools still have their

New tools: intrstat (some
others I haven't used)

Privs -- Root level or RBAC...

Zero Probe Effect -- via
instruction replacement


High User Time dtrace toolkit

C++ Apps

Watch for small
allocations and short allocations

High System Calls

use aggregation

use pfiles to determine target of File descriptor

System time (>10%
or user:sys near 2:1)

prsatst to
find it, dtrace to examine it

Threaded App.

prstat to find it

plockstat ... to see it, single process


use jstack

Java 1.6 has static dtrace providers


Look at I/O and File
systems first

ONLY after regular investigation by DBA (statspack etc)


Watch for TCP Nagle
(buffering requests before sending)

Try TCP no delay on client and server

File system

Watch for
periodic pauses check autoup in large memory (>8G) systems

Hints and Tools

  • Use a sample rate at not quite 1000, to help keep the Dtrace
    probe func from running with kernel actions 

  • Dtrace is running in kernel can't just dump
    the memory location, you need to dump the user memory space

  • .mul() and .div() are SPARC V7...the application should be

  • -c flag...use pid$target ... woot

  • look for system call errors, when things are not working as

  • see atomic_ops(3c) on mutex_lock/unlock for simple variable

  • High ctx...look at FX scheduler (good for some apps)

  • High Migrations...look at binding to a specific processor

  • libumem for memory leaks

  • Chime (GUI...uses Sparklines!!!...I like sparklines)


  • RFE 6311947

  • Dtrace in a zone

  • aggregation printing

No comments:

Post a Comment