Friday, October 6, 2006

CEC: Dtrace Approaches to Real-World Problems

Jim Fiori


Apparently the presentation
is normally a 3+ hour presentation and Jim requested 3 hours, but all
the slots are 60min.

This is a presentation on approach not learning D.


This entry is a horrible hack job on content, too much and I am
learning more than I


can easily condense/digest usefully on the fly.


INTRO



Everyone needs
dtrace...but it is not the first thing to run.


Identify a possible issue use Dtrace to figure out what is going on.


Advice: Practice, Practice, Practice

Approach



use the manual all
examples are in /usr/demo/dtrace


use quantize(), min()/max()/avg() can hide data




Be careful using the PID provider it can impose load on a highly active
process.




Normal system tools still have their
place.

New tools: intrstat (some
others I haven't used)



Privs -- Root level or RBAC...


Zero Probe Effect -- via
instruction replacement


Scenarios


High User Time

hotuser.sh dtrace toolkit




C++ Apps


Watch for small
allocations and short allocations




High System Calls
(>100s)

use aggregation

use pfiles to determine target of File descriptor





System time (>10%
or user:sys near 2:1)

prsatst to
find it, dtrace to examine it




Threaded App.

prstat to find it


plockstat ... to see it, single process


Java

use jstack

Java 1.6 has static dtrace providers




Oracle

Look at I/O and File
systems first


ONLY after regular investigation by DBA (statspack etc)




Sybase

Watch for TCP Nagle
(buffering requests before sending)


Try TCP no delay on client and server

File system


Watch for
periodic pauses check autoup in large memory (>8G) systems






Hints and Tools



  • Use a sample rate at not quite 1000, to help keep the Dtrace
    probe func from running with kernel actions 


  • Dtrace is running in kernel context...you can't just dump
    the memory location, you need to dump the user memory space



  • .mul() and .div() are SPARC V7...the application should be
    recompiled



  • -c flag...use pid$target ... woot



  • look for system call errors, when things are not working as
    expected



  • see atomic_ops(3c) on mutex_lock/unlock for simple variable
    updates



  • High ctx...look at FX scheduler (good for some apps)



  • High Migrations...look at binding to a specific processor



  • libumem for memory leaks



  • Chime (GUI...uses Sparklines!!!...I like sparklines)
    OpenSolaris


Coming



  • RFE 6311947 svn_43...fun/mod/ufunc/umod



  • Dtrace in a zone



  • aggregation printing




No comments:

Post a Comment