PERFORMANCE AND TUNING – CPU BOTTLENECKS

Symptoms:
• zero percent idle CPU
• high percent user CPU
• large run-queue size sustained over time
• processes blocked on priority
• slow response time
• high percent system CPU

Metrics:
• user CPU utilization
• system CPU utilization
• idle CPU
• run queue size
• load average
• system call rate
• context switch rate

Tunable kernel parameters:
• maxdsiz
• maxssiz
• maxtsiz
• maxuprc
• nproc
• semmap
• semmni
• semmns
• semmnu
• semume
• semvmx
• maxusers
• npty
• unlockable_mem

Hardware solutions:
• upgrade to a faster processor
• upgrade with larger data/instruction cache
• add a processor
• spread applications to multiple systems (client-server architecture)
• use diskless nodes
• add a floating point coprocessor

Software solutions:
• run long batch jobs during non-peak hours
• nice unimportant applications
• nasty i/o hogs
• use rtprio to help important processes
• use plock to help important processes
• turn off system accounting
• load balance software
• optimize the application
• use Process Resource Manager (PRM)

Tools:
• sar -u #seconds #intervals – For cpu utilization. Not good: %wio > 7 and %idle near 0
• sar -q #seconds #intervals – For queue lengths. Not good: runq-sz > 4 or %swapocc > 5
• sar -c #seconds #intervals – For system calls. Watch for scall/s > 1000 or # of fork/s and # of exec/s
• top – For top processes on the system. Also, system load average.
• uptime – For system load average. Also, how long system has been up and number of users
• uptime -w – For current activity on system for each user.
• ps -ealf – For long output of the ps command. Good for priority, user id, and run state
• ps -lt tty?? – For tracing processes on a specific tty
• glance
• gpm
PERFORMANCE AND TUNING – MEMORY BOTTLENECKS

Symptoms:
• Large run queue with idle CPU utilization
• Out of memory errors
• CPU time to vhand and swapper
• High global system CPU utilization
• High pageout rate
• Process deactivation activity
• Very small free memory and large active virtual memory
• High disk activity on swap devices

Metrics:
• Buffer cache size
• Page in/out rates
• Swap in/out rates
• Available memory size
• Free memory size
• Swap queue length
• Resident set size
• Virtual set size
• VM reads and writes

Tunable kernel parameters:
• shmmax
• shmmni
• shmseg

Hardware solutions:
• Add physical memory.
• Use diskless workstations rather than X-terminals

Software solutions:
• Reduce maxdsiz
• Reduce the use of memory locking in the application.
• Kill unnecessary windows.
• Identify programs with memory leaks.
• Redesign the application.
• Reduce the size of the kernel (limit drivers and subsystems).
• Reduce the size of system tables.
• Reduce the size of the file system buffer cache.

Tools:
• vmstat -d #seconds #intervals – For disk activity.
• vmstat -S #seconds #intervals – For swap in/out activity.
• ipcs -qop – For active message queues.
• ipcs -m – For shared memory.
• ipcs -s – For semaphores.
• glance
• gpm
PERFORMANCE AND TUNING – DISK BOTTLENECKS

Symptoms:
• High disk utilization
• Large disk queue length (over 4)
• Large percentage of time waiting for disk I/O
• Large physical I/O rates
• Low buffer cache hit ratio
• Large run queues with idle CPU

Metrics:
• Disk utilization
• Disk queue length
• Read/write rates per spindle/section
• Raw I/O
• Swap queue length
• Buffer cache hit ratio
• NFS and diskless rates (server)

Tunable kernel parameters:
• Increase buffer cache size
• Increase the kernel table sizes
• bufpages
• nbuf
• dbc_min_pct
• dbc_max_pct
• fs_async
• maxfiles
• maxfiles_lim
• nfile
• nflocks
• ninodes
• nswapdev
• nswapfs
• swchuck

Hardware solutions:
• Add disk drives
• Add disk channels
• Use faster disks
• Use disk striping – better performance
• Use disk mirroring – lower performance

Disk Tuning – Configuration Solutions:
• Use multiple swap areas on different spindles
• Balance disk I/O across multiple spindles
• Dedicate a disk section to an application
• Dedicate a disk section to a file access type
• Use raw disk I/O
• Use mkfs options

Disk Tuning – File System Solutions:
• Use a clean, empty file system.
• Keep the directory hierarchy flat.
• Use relative path names.
• Minimize symbolic links.
• Tune the file system.
• Tune the path variable.

Disk Tuning – File System Tuning:
• block size – smaller block sizes favor smaller file sizes.
• fragment size – smaller fragment sizes favor smaller files
• bytes per inode – with databases, you must have large number of inodes
• minfree
• maxbpg
• rotational delay

Tools:
• bdf -i – free inodes
• iostat #seconds #intervals – cpu utilization
• iostat -t #seconds #intervals – tty activity and cpu utilization
• vmstat -d #seconds #intervals – snapshot of paging activity
• sar -d #seconds #intervals – disk activity
• sar -b #seconds #intervals – buffer activity
• sar -w #seconds #intervals – swapping activity
• swapinfo -t – swap space utilization
• nfsstat -c – nfs statistics for the client
• nfsstat -s – nfs statistics for the server
• nfsstat -rc – rpc statistics for the client
• nfsstat -rs – rpc statistics for the server
• glance gpm
PERFORMANCE AND TUNING – NETWORKING AND I/O BOTTLENECKS
Symptoms:
• Slow application performance
• Time-out requests
• High utilization from LAN tool
• Retries
• Collisions
• Errors
• High I/O activity

Tunable kernel parameters:
• nproc
• ninode
• nfile
• nflocks

Tools:
• ping
• netstat -i – for interface analysis – not good: Ierrs > 5%
• netstat -a – for socket analysis
• netstat -m – for memory analysis
• netstat -r – for router analysis
• iostat -t #seconds #intervals – for terminal activity

PERFORMANCE MONITORING TOOL – MEMORY USAGE

memusg – A tool to compute the memory/IPC resources on a machine.

This utility will do the following :
a) Prints a status of all IPC configurable parameters status per machine.
b) Prints memory usage of a Unix user-id. This particular feature takes a few seconds to generate as it makes an expensive call to the process table. The usage is as follows :

memusg -i Prints a status of the IPC parameters
memusg -m Prints memory usage of a Unix user-id
memusg -h Prints a description of all IPC parameters

This should help us in determining the memory being used per user, there by plan on the memory requirements for a machine.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.