October 1995

Welcome Back

Performance Guru: “I've tuned a few kernel variables, but it looks pretty idle to me right now.”

Sysadmin: “Yes, but it gets really, really busy some of the time.....”

Performance Guru: “You should start collecting some data. Call me again when you have some logs of when it is busy.”

Have you ever found that your most troublesome system is idle, or behaves perfectly, when you call in the consultants and performance gurus? Then, when you collect data that shows the problem, the guru wants some extra data that you didn't bother to collect.

The SE toolkit provides a way to capture and distribute the expertise of a performance specialist. I have used the toolkit to build a personalized performance monitoring and tuning tool called virtual_adrian.se. It starts up when the system boots, does some simple tuning, and every 30 seconds, it takes a look at the state of the system. When run with superuser permissions, it can monitor more of the system than ruletool.se. It can also make immediate changes to some kernel tunables.

When you run virtual_adrian.se, it keeps quiet until it sees a problem. Then it complains and shows the data that caused the complaint. For example, if it keeps complaining about a slow disk, then you should find a way to reduce the load on that disk. Unlike a real performance guru, the virtual guru can look at lots of data very quickly, and never gets bored or falls asleep. Most performance problems occur over and over again. The script picks out a large number of “obvious” problems. (Obvious to a guru :-).

If you disagree with my rules (which are a bit crude in places), or want to add your own rules, you can easily modify the script and rename it virtual_kimberley.se or whatever.

The virtual_adrian.se Performance Tuner and Monitor

Check and Tune the System

The usual method is to set variables in /etc/system. The problem is that these variables may be incorrectly set due to folklore. They may also be unnecessary or harmful in later releases, and the /etc/system file is often propagated intact to very different systems. Deciding when a tweak is useful and what the value should be for a particular release is complex. Your motto should be, 'If in doubt, do without!'

My solution is to remove everything from /etc/system that doesn't have a big comment that explains why it is there and how its value was derived. The well known maxusers tunable automatically scales as you add memory, so you don't need to set it unless you have several GB of RAM and are short on kernel memory. I implemented a check-and-tune routine in virtual_adrian.se that checks and tunes some values directly. It tells you what it does, and tells you to set variables in /etc/system for the next reboot if they cannot be set on line. The check is performed once only (at start-up) and is implemented by a routine called static_check(). Note that the current value of each tunable is checked by my script. Please do not blindly set all these values in your own /etc/system file as you may decrease something that you thought you were increasing.

The tuning actions performed for Solaris 2.3 are:

• Reduce slowscan to 100, to make memory reclaims less aggressive, and reduce fastscan to 16000. These are the maxima in Solaris 2.4.
• Increase inode cache size by setting ufs_ninode to 10000. Request that the DNLC size be increased to 5000 by adding set ncsize=5000 in /etc/system.
  (If the values are already this big, leave them alone.)

The tuning actions performed for Solaris 2.4 and 2.5 are:

• Increase the free list threshold lotsfree as CPUs are added, to 128 pages per CPU. This has no effect on uniprocessor systems, which default to 128. A larger free list helps prevent transient kernel memory allocation failures. If the free list is at zero when the kernel needs memory and can't wait, kmem errors will be reported by the kmem error rule.
• Increase inactive inode cache size by setting ufs_ninode to 5000. Request that the DNLC size be increased to 5000 by adding set ncsize=5000 in /etc/system. (If the values are already this big, leave them alone.)

NFS Mount Point Service Time Monitor

Figure 1 Example output from nfsstat -m

-------------------------------------------------------------------------
/home/username from server:/export/home3/username
 Flags:   vers=2,hard,intr,down,dynamic,rsize=8192,wsize=8192,retrans=5
 Lookups: srtt=7 (17ms), dev=4 (20ms), cur=2 (40ms)
 Reads:   srtt=16 (40ms), dev=8 (40ms), cur=6 (120ms)
 Writes:  srtt=19 (47ms), dev=3 (15ms), cur=6 (120ms)
 All:     srtt=15 (37ms), dev=8 (40ms), cur=5 (100ms)
-------------------------------------------------------------------------

Sample Output from virtual_adrian.se

-------------------------------------------------------------------------
% virtual_adrian.se
Warning: Cannot init kvm: Permission denied
Warning: Kvm not initialized: Variables will have invalid values
Adrian is monitoring your system starting at: Thu Sep 21 00:37:26 1995

Warning: Cannot get info for pid 3.
superuser permissions are needed to access every process
Using predefined rules for disk, net, rpcc, swap, ram, kmem,
cpu, mutex, dnlc and inode

Checking the system every 30 seconds...
-------------------------------------------------------------------------

The warning about kvm (the raw kernel data interface) can be ignored, as the script does not attempt to use any kvm data unless it runs as root. When you run it as root you see the extra rules are configured.

-------------------------------------------------------------------------
# /opt/RICHPse/examples/virtual_adrian.se
Adrian is monitoring your system starting at: Thu Sep 21 00:46:01 1995

Process watcher pid set to 3, process name fsflush, max CPU usage  5.0%
NFS client threshold set at All: srtt=20 (50ms) max NFS round trip
Minimum client NFS ops/sec considered active 2.00/s
Using predefined rules for disk, net, rpcc, swap, ram, kmem,
cpu, mutex, dnlc and inode

Checking the system every 30 seconds...
-------------------------------------------------------------------------

I ran the command % find / -ls >/dev/null which makes the disk and the name caches rather busy, and got this output on a 32MB SPARCstation IPX.

-------------------------------------------------------------------------
Adrian detected slow disk(s): Thu Sep 21 00:48:33 1995
Move load from busy disks to idle disks
State  disk      r/s  w/s   Kr/s   Kw/s wait actv  svc_t  %w  %b  delay
red    c0t3d0   23.9  5.6   81.1   44.5  2.6  1.3  130.7  17  78 3856.1
amber  c0t5d0    1.3  1.9    7.1   35.2  0.0  0.1   30.3   0   6   94.9

Adrian detected Directory Name Cache problem (amber): Thu Sep 21 00:48:33 1995
Poor DNLC hitrate, increase ncsize
DNLC hitrate 44.1%, reference rate 125.50/s
DNLC has 617 entries, try increasing it (and inode cache) to 1234

Adrian detected Inode Cache problem (amber): Thu Sep 21 00:48:33 1995
Poor inode cache hitrate, increase ufs_ninode
Inode hitrate 26.9%, reference rate 64.63/s

Adrian detected RAM shortage (amber): Thu Sep 21 00:49:05 1995
The system is getting short on RAM, perhaps add some more
  procs        memory         page             faults              cpu
 r  b  w    swap    free    pi  po  sr   in    sy    cs  smtx us sy wt id
 0  0  0   43940     700     8   2  55  342   842   205    11 23 18 20 39
-------------------------------------------------------------------------

As you can see, the output is somewhat verbose, and is based on extended versions of familiar command output where appropriate.

The whole point of the SE toolkit is that you can customize the tools very easily. To encourage you to do this, the rest of this column is a tour of the SE language and toolkit classes.

• Scripts are passed through the standard C preprocessor before being executed.
• C statement types are all implemented apart from goto.
• C operators are all implemented apart from a few unary and ternary operators.
• Most C data types are implemented, including floating point, but there are no pointers in the language. A new type called string is added. Some common C library routines are redefined to avoid pointer arguments.
• Arrays, structures, and initialized structures are implemented.
• A mechanism for defining entry points into the standard dynamically linked C libraries is provided. This avoids the need to supply more than a basic set of built-in functions in the interpreter.
• A large set of header files is provided. These files define structures that map onto the many kernel data structures which provide performance information.
• The most significant extension to the language is a simple object oriented class. This is just a structure definition with a function definition embedded in it. A variable of this type can be defined so that whenever an element of the structure is accessed, the function is executed first.
• By defining classes that map onto the kernel data sources, every time they are read the current values of each metric are provided.
• The code required to provide data in ready-to-use formats is also embedded in classes. For example, an iostat class provides the per-second rates for each disk as floating point values in a data structure.

Iostat Written in SE

Figure 2 Code for iostat.se

-------------------------------------------------------------------------
#! /opt/RICHPse/bin/se
 
#include <stdio.se>
#include <tdlib.se>
#include <unistd.se>
#include <string.se>
#include <kstat.se>
#include <sysdepend.se>
#include <p_iostat_class.se>
#include <dirent.se>
#include <inst_to_path_class.se>

#define SAMPLE_INTERVAL   5

main(int argc, string argv[2])
{
  p_iostat p_iostat$disk;
  p_iostat tmp_disk;
  int i;
  int interval = SAMPLE_INTERVAL;
  int ndisks;

  switch(argc) {
  case 1:
    break;
  case 2:
    interval = atoi(argv[1]);
    break;
  default:
    printf("use: %s [interval]\n", argv[0]);
    exit(1);
  }
  ndisks = p_iostat$disk.disk_count;
  for(;;) {
    sleep(interval);
    printf("extended disk statistics\n");
    printf("disk      r/s  w/s   Kr/s   Kw/s wait actv  svc_t  %%w  %%b\n");
    for(i=0; i < ndisks; i++) {
      p_iostat$disk.number$ = i;
      tmp_disk = p_iostat$disk;
      printf("%-8.8s %4.1f %4.1f %6.1f %6.1f %4.1f %4.1f %6.1f %3.0f %3.0f\n",
        tmp_disk.name$,
        tmp_disk.reads, tmp_disk.writes,
        tmp_disk.kreads, tmp_disk.kwrites,
        tmp_disk.avg_wait, tmp_disk.avg_run,
        tmp_disk.service,
        tmp_disk.wait_percent, tmp_disk.run_percent);
    }
  }
}
-------------------------------------------------------------------------

1. The first line tells the shell where to find the interpreter for this script.
2. The #include lines bring in the definitions, and the code that does the real work.
3. The definitions in main() define two copies of the per-disk iostat class. One has the special prefix “p_iostat$” that matches the function name defined in the class. This is an active variable. By convention, dollar signs are embedded in active variables to indicate that they are special. The other is just a regular data structure of the same storage type. It is used to hold a temporary snapshot copy of the data. Internally, a snapshot of the kernel's disk counters is made as the class initializes itself.
4. Simple command line processing allows the measurement interval to be specified.
5. The first read of the class picks out how many disks there are.
6. The script loops forever until it is interrupted. It sleeps for a few seconds to start with, so the subsequent measurement is made over the specified interval.
7. After printing the headers, the script loops through the disks. For each disk, you must write the index into the class. The index tells the class which disk you want data for.
8. A structure-to-structure copy takes a snapshot of all the data for a single disk. Before the data is read from the class the kernel counters are read, differenced from the previous measurements, and processed into floating point values.
9. All that remains is to print out the data in the right format.

-------------------------------------------------------------------------
% iostat.se
extended disk statistics
disk      r/s  w/s   Kr/s   Kw/s wait actv  svc_t  %w  %b
sd3       0.0  0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd5       0.0  4.6    0.0   29.2  0.0  0.1   18.3   1   5
-------------------------------------------------------------------------

States and Actions

• White state - completely idle, untouched
• Blue state - imbalanced, idle while other instances of the resource are overloaded
• Green state - no problem, normal operating state
• Amber state - warning condition
• Red state - overloaded or problem detected
• Black state - critical problem that may cause processes or the whole system to fail

Rules as Objects

• A way to package each rule
• A common interface based on a data structure
• The implementation is hidden in the class function
• Changes to the rule can be made without changing the interface as long as the input data is the same
• Rules become reusable components stored in a header file
• Existing rules provide templates for the creation of new rules
• Common operations, such as setting custom thresholds, can be performed using generic functions

Live Rule Objects

• Read data from the current system
• Feed it to a copy of the pure rule
• Pass on the state code and action string
• Pass on derived data used to determine the state and action
• Maintain globally available copies of the complete vmstat, iostat, and netstat classes

-------------------------------------------------------------------------
lr_disk_t lr_disk$dr;
lr_disk_t tmp_dr;
/* use the live disk rule */
tmp_dr = lr_disk$dr;
if ( tmp_dr.state > ST_GREEN) {
	printf("The disks are in the %s state: %s\n",
		state_string(tmp_dr.state), tmp_dr.action);
}
-------------------------------------------------------------------------

Rule Definition

The number of CPUs on the system must be known. It is referred to as ncpus in the rules. The run queue length is divided by the number of CPUs. This is based on the assumption that every CPU takes a job off the run queue in each time slice.

Table 1 CPU Rule

-------------------------------------------------------------------------
CPU RULE				LEVEL		ACTION
-------------------------------------------------------------------------
0 == vmstat30.r				White		1. CPU Idle
0 < (vmstat30.r / ncpus) < 3.0		Green		No Problem
3.0 <= (vmstat30.r / ncpus) < 5.0	Amber		2. CPU Busy
5.0 <= (vmstat30.r / ncpus)		Red		2. CPU Busy
-------------------------------------------------------------------------

1. CPU Idle - The CPU power of this system is underutilized. Fewer or less powerful CPUs could do this job.

2. CPU Busy - There is insufficient CPU power, and jobs spend an increasing amount of time in the queue before being assigned to a CPU. This reduces throughput and increases interactive response times.

Figure 3 Code for Pure CPU Rule

-------------------------------------------------------------------------
rule_thresh_dbl cpu_runq_idle = {"RUNQ_IDLE", 0.0, "",
	4, 1, "Spare CPU capacity" };
rule_thresh_dbl cpu_runq_busy = {"RUNQ_BUSY", 3.0, "",
	4, 1, "OK up to this level" };
rule_thresh_dbl cpu_runq_overload = {"RUNQ_OVERLOAD",
	5.0, "", 4, 1, "Warning up to this level" };
 
print_pr_cpu(ulong file) {
    print_thresh_dbl(file, cpu_runq_idle);
    print_thresh_dbl(file, cpu_runq_busy);
    print_thresh_dbl(file, cpu_runq_overload);
}

class pr_cpu_t {
  /* output variables */
  int state;
  string action;
  /* input variables */
  ulong timestamp;
  int runque; /* i.e. p_vmstat.runque load level */
  int ncpus; /* i.e. sysconf(_SC_NPROCESSORS_ONLN) */
  /* threshold variables */
  double cpu_idle;
  double cpu_busy;
  double cpu_overload;

  pr_cpu$()
  {
    double cpu_load;
    ulong lasttime;	   /* previous timestamp */
    if (timestamp == 0) { /* reset defaults */
	 cpu_idle = get_thresh_dbl(cpu_runq_idle);
	 cpu_busy = get_thresh_dbl(cpu_runq_busy);
	 cpu_overload = get_thresh_dbl(cpu_runq_overload);
	 return;
    }
    if (timestamp != lasttime) {
      cpu_load = runque; cpu_load /= ncpus;
      if (cpu_load <= cpu_idle)
        {
        state = ST_WHITE;
        action = "There is more CPU power configured than you need right now";
        }
      else
        {
        if (cpu_load < cpu_busy)
          {
          state = ST_GREEN;
          action = "No problem";
          }
        else
          {
          if (cpu_load < cpu_overload)
            {
            state = ST_AMBER;
            action = "The CPU is quite busy, perhaps add more CPU power";
            }
          else
            {
            state = ST_RED;
            action = "CPU overload, add more power or quit some programs";
            }
          }
        }
	 lasttime = timestamp;
    }
  }
};
-------------------------------------------------------------------------

The code defines three threshold structures and a function to print them to a file descriptor. The function is provided for convenient use in scripts.

The CPU rule itself is defined as a class. The first part is just like a regular C structure definition. By convention, output variables for the state and action are always defined. Input variables are used to provide information needed by the rule (runque and ncpus) and time stamp each invocation of the rule. Threshold variables hold the current values of the thresholds.

The final element of the class is the block of code that is first executed when the class is declared and againwhenever its data is read. The function name ends in a “$” and must be used as the prefix for the name of any active instances of the class.

Some local variables are defined. By convention the time stamp is used in two special ways. A zero time stamp executes rule initialization code. In this case, the thresholds are set using a function that checks the environment variable and if not defined goes to the default. If the time stamp is unchanged from the last invocation of the rule, the code is not executed. This prevents unnecessary evaluations of the rule. I use a one-second resolution time stamp.

Finally, the per-CPU run queue length is calculated and compared with the thresholds to determine the state code and action string.

The rule is used when setting the input variables, updating the time stamp, and reading the output variables. This will be shown in the definition of the live rule.

CPU Live Rule Code

The live rule initializes itself when it is first declared, i.e. before the script starts to run, by reading the time, updating the global copy of the vmstat class data, setting the number of CPUs correctly in the pure rule, then resetting the pure rule while initializing the temporary copy. The state code and action string are set up.

The first time the live rule is actually read, it updates the global copy of the vmstat class, then sets the run queue and time stamp in the pure rule, and invokes it by reading it into the temporary copy. The state code and action string are propagated unchanged, up to the live rule values.

Figure 4 Code for Live CPU Rule

-------------------------------------------------------------------------
class lr_cpu_t {
  /* output variables */
  int state;
  string action;

  lr_cpu$()
  {
    ulong lasttime = 0; /* previous timestamp */
    ulong timestamp = 0;
    pr_cpu_t pr_cpu$cpu;
    pr_cpu_t tmp_cpu;

    if (timestamp == 0) {
      timestamp = time();
      pvm_update(timestamp);
      pr_cpu$cpu.ncpus = GLOBAL_pvm_ncpus;
      pr_cpu$cpu.timestamp = 0;
      tmp_cpu = pr_cpu$cpu; /* reset pure rule */
      action = uninit;
      state = ST_WHITE;
      lasttime = timestamp;
      return;
    }
    timestamp = time();  
    if (timestamp == lasttime) { 
      return;
    }
    /* use the rule */
    pvm_update(timestamp);
    pr_cpu$cpu.runque = GLOBAL_pvm[0].runque;
    pr_cpu$cpu.timestamp = timestamp;  
    tmp_cpu = pr_cpu$cpu;
    state = tmp_cpu.state;
    action = tmp_cpu.action;
    lasttime = timestamp;   
  }
};
-------------------------------------------------------------------------

This is one of the simplest live rules, but it is used in the same way as the most complex ones. Whenever you want to know what state the CPU is in, simply define an instance of the live rule class, then read it.

That's All Folks!

Send your comments and questions to adrian.cockcroft@sun.com.

Previous Column