Linux Tools Project/Systemtap/User Guide

Overview

Welcome to the SystemTap plug-in Help pages. This section is intended to provide users with information and links about the SystemTap scripting language itself, in addition to a brief introduction to SystemTap plug-in's structure for new users and a brief explanation of SystemTap.

The SystemTap Plugin

The SystemTap Plugin is designed to help you create, edit, and run Systemtap scripts. It also provides the ability to graph the output of your scripts and run Systemtap scripts remotely.

It provides the SystemTap IDE perspective. When in this perspective you should be able to see the Probe view, the Function view, and the Project Explorer. You should also be able to create Systemtap Scripts.

Systemtap

SystemTap provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux kernel. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.

SystemTap provides a simple command line interface and scripting language for writing instrumentation for a live running kernel. The internal tapset library as well as the published samples can be used to aid reuse and abstraction.

Current project members include Red Hat, IBM, Intel, and Hitachi.

Installing

Requirements

Hardware Requirements:

• x86, x86_64, ppc64, s390

Sotware Requirements:

• Linux kernel v2.6.14 or newer
  • Linux kernel devel and debuginfo packages
• Eclipse v3.8/v4.2 or newer
• SystemTap v0.7 or newer

Once all dependencies are installed, the easiest way to install the SystemTap plug-in for Eclipse is through the Software Updates and Add-ons menu. For information on how to use this menu, refer to this link.

Kernel Source Location

In order to use the kernel source browser you'll need to point it to the kernel source initially. Do this by either selecting the Kernel Source Browser in the browser pane or by going to Window->Preferences, then going to the SystemTap->IDE->'Kernel Source Path' tab. It is possible to explore the kernel source in your local machine or in the same machine used to run SystemTap scripts. Enter your option and inform the kernel source location. If you selected the remote option, it is necessary to set up the remote machine information at SystemTap->'Remote Server' tab.

Tutorials

If you haven't already, you will want to take the IDE and Graphing tutorials. You can find them in the Getting Started section of each perspective's help pages. These tutorial provides fundamental knowledge on graphing in SystemTap plug-in.

Preferences (Optional)

Finally, you may want to set certain non-essential preferences before using SystemTap plug-in regularly. Use Window->Preferences to access the Preferences page.

Of particular note are Logging->Log To, SystemTap->Environment Variables, SystemTap->Graphing->Refresh Delay, SystemTap->IDE->Editor->Syntax Coloring.

Links to SystemTap Resources

The following links contain information specific to the SystemTap scripting language.

• SystemTap website - http://sourceware.org/systemtap/
• SystemTap wiki - http://sourceware.org/systemtap/wiki
• SystemTap tutorial - http://sourceware.org/systemtap/tutorial/
• Full list of documentation - http://sourceware.org/systemtap/documentation.html

The following links are related to the SystemTap GUI project:

• SystemTap GUI Sourceforge - https://sourceforge.net/projects/stapgui/
• SystemTap GUI Homepage - http://stapgui.sourceforge.net/

SystemTap IDE

Getting Started

IDE Tutorial

Lesson 1: Getting Started

In this tutorial we will orient you with the Systemtap IDE and its functions. At this point we assume that you are able to run Systemtap plug-in and are unfamiliar with how to proceed. If you cannot yet start the application, please refer to our Installation help page. NOTE: If this is your the first time opening the Systemtap IDE, it will take a while to load as it has to parse the tapsets.

The Systemtap IDE's purpose is to allow the user to create, view, and edit Systemtap scripts. There are other editors built in (plain text and the C editor), however, they ultimately exist to facilitate the writing of Systemtap scripts.

In this lesson we will open a new Systemtap script file to familiarize ourselves with the Systemtap IDE. If you don't already have a project you plan to put your new script in start by creating a new one. Select the File menu -> New -> Project -> General -> Project. Follow the wizard to create a new project.

Now you are ready to create the script. Select File -> New -> Other -> Systemtap -> Systemtap Script. Follow the wizard to provide your script name and the project which will contain it (Use the project we created above).

This should open up a new editor with a small stub of a script.

At this point proceed to Lesson 2: Writing Your First Script if you wish to learn how to write your own script file or Lesson 3: Running Your Script if you wish to run this one.

Lesson 2: Writing Your First Script

In this tutorial we will guide you through the process of writing your first Systemtap script. It is strongly recommended that you review the Systemtap website's tutorial - http://sourceware.org/systemtap/tutorial for up-to-date information on the latest version of Systemtap.

In the previous lesson we created a new script and opened it in the editor.

Now try writing a new script or type/copy the following example:

 
	global read, write, start

	probe begin {
		start = gettimeofday_s()
	}
	probe syscall.write {
		write += count
	}

	probe timer.ms(1000) {
		printf("%d\t%d\t%d\n", (gettimeofday_s()-start), read, write)
		read=0
		write=0
	}

Now to demonstrate the functionality of the Probe Alias browser we will have you complete the read probe yourself. Start by opening the syscall folder in the Probe Alias browser. If you do not have any content in the browser you are experiencing a problem with Systemtap installation and should refer to our Installation help page. Ensure your cursor is located at the end of the file. Now scroll down and double click the read probe alias. Systemtap GUI will insert the skeleton probe at the point at which your cursor is at, and should look similar to the following:

probe syscall.read
{
	/*
	 * available variables on this probe:
	 * argstr, buf_uaddr, count, fd, name
	 */

}

Now insert the following line into the syscall.read probe:

read += count

You may remove the comment (/* ... */) if you wish This will count the number of bytes read and written each second and print it out. The begin probe executes first, by getting the time of day. The read and write probes increment each time the function is called. The timer probe prints the information every second. If you typed the script in manually you may have noticed that the editor provides code completion for probe alias. If you did not, type "syscall.". You'll see a box come up that you may use to select an item to complete your probe alias.

Another way to get assistance while writing a new probe is to press CTRL+Space at any point during editing. Doing so will provide autocompletion for partially spelled probe names, what variables and functions are available inside a probe and documentation where available.

In Lesson 3 you will learn how to run Systemtap scripts in the IDE Perspective.

Lesson 3: Running Your Script

In this lesson we will cover running your script. You may use either the script you wrote in Lesson two or write a new one.

There are two ways to run a script:

• Simple Run - Clicking this button in the toolbar will run the current script as the current user on the local machine, and will print all script output to the Eclipse console.
• Run Configuration - For more control over running options, you can use a Run Configuration. To run a script with an existing Run Configuration, open the script file you want to run, then do any of the following:
  • right-click on the script editor or the script file in the Project Explorer, and select Run As->Systemtap Script
  • click on the Run As... button on the toolbar
  • select either Run->Run or Run->Run As->Systemtap Script on the menu bar
  • press Ctrl+F11 on the keyboard

If any of these options are used when a Run Configuration is not yet created for the script, a new one with default settings will be created for it. If multiple Run Configurations exist, you will be prompted to select which one to run the script with.

If you know which Run Configuration you want to use, you may simply click the arrow next to the Run As button on the toolbar and select from a list of recently-run configurations.

To create a new Run Configuration, right-click on the script editor or the script file in the Project Explorer then select Run As -> Run Configurations.... Once the dialogue pops up double click on the SystemTap tree item and you should be presented with the following dialogue:

Here you can specify the path of the script to run, enter remote host information, run as a different user, as well as various other options for running systemtap. Explore the various tabs and options. The tooltips should provide help in understanding what the various options do. When you are ready click Run to run your script.

Now we will demonstrate running Systemtap scripts with graphs. In order to run the chart example you'll need to use the code provided in Lesson 2: Writing Your First Script:

 global read, write, start
 
 probe begin {
   start = gettimeofday_s()
 }
 
 probe syscall.write {
   write += count
 }
 
 probe syscall.read {
   read += count
 }
 
 probe timer.ms(1000) {
   printf("%d\t%d\t%d\n", (gettimeofday_s()-start), read, write)
   read=0
   write=0
 }

To do this select Run As -> Run Configurations... and create a new SystemTap Launch Configuration as you did previously and switch to the graphing tab.

Select Graph the output of this script run, and in the text box labelled Regular Expression, enter a regular expression that will match the output of your script. Place parentheses () around whatever data you want to capture for graphing.

For this example, enter the following as your Regular Expression:

	(\d+)\t(\d+)\t(\d+)

This expression will capture the output produced by the example script. Note that in order to capture data from script output, the output must be formatted into lines/rows. Output-parsing regular expressions try to match individual lines of script output, so make sure your scripts print a line break ("\n") to mark the end of a line. (The line break character does not need to be included in the regular expression.)

Once this expression is entered, a table of Titles and Sample Extracted Values will appear. To test that your regular expression is working as intended, you may enter as Sample Output a line of text similar to what you expect your script to produce.

For this example, copy and paste the following line into the Sample Output box:

	10	1000	100

and enter the following as Titles:

	Time
	Read
	Write

The text boxes displayed under Sample Extracted Value now show you the values that are captured by your regular expression, which will be plotted in any graphs you create that use those values.

Now to add a graph using the parsed output, click the Add... button in the Graph Set section, which will bring up the Create Graph wizard. Select a line graph as the Graph Type, and enter the following as your Title and series values, then click Finish to save your changes:

	Title: 		Read Write Count
	X Series: 	Time
	Y Series 0: 	Read
	Y Series 1: 	Write

You may also specify additional regular expressions to capture output with. To do this, either press the Down arrow key while typing into the Regular Expression text box, or click the drop-down arrow of the text box and click the entry labelled New Regular Expression...

You will notice that all data entered in the Graphing menu so far, such as the sample output and graphs, will no longer visible after adding a new regular expression. This is because each regular expression has its own sample output, column titles, and set of graphs (or Graph Set) associated with it. Data for the currently-selected expression is displayed, and since the current expression is blank, so are the column & graph settings.

To capture the example script's output only at intervals of 10 units of time, use the line below as your second regular expression:

	(\d+0)\t(\d+)\t(\d+)

Enter the same sample output & title values that were used for the first regular expression. Also add a graph with the same series data as before, and with a title of "Decade Read Write Counts".

When the script is executed under this Run Configuration, the Graph Selector view is opened and it will contain a Data View and any graphs you have created, with a tab for each regular expression's set of graphs. The Data View chart is populated with the output from the console. If you click on the tab labelled Graph Set 1, you should see a screen similar to the following:

Switch between the graphs in Graph Set 1 and 2 to see the difference between the two graphs you plotted. If you click on the Data View tab of either Graph Set, you will see that the data that is being parsed live from the console:

You may also create new graphs while in this view, to display data captured by your regular expressions in alternate ways. To create a new graph, first select the Graph Set you want to add a new graph to, then click on the Create Graph button, which appears to the left of the Data View tab. This will bring up the same Create Graph wizard that appears when creating a graph in a script's Run Configurations menu.

Those are the basics behind running a script in Systemtap GUI.

Concepts

SystemTap IDE Perspective

This is the primary perspective for SystemTap script writers. It contains the Function Browser, Kernel Browser, Probe Alias Browser, and Views and Editors. The IDE Perspective contains the features necessary execute SystemTap scripts and the syntax highlighting and code completion features that facilitate efficient development of scripts.

STP Editor

This editor is the most commonly used; it is used for the authoring and editing of Systemtap scripts. It is invoked when the user starts a new file or opens a file with the .stp extension. It provides syntax highlighting, code completion, and run support.

See also: Menus, Example SystemTap Scripts, Toolbars

Function Browser

The Funtion Browser lists all of the functions that are defined in the tapset library. Functions are grouped by the file that they are defined. Each listed function has an icon that corresponds to its return type.

• String
• Long
• Void

If you double click on any function a call to that function will be added to the active editor at the current cursor location. Also if you right click on a function and click "View Definition" the tapset file where the function is defined will be opened.

The default behavior of this view is to look for functions in the tapset library in the local machine. To look for functions in the same machine used to run the SystemTap scripts, go to Window->Preferences, select SystemTap->SystemTap IDE and check the Use remote connection to load SystemTap probes and functions option. Don't forget to configure the remote server used by remote execution in SystemTap->Remote Server.

Kernel Browser

The Kernel Source Browser is used to display the kernel tree. This provides you with a quick way to locate places to add probes. Any file in the Kernel Source tree can be opened by double clicking on it. This will open up the selected file in the Editor view. You can then double click on the ruler next to any line that you wish to probe. If SystemTap can probe that line a probe point will be inserted at the end of the script you were most recently working on. If the line can't be probed, an error message will be displayed.

To use this feature it is necessary to configure the Kernel Source Location.

Probe Alias Browser

The Probe Alias Browser lists all of the probe aliases that are defined in the tapset library. Probes are grouped based on their type. In addition to the list of probes, each probe lists all of the variables that it has access to. Each variable is listed with an icon that represents its type. Types are as follows:

• String
• Long
• Unknown

If you double click on any probe alias, probe point will be added to the end of the active editor. Also if you right click on a probe and click "View Definition" the tapset file where the probe is defined will be opened.

The default behavior of this view is to look for probes in the local machine. To look for probes in the same machine used to run the SystemTap scripts, go to Window->Preferences, select SystemTap->SystemTap IDE and check the Use remote connection to load SystemTap probes and functions option. Don't forget to configure the remote server used by remote execution in SystemTap->Remote Server.

Tasks

Creating SystemTap Scripts

Create a SystemTap script by selecting File -> New -> Other... -> Systemtap -> Systemtap Script from the menu bar. You'll be prompted to enter a name for the file; make sure that you use a .stp extension or else the SystemTap IDE will not recognize the file as a SystemTap script and certain editor and run features will not be active.

Your new file will be open and ready for coding in the Editor pane.

A IDE Tutorial is also provided in the help pages for a step by step demo.

For advanced SystemTap langauge help please see the SystemTap Development User's Guide and the documentation available at the SystemTap website's documentation page - http://sourceware.org/systemtap/documentation.html

Using the STP Editor

The STP Editor is used for the authoring and editing of Systemtap scripts. It is invoked when the user starts a new file or opens a file with the .stp extension. Using the editor is as easy as setting the focus to the editor pane and typing.

There are certain features within the editor you may want to take advantage of:

• Syntax Highlighting - Certain code items are colored in .stp and .c files based on a set of rules and colors. You can turn this on or off and change the colors in Window->Preferences.
• Code Assist - SystemTap GUI will attempt to finish your probe alias names based on what you have already typed. You can then select the appropriate probe alias or continue typing to filter the list down even more. You can turn this feature on and off or set the speed on it in Window->Preferences.
• Run Support - See the Lesson 3: Running Your Script help page on a walk through use of the Run command. The Run command allows scripts to be executed directly through SystemTap GUI.

See also: Menus, Toolbars, Example SystemTap Scripts

Viewing Data Charts

Viewing data charts from a SystemTap script is accomplished by running that script with the Run Configurations containing your graphs. This is covered in Lesson 3: Running Your Script. After running a script with the graphing option, the user is presented with the Graph Selector View, and the last graph of the final Graph Set is loaded by default. When that view appears, click on the Data View tab of any Graph Set to see it begin populating as it receives input back through the console from the running SystemTap script.

Reference

Views and Editors

This section details the views and editor in relation to the Systemtap IDE.

Views

The IDE Perspective consists of 5 views and and two editors.

• Probe Alias Browser - A list of all probe aliases currently available in your version of Systemtap. Covered in detail here.
• Function Browser - A list of functions currently supported in your version of Systemtap. Covered in detail here.
• Kernel Source Browser - Used to display the kernel tree. Covered in detail here.
• Console - The console view is where all of the output from the running script is shown. If more then one script is running at the same time you are able to switch between which console is currently displayed by clicking on the "Display Selected Console" button. The script can halt execution here if the user selects the Stop button. The user also has the option of saving the console output to a file, which can be done by clicking the Save log output button at the top of the console.

• Error Log - If the script you tried to run contained errors, this view will become active and display the errors. You can double click on the eror message to jump to the line in the file where the error occured.

Editors

Three editors are available in the Systemtap IDE.

• STP Editor - This editor is the most commonly used; it is used for the authoring and editing of Systemtap scripts. It is invoked when the user starts a new file or opens a file with the .stp extension. It provides syntax highlighting, code completion, and run support.
• C Editor - This editor is available for the read only viewing of kernel source. It is invoked when the user opens a file of extension .c. Syntax highlighting is supported in this mode.
• Plain Text Editor - A plain text editor is also implemented for the user should they need it.

Menus

What follows is a comprehension list of the menu options available within the Systemtap IDE.

Edit

• Undo - Undos the previous editor related action. (Shortcut: Ctrl+Z)
• Redo - Redos the previous editor related action removed by Undo. (Shortcut: Ctrl+Y)
• Cut - Removes the selected text from the editor pane and places it in the clipboard. (Shortcut: Shift+Delete, Ctrl+X)
• Copy - Copies the selected text from the editor pane and places it in the clipboard. (Shortcut: Ctrl+Insert, Ctrl+C)
• Paste - Copies the text from the clipboard to the location of the focus in the editor pane. (Shortcut: Shift+Insert, Ctrl+V)
• Select All - Selects all text within the file in the editor that the focus is currently in. (Shortcut: Ctrl+A)
• Find/Replace - Opens the Find/Replace dialogue in which users may specify a text string to find and/or replace, with options to search forwards or backwards, the entire document or just the selected lines, and whether or not they want the following options:
  • Case sensitive
  • Wrap Search
  • Whole Word
  • Incremental
  • Regular expressions

(Shortcut: Ctrl+F)

• Find Next - Finds the next instance of the search string in the direction indicated in the Find/Replace dialogue box. (Shortcut: Ctrl+K)
• Find Previous - Finds the previous instance of the search string in the direction indicated in the Find/Replace dialogue box. (Shortcut: Shift+Ctrl+K)
• Incremental Find Next - Finds the next occurrence of the text and updates the selection after each character typed. To use Incremental Find:
  • In the text (or Java) editor, press Ctrl+J or select Edit > Incremental Find Next from the menu bar.
  • The workbench status line displays "Incremental Find:". The editor is now in the Incremental Find mode.
  • As you type, the editor finds the next occurrence of the text and updates the selection after each character typed.
  • Navigate to the next or previous match by pressing Arrow Down or Arrow Up.
  • Undo the last action within the Incremental Find mode by pressing Backspace.
  • You can leave the Incremental Find mode by pressing Esc
• Incremental Find Previous - Operates as illustrated in above bullet but in the reverse. (Shortcut: Shift+Ctrl+J)
• Select All - Selects all the text in the file that currently has the focus. (Shortcut: Ctrl+A)
• Word Completion - Completes the string being typed by using the last example typed by the user. (Shortcut: Alt+/)

Navigate

• Forward - Navigates to the next file in sequence as they were opened in the editor pane. (Shortcut: Alt+Right)
• Back - Navigates to the previous file in sequence as they were opened in the editor pane. (Shortcut: Alt+Left)
• Last Edit Location - Moves the focus to the file containing the last known edit. (Shortcut: Ctrl+Q)
• Goto Line... - Moves the focus to the specified line number, with the range of line numbers listed in the dialogue box. (Shortcut: Ctrl+L)

Run

• Run -Available when a Systemtap script is loaded into the editor, this feature runs the script. You will be prompted to run the script either locally, or on a remote machine running the systemtapgui server. If run locally and multiple Run Configurations exist for the script, you will be prompted to choose which of them to run. (Shortcut: Ctrl+F11)

Window

• New Window - This option opens a new instance of the SystemTap IDE.
• Open Perspective - This submenu lists each of the available perspectives to open:
  • Systemtap IDE
  • Other... - Lists all other perspectives available to Eclipse.
• Show Views - This submenus lists each of the available views to open:
  • Console
  • Functions
  • Kernel Source
  • Probe Alias
  • Systemtap Error Log
  • Other... - Lists all views in the application.
• Preferences - Due to the breath of this topic it is covered in another section. Please see Preferences.

Toolbars

Action Bar

• New File - This button will bring up a dialogue to create a new file. When you click ok the file will be opened in the editor window.
• Simple Run - This will compile the open script and start running it, without using a Run Configuration. All output will be sent to the console. If there is an error in the script the error will be reported in the ErrorLog.
• Run As - This compiles & runs the current script, but also searches for a Run Configuration that targets the script, and runs the script with the launch options provided by it. Use this instead of Simple Run when you want graphs to be plotted. (Shortcut: Ctrl+F11)

Preferences

This section details each of the options listed under Window->Preferences. The Preferences dialog is used to set user preferences. Its pages can be searched using the filter function. To filter by matching the page title, simply type the name of the page you are seeking in the text box in the top-left corner of the dialog, and the available pages will be presented in the list below. The filter also searches on keywords such as "appearance" and "Java".

• SystemTap - This section contains options relevant to SystemTap itself.
  • Environment Variables - Allows the user to set specific values to the following environment variables
    • LD_LIBRARY_PATH
    • PATH
    • SYSTEMTAP_TAPSET
    • SYSTEMTAP_RUNTIME
  • Graphing - Preferences related to the Graphing Perspective.
    • Refresh Delay (ms) - Milliseconds in between re-painting the graph.
    • Data Table - Preferences related to the data table.
      • Jump to the newest entry - Determines whether the scroll bar maintains its location at the newest entry.
      • Auto resize columns - Determines whether columns automatically resize to the width of the contents or whether they remain placed when moved.
      • Max data items - Maximum number of entries that the table shows, though the table itself may have more not visible.
    • Graph - Preferences related to graphs.
      • Show X/Y grid lines - Turns on and off painting of the grid lines for either axis.
      • Max data items - Number of maximum samples read from a script's output.
      • Viewable data items - Number of maximum samples plotted at any time.
      • X/Y series ticks - Specify the number of labelled number markers on either axis.
  • IDE - Preferences when loading the IDE Perspective.
    • Use stored tapset trees - Loads the stored tapset trees on initialization.
    • Editor - Preferences related to the editor.
      • Editor Background Color - Sets the color painted in the background of the editor pane.
      • Show Line Numbers - Option to turn on and off line numbers in the editor pane.
      • Code Assist
        • Use Code Assist - Turns on and off the option to use Code Assist, which is a method that predicts what you are typing based on existing probe alias and opens a window that allows you to scroll down and select and/or continue typing for refined results.
        • How Code Assist adds code - Method used in Code Assist.
          • Insert - Inserts the selected text directly into the cursor point.
          • Overwrite - Overwrites as you type.
        • Activation Delay (requires restart) - Delay in milliseconds before the code assist routines attempts to run.
        • Activation Trigger - This is the key that actives Code Assist. For SystemTap GUI it is set to the divider between tapset and probe alias.
      • Syntax Coloring - These options assign color values to identified types within the .stp and .c editors.
        • STP editor
          • Default Color - Color of normal code.
          • Keyword Color - Color of keywords.
          • Embedded C Color - Color of embedded C code.
          • Embedded Color - Color of embedded code.
          • Comment Color - Color of comments.
          • Type Color - Color of items identified as types.
          • String Color - Color of strings.
    • Kernel Source Path - Location of the directory that the kernel source code is pulled from.
      • Tapsets - Preferences related to tapsets.
        • Additional Tapsets - Allows you to specify additional tapsets scripts to be added in with every script.
  • Remote Server - Details regarding the remote machine running the systemtapgui Server
    • Host Name - IP address of the remote machine
    • Port - Port used for communication. Default is 22462.
    • User Name - User name that has sufficient permissions to transfer and run the script on the remote machine.
    • Password - Password for the above user.
    • Seconds to Save Data

Frequently Asked Questions

This section details frequently asked questions relating to the IDE Perspective.

Where can I find information regarding the Systemtap scripting language?

A: The best place to look for up to date information regarding Systemtap itself is its homepage - http://sourceware.org/systemtap/documentation.html

How do I run a script once I've written it?

A: There are two ways to run a script: -with the Simple Run button on the toolbar, which runs the script that has editor focus. It will be run without drawing graphs or any other configurations. -by launching it with a Run Configuration that targets any specified script, even one that is not in the workspace.

Read Lesson 3: Running Your Script and Menus to learn more about the ways of running Systemtap scripts.

Why does the application seem to hang when switching to the Systemtap IDE?

A: If this is the first time you are trying to open the Systemtap IDE, or have recently gotten a new version of SystemTap this is expected. In order to generate a full list of all of the available probes, and functions the GUI has to run the stap command. If there are tapsets that do not run correctly using the 'stap -up2' comamnd this takes a long time. In order to build as complete a list as possible it will use a binary filtering algorithm to figure out which tapset function is broken and generate data for all of the others.

Example SystemTap Scripts

What follows is a few SystemTap Scripts that you may use in SystemTap GUI, taken from and linked to http://sourceware.org/systemtap/documentation.html

There is also a description of the examples - http://sourceware.org/systemtap/examples/demo_script.txt

top.stp - Prints the top 20 system calls (http://sourceware.org/systemtap/examples/top.stp)

#!/usr/bin/env stap 
#
# This script continuously lists the top 20 systemcalls on the system
#

global syscalls

function print_top () {
	cnt=0
	log ("SYSCALL\t\t\t\tCOUNT")
	foreach ([name] in syscalls-) {
		printf("%-20s\t\t%5d\n",name, syscalls[name])
		if (cnt++ = 20)
			break
	}
	printf("--------------------------------------\n")
	delete syscalls
}

probe kernel.function("sys_*") {
	syscalls[probefunc()]++
}

# print top syscalls every 5 seconds
probe timer.ms(5000) {
	print_top ()

prof.stp - Simple profiling (http://sourceware.org/systemtap/examples/prof.stp)

#!/usr/bin/env stap

# This is an example of profiling a specific command or pid.
# It works by recording the time when a system call is entered
# exited. 

# Usage: prof.stp -c "top -n5"
# Will start up "top" and after 5 iterations, will exit.
#
# Usage: prof.stp -x 3323
# Will profile pid 3323 until it ^c is hit.
#

probe kernel.function("sys_*") {
	if (target() = tid())
		calltime[tid()] = gettimeofday_us()
}

probe kernel.function("sys_*").return {
	if (target() != tid())  next
	now = gettimeofday_us()
	c = calltime[tid()]
	if (!c) next
	ttime[probefunc()] <<< now - c
	delete calltime[tid()]
}

probe end {
	printf("\n")
	foreach (x in ttime)
		printf("%-20s\tcalls:%6d\tavg time (ms):%5d\ttotal(ms):%7d\n", 
			x, @count(ttime[x]), @avg(ttime[x]), @sum(ttime[x]))
}

global calltime, ttime

keyhack.stp - Modifying variables in the kernel. (http://sourceware.org/systemtap/examples/keyhack.stp)

#! /usr/bin/env stap

# This is not useful, but it demonstrates that
# Systemtap can modify variables in a running kernel.

# Usage: ./keyhack.stp -g

probe kernel.function("kbd_event") {
	# Changes 'm' to 'b' .
	if ($event_code = 50) $event_code = 48
}

probe end {
	printf("\nDONE\n")
}

kmalloc.stp - Statistics example. (http://sourceware.org/systemtap/examples/kmalloc.stp)

#! /usr/bin/env stap

# Using statistics to examine kernel memory allocations

global kmalloc

probe kernel.function("__kmalloc") { 
	kmalloc <<< $size
}

# Exit after 10 seconds
probe timer.ms(10000) { exit () }

probe end {
	printf("Count:   %d allocations\n", @count(kmalloc))
	printf("Sum:     %d Kbytes\n", @sum(kmalloc)/1000)
	printf("Average: %d bytes\n", @avg(kmalloc))
	printf("Min:     %d bytes\n", @min(kmalloc))
	printf("Max:     %d bytes\n", @max(kmalloc))	
	print("\nAllocations by size in bytes\n")
	print(@hist_log(kmalloc))
}

kmalloc2.stp - Example using arrays of statistics. (http://sourceware.org/systemtap/examples/kmalloc2.stp)

#! /usr/bin/env stap

# Using statistics and maps to examine kernel memory allocations

global kmalloc

probe kernel.function("__kmalloc") { 
	kmalloc[execname()] <<< $size
}

# Exit after 10 seconds
probe timer.ms(10000) { exit () }

probe end {
	foreach ([name] in kmalloc) {
		printf("Allocations for %s\n", name)
		printf("Count:   %d allocations\n", @count(kmalloc[name]))
		printf("Sum:     %d Kbytes\n", @sum(kmalloc[name])/1000)
		printf("Average: %d bytes\n", @avg(kmalloc[name]))
		printf("Min:     %d bytes\n", @min(kmalloc[name]))
		printf("Max:     %d bytes\n", @max(kmalloc[name]))	
		print("\nAllocations by size in bytes\n")
		print(@hist_log(kmalloc[name]))
		printf("-------------------------------------------------------\n\n");
	}
}

Guide to Regular Expressions

Regular expressions are used for SystemTap graphs to interpret information that it recieves from a script's output.

Basic regex (taken from "http://java.sun.com/j2se/1.5.0/docs/api/ ) -

 Summary of regular-expression constructs

Construct 	Matches
 
Characters
x 	The character x
\\ 	The backslash character
\0n 	The character with octal value 0n (0 <= n <= 7)
\0nn 	The character with octal value 0nn (0 <= n <= 7)
\0mnn 	The character with octal value 0mnn (0 <= m <= 3, 0 <= n <= 7)
\xhh 	The character with hexadecimal value 0xhh
\uhhhh 	The character with hexadecimal value 0xhhhh
\t 	The tab character ('\u0009')
\n 	The newline (line feed) character ('\u000A')
\r 	The carriage-return character ('\u000D')
\f 	The form-feed character ('\u000C')
\a 	The alert (bell) character ('\u0007')
\e 	The escape character ('\u001B')
\cx 	The control character corresponding to x
 
Character classes
[abc] 	a, b, or c (simple class)
[^abc] 	Any character except a, b, or c (negation)
[a-zA-Z] 	a through z or A through Z, inclusive (range)
[a-d[m-p]] 	a through d, or m through p: [a-dm-p] (union)
[a-z&&[def]] 	d, e, or f (intersection)
[a-z&&[^bc]] 	a through z, except for b and c: [ad-z] (subtraction)
[a-z&&[^m-p]] 	a through z, and not m through p: [a-lq-z](subtraction)
 
Predefined character classes
. 	Any character (may or may not match line terminators)
\d 	A digit: [0-9]
\D 	A non-digit: [^0-9]
\s 	A whitespace character: [ \t\n\x0B\f\r]
\S 	A non-whitespace character: [^\s]
\w 	A word character: [a-zA-Z_0-9]
\W 	A non-word character: [^\w]
 
POSIX character classes (US-ASCII only)
\p{Lower} 	A lower-case alphabetic character: [a-z]
\p{Upper} 	An upper-case alphabetic character:[A-Z]
\p{ASCII} 	All ASCII:[\x00-\x7F]
\p{Alpha} 	An alphabetic character:[\p{Lower}\p{Upper}]
\p{Digit} 	A decimal digit: [0-9]
\p{Alnum} 	An alphanumeric character:[\p{Alpha}\p{Digit}]
\p{Punct} 	Punctuation: One of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
\p{Graph} 	A visible character: [\p{Alnum}\p{Punct}]
\p{Print} 	A printable character: [\p{Graph}\x20]
\p{Blank} 	A space or a tab: [ \t]
\p{Cntrl} 	A control character: [\x00-\x1F\x7F]
\p{XDigit} 	A hexadecimal digit: [0-9a-fA-F]
\p{Space} 	A whitespace character: [ \t\n\x0B\f\r]
 
java.lang.Character classes (simple java character type)
\p{javaLowerCase} 	Equivalent to java.lang.Character.isLowerCase()
\p{javaUpperCase} 	Equivalent to java.lang.Character.isUpperCase()
\p{javaWhitespace} 	Equivalent to java.lang.Character.isWhitespace()
\p{javaMirrored} 	Equivalent to java.lang.Character.isMirrored()
 
Classes for Unicode blocks and categories
\p{InGreek} 	A character in the Greek block (simple block)
\p{Lu} 	An uppercase letter (simple category)
\p{Sc} 	A currency symbol
\P{InGreek} 	Any character except one in the Greek block (negation)
[\p{L}&&[^\p{Lu}]]  	Any letter except an uppercase letter (subtraction)
 
Boundary matchers
^ 	The beginning of a line
$ 	The end of a line
\b 	A word boundary
\B 	A non-word boundary
\A 	The beginning of the input
\G 	The end of the previous match
\Z 	The end of the input but for the final terminator, if any
\z 	The end of the input
 
Greedy quantifiers
X? 	X, once or not at all
X* 	X, zero or more times
X+ 	X, one or more times
X{n} 	X, exactly n times
X{n,} 	X, at least n times
X{n,m} 	X, at least n but not more than m times
 
Reluctant quantifiers
X?? 	X, once or not at all
X*? 	X, zero or more times
X+? 	X, one or more times
X{n}? 	X, exactly n times
X{n,}? 	X, at least n times
X{n,m}? 	X, at least n but not more than m times
 
Possessive quantifiers
X?+ 	X, once or not at all
X*+ 	X, zero or more times
X++ 	X, one or more times
X{n}+ 	X, exactly n times
X{n,}+ 	X, at least n times
X{n,m}+ 	X, at least n but not more than m times
 
Logical operators
XY 	X followed by Y
X|Y 	Either X or Y
(X) 	X, as a capturing group
 
Back references
\n 	Whatever the nth capturing group matched
 
Quotation
\ 	Nothing, but quotes the following character
\Q 	Nothing, but quotes all characters until \E
\E 	Nothing, but ends quoting started by \Q
 
Special constructs (non-capturing)
(?:X) 	X, as a non-capturing group
(?idmsux-idmsux)  	Nothing, but turns match flags on - off
(?idmsux-idmsux:X)   	X, as a non-capturing group with the given flags on - off
(?=X) 	X, via zero-width positive lookahead
(?!X) 	X, via zero-width negative lookahead
(?<=X) 	X, via zero-width positive lookbehind
(?<!X) 	X, via zero-width negative lookbehind
(?>X) 	X, as an independent, non-capturing group                                                                    
Match a position that is not a word boundary.

Information regarding regular expressions can be found all over the Internet. A few examples have been listed below:

• http://www.javascriptkit.com/javatutors/redev.shtml
• http://www.zytrax.com/tech/web/regex.htm

SystemTap Graphing

Getting Started

Introduction

TWhen a SystemTap script is launched under a Run Configuration with the Graph the output of this script run option enabled, it is possible to plot its output in graph form. (If you are unfamiliar with how to load or write a SystemTap script, please review the IDE Tutorial in the SystemTap IDE User's Guide.) A variety of graph types are available for plotting the output of Systemtap scripts:

• Scatter Graph
• Line Graph
• Area Graph
• Bar Graph
• Pie Graph

These graph types can be instantiated as soon as a script is run by creating them in a script's Run Configuration, or after a script run has been initiated by using the Create Graph wizard in the Graph Selector View. Please refer to Lesson 3: Running Your Script for information on how to use Run Configurations and regular expressions to capture data from a script's output, and how to plot that data in graphs.

Graphing Tutorial

This tutorial picks up from where Lesson 3: Running Your Script left off, and provides more information on graph configuration. If you followed that tutorial already, click Run As -> Run Configurations... and view the configuration you created earlier. If not, follow that tutorial now, and return here when you have finished.

Once the Run Configuration menu is opened with your configuration selected, click on the Graphing tab. At the bottom of the window, a section labelled Graph Set # displays a list of all graphs created for the currently-selected Regular Expression (the # in "Graph Set #" indicates which expression is chosen). There is also a set of buttons related to graph configuration:

• Add... - Brings up the Create Graph wizard to allow you to create a new graph.
• Duplicate - When one of the graphs in the list of entries is selected (by clicking on it), this button creates a duplicate of that graph.
• Edit... - When one of the graphs in the list of entries is selected, this button allows you to edit the settings of that graph with the Edit Graph wizard.
• Remove - Removes the currently selected graph.

Note: Each regular expression you create has its own Graph Set. When creating & editing graphs, the only graphs that will be visible in the Graph Set # section will be the graphs used for the expression currently selected in the Regular Expression combo box.

Create Graph Wizard

The Create Graph (or Edit Graph Wizard is used when creating a new graph or editing an existing one. It allows you to set all properties of a graph, such as its title, graph type, and series data.

• Graph Type - The type of graph to draw. See Graph Types for the list of graph types available, and an explanation of each.
• Title - The title the graph will have when displayed in the Graph Selector View.
• X Series - The class of data that will provide the graph's X series data. This data can come from one of the capturing groups of your regular expression (which appear as their user-defined titles), or, if set to Row Num, the graph will plot a new data point for every line/row of output that is matched by your regular expression.
• Y Series 0 - The class of data that will provide the graph's Y series data, relative to the X axis. A capturing group from your regular expression must be used for this.
• Y Series 1, 2, 3... - If you have included more than one capturing group in your regular expression, you may choose to provide your graph with multiple Y series data, each of which will be plotted relative to the same X series axis you specified earlier. They will appear as a seperate set of data points on your graph, labelled by title. When you do not want to include any extra Y series values, choose NA.

Note that two different series may not share the same data selection.

Concepts

Data Table

The Data Table is a view that is always available when running a script with the chart option. It intitially starts empty and populates as it recieves information parsed from the console originally from the script. The Data Table is a visualization of the data structure that every other graph type spawns from.

The columns of the Data Table can be resized at will by dragging the seperation lines between column headers. In addition, the information in the data table can be saved and loaded without having to re-run the script. Furthermore, the user can populate the Data Table without having to run a script if a script's output has been saved into a file, see Opening Script Output.Right clicking the Data Table provides a menu with the following options:

• Format as... - Depending on which column the user right clicked, this allows the user to format that column's data as one of the following:
  • Unformated - No formating is applied.
  • String
  • Date
  • Double
  • Hex
  • Octal
  • Binary
• Add Filter... - Adds a filter to the Data Table.
• Remove Filter... - Removes a filter from the Data Table.
• Manually Resize - Allows the user to resize column width without bouncing back to the minimum based on the largest value.

Data Graph Types

Currently there are five graph types built into SystemTap Plug-in.

These graph types can be used instantiated by using the Create Graph wizard once the Data Table is populated. The Graphing Perspective can support multiple graphs from one Data Table. Each are updated and can be viewed by navigating through the tabs shown at the top of each Graph Set while in the Graph Selector View.

Scatter Graph

A Scatter Graph uses Cartesian coordinates to show the relation of two or more quantitative variables.

Creating a Scatter Graph will prompt the user for three values. The first is the title and is fully cosmetic. The option to turn on and off the title is given to the user, and its off by default. The second option is X Series; here you select which series of data you want to plot along the X Axis. The third option is the Y Series; here you select which series of data you want to plot along the Y Axis. Optionally you can add extra Y Series; here you select another series of data you want to plot along the Y Axis.

Line Graph

The Line Graph is traditionally ideal for horozontally depicting non-cumulative data over, say, time.

Creating a Line Graph will prompt the user for three values. The first is the title and is fully cosmetic. The option to turn on and off the title is given to the user, and its off by default. The second option is X Series; here you select which series of data you want to plot along the X Axis. The third option is the Y Series; here you select which series of data you want to plot along the Y Axis. Optionally you can add extra Y Series; here you select another series of data you want to plot along the Y Axis.

Area Graph

The Area Graph is traditionally ideal for horozontally depicting non-cumulative data over, say, time.

Creating an Area Graph will prompt the user for three values. The first is the title and is fully cosmetic. The option to turn on and off the title is given to the user, and its off by default. The second option is X Series; here you select which series of data you want to plot along the X Axis. The third option is the Y Series; here you select which series of data you want to plot along the Y Axis. Optionally you can add extra Y Series; here you select another series of data you want to plot along the Y Axis.

Bar Graph

The Bar Graph typically uses bars to show frequencies or values.

Creating a Bar Graph will prompt the user for three values. The first is the title and is fully cosmetic. The option to turn on and off the title is given to the user, and its off by default. The second option is X Series; here you select which series of data you want to plot along the X Axis. The third option is the Y Series; here you select which series of data you want to plot along the Y Axis. Optionally you can add extra Y Series; here you select another series of data you want to plot along the Y Axis.

Pie Graph

The Pie Graph uses a pie to show percentages.

Creating a Pie Graph will prompt the user for three values. The first is the title and is fully cosmetic. The option to turn on and off the title is given to the user, and its off by default. The second option is X Series; here you select the series to be used as labels of each area of the pie chart. The third option is the Y Series; here you select which series of data will be used to calculate the percentages of each area of the pie chart. Optionally you can add extra Y Series, each of which will draw another Pie containing that series' data.

Filters

Filters are screens that you can set on your Data Table to limit the quantity of samples you wish to work with. Filters are very often used in restricting the sheer amount of information available to make data tables and graphs more substantial to people.

The following options exist to work with Filters in the Data Table:

• Add Filter... - Adds a Filter to the Data Table.
• Remove Filter... - Removes a Filter from the Data Table.

When you select Add Filter you see the following dialogue box:

What follows is a breakdown of each Filter type:

• Match Filter - This filter removes everything from the data set where the selected column's

value does not match the chosen value.

Result:

• Range Filter - This filter removes everything from the data set that is not included in the

selected range of values.

Result:

• Sort Filter - This filter will sort the data set based on the selected column. No data will be

removed.

Result:

• Unique Filter - This filter will remove all merge all entries in that data set that share a common

value in the selected column. Data for the removed rows will be aggregated together based on the chosen aggregation method.

• • Average Aggregate - This aggregate will return the average of all the items consolidated.
  • Count Aggregate - This aggregate will return the number of items consolidated.
  • Max Aggregate - This aggregate will return the maximum value of all the items consolidated.
  • Min Aggregate - This aggregate will return the minimum value of all the items consolidated.
  • Sum Aggregate - This aggregate will return the sum of all the items consolidated.

Graph Interaction

Each of the Graph Types have similar options that can be used in the Graphical Perspective:

• Zoom Bar - The graph types that use axes (Scatter, Line, Area, and Bar graphs) support this control. This enables the user to reduce the number of X-axis values that are plotted on the graph, to effectively zoom in and out on the graph's display. Sliding the bar left reduces the range of the X-axis that is plotted (zoom in), and sliding right increases it (zoom out).

Use Preferences to configure other graph settings, such as grid lines and viewable data items.

Reference

This section of the Graphing User Guide is intended to provide the following resources should you have any very specific questions regarding the graphing in general or one of the options associated with it. It also contains example scripts and FAQs.

Frequently Asked Questions

This section details frequently asked questions relating to graphing script output.

Can I make more than one graph at one time? Do they update concurrently?

A: Yes and yes. In the Run Configuration of the script you want to create plots of, you can create multiple graphs at once. Also, after a graph-enabled script is run, you may create several additional graphs by clicking the Graph Wizard button in the Graph Selector View.

Can I increase the rate at which SystemTap GUI update the graph?

A: Yes. By default it is set to once per second, but you can increase this in Preferences.

Is there any way to save or export the graph images?

A: Currently, no. This feature will be restored in a newer version of Systemtap IDE.

What if I want to preserve this data for manipulation later?

A: This feature will be restored in a newer version of Systemtap IDE.

Copyright Information

© Copyright IBM Corporation 2012
© Copyright Red Hat, Inc. 2009
© Copyright Phil Muldoon <pkmuldoon@picobot.org> 2006.
This documentation is licensed under the Eclipse Public License v. 1.0.