Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Linux Tools Project/LTTngLegacy/User Guide"

(How to use LTTng to debug a problem)
(Getting Started)
Line 42: Line 42:
 
If not already installed, you can download and install the LTTng plug-ins from the Eclipse update site (search for LTTng under Linux Tools).  
 
If not already installed, you can download and install the LTTng plug-ins from the Eclipse update site (search for LTTng under Linux Tools).  
  
== LTTng 2.0 ==
 
 
LTTng 2.0 viewing is supported out of the box with Eclipse Juno and later. LTTng 2.0 is supported also in Ubuntu 12.04.
 
 
To install LTTng on previous versions of Ubuntu or another distribution, see the website [http://lttng.org] for more information.
 
 
=== Installing the LTTng 2.0 parsing library ===
 
 
No steps are needed, the parser is included in the eclipse package.
 
 
=== Downloading sample traces ===
 
 
To download a sample trace, go to [http://lttng.org/download]. At the bottom of the page there is a link to some sample LTTng 2.0 kernel traces.
 
 
=== Installing the control agent ===
 
 
LTTng 2.0 is controlled via ssh and RSE in eclipse. This is installed with Juno, no additional steps are needed.
 
 
=== Installation Verification ===
 
 
Here are the quick steps to verify that your installation is functional. If needed, refer to the [[#LTTng Perspective | LTTng Perspective]] section for clarifications.
 
 
* Start Eclipse
 
* Open the LTTng perspective
 
* Create an LTTng project
 
** Right-click in the Project view and select "New Project"
 
** Enter the name of your project (e.g. "MyLTTngProject")
 
** The project will be created. It will contain 2 empty folders: "Traces" and "Experiments"
 
* Import a sample trace
 
** Right-click on the newly created project and select "Import Trace"
 
** Navigate to the sample LTTng trace that you want to visualize
 
** Note: you have to drill down into the trace directory to import the trace correctly (yes, we know that the Import function needs to be re-written...)
 
** The newly imported trace should appear under the Traces folder
 
** Select the LTTng 2.0 kernel as the trace type.
 
* Visualize the trace
 
** Expand the Traces folder
 
** Double-click on the trace
 
** The trace should load and the views be populated
 
 
== LTTng 0.X (Classic) ==
 
  
 
The rest of this section provides instructions on how to complete the LTTng for Eclipse installation and verify that it is correct.
 
The rest of this section provides instructions on how to complete the LTTng for Eclipse installation and verify that it is correct.
Line 91: Line 51:
  
  
=== Installing the LTTng parsing library ===
+
== Installing the LTTng parsing library ==
  
 
If you are using Ubuntu, you might want to use the LTTng PPA and avoid the detailed library installation. See the instalaltion video at [http://lttng.org/eclipse] for more information.
 
If you are using Ubuntu, you might want to use the LTTng PPA and avoid the detailed library installation. See the instalaltion video at [http://lttng.org/eclipse] for more information.
Line 189: Line 149:
  
  
=== Downloading Sample Traces  ===
+
== Downloading Sample Traces  ==
  
 
To verify that your installation is functional, you can download sample traces from GitHub. Each trace as its own characteristics and complexity. The size given refers to the size of the trace once "untarred", the actual tar size should be ~10 times smaller. To download a trace: click on the selected trace and then click on the "Downloads" button.
 
To verify that your installation is functional, you can download sample traces from GitHub. Each trace as its own characteristics and complexity. The size given refers to the size of the trace once "untarred", the actual tar size should be ~10 times smaller. To download a trace: click on the selected trace and then click on the "Downloads" button.
Line 204: Line 164:
  
  
=== Installation of the LTTng Agent ===
+
== Installation of the LTTng Agent ==
  
 
For the trace control version v0.x an LTTng agent has to be running on the remote system. To download the source code from the relevant git repository, use '''git clone git://git.dorsal.polymtl.ca/git/lttng-agent.git''' from a Linux command shell. Then go to directory '''lttng-agent/manual''' in the cloned git repository and type '''make'''. This will generate the '''LTTng Agent Manual''' in HTML format. Please follow the instructions in this manual for the manual installation of the agent.  
 
For the trace control version v0.x an LTTng agent has to be running on the remote system. To download the source code from the relevant git repository, use '''git clone git://git.dorsal.polymtl.ca/git/lttng-agent.git''' from a Linux command shell. Then go to directory '''lttng-agent/manual''' in the cloned git repository and type '''make'''. This will generate the '''LTTng Agent Manual''' in HTML format. Please follow the instructions in this manual for the manual installation of the agent.  
Line 210: Line 170:
 
To install an agent on Ubuntu using a PPA package follow the instruction of the web site [https://launchpad.net/~lttng/+archive/ppa LTTng PPA]. However, this version won't always include the latest bugfixes. To get the latest version, the user should download the source code, compile and install the agent manually.
 
To install an agent on Ubuntu using a PPA package follow the instruction of the web site [https://launchpad.net/~lttng/+archive/ppa LTTng PPA]. However, this version won't always include the latest bugfixes. To get the latest version, the user should download the source code, compile and install the agent manually.
  
=== Installation Verification ===
+
== Installation Verification ==
  
 
Here are the quick steps to verify that your installation is functional. If needed, refer to the [[#LTTng Perspective | LTTng Perspective]] section for clarifications.
 
Here are the quick steps to verify that your installation is functional. If needed, refer to the [[#LTTng Perspective | LTTng Perspective]] section for clarifications.

Revision as of 16:36, 8 June 2012

{{#eclipseproject:technology.linux-distros}}

Linux Tools
Website
Download
Community
Mailing ListForumsIRCmattermost
Issues
OpenHelp WantedBug Day
Contribute
Browse Source

Introduction

LTTng 2.0 (Linux Trace Toolkit, next generation 2.0) is a highly efficient tracing tool for Linux that can be used to track down kernel and application performance issues as well as debug problems involving multiple concurrent processes and threads. It consists of a set of kernel modules, a daemon - to collect the raw tracing data - and a set of tools to control, visualize and analyze the generated data.

The LTTng plug-in for Eclipse provides an Eclipse integration for the control, fetching and visualization of LTTng traces. It also provides the foundation for user-defined analysis tools.

At present, the LTTng Eclipse plug-in for Eclipse supports the following kernel-oriented analyses:

  • Control Flow - to visualize processes state transition
  • Resources - to visualize system resources state transitions
  • Statistics - to provide simple statistics on event occurences

For more information about LTTng, refer to the project site


About Tracing

Tracing is a troubleshooting technique used to understand the behavior of an instrumented application by collecting information on its execution path. A tracer is the software used for tracing. Tracing can be used to debug a wide range of bugs that are otherwise extremely challenging. These include, for example, performance problems in complex parallel systems or real-time systems.

Tracing is similar to logging: it consists in recording events that happen in a system at selected execution locations. However, compared to logging, it is generally aimed at developers and it usually records low-level events at a high rate. Tracers can typically generate thousands of events per second. The generated traces can easily contain millions of events and have sizes from many megabytes to tens of gigabytes. Tracers must therefore be optimized to handle a lot of data while having a small impact on the system.

Traces may include events from the operating system kernel (IRQ handler entry/exit, system call entry/exit, scheduling activity, network activity, etc). They can also consists of application events (a.k.a UST - User Space Tracing) or a mix of the two.

For the maximum level of detail, tracing events may be viewed like a log file. However, trace analyzers and viewers are available to produce useful information from the raw data. These programs must be specially designed to handle quickly the enormous amount of data a trace may contain.

In the case of LTTng, low tracing overhead is achieved by instrumenting the Linux kernel with a set of custom patches. The same set of patches can be used for tracing both the Linux kernel and user applications (UST).


Traces and Experiments

In the scope of the LTTng plug-in for Eclipse, a trace is essentially a set of time-ordered LTTng events. The LTTng application can read these traces and provide a number of standard views to analyze their contents.

An experiment consists in an arbitrary number of aggregated LTTng traces for purpose of correlation. In the degenerate case, an experiment can consist of a single trace.

The experiment provides a unified, time-ordered stream of the individual LTTng trace events.

Getting Started

The LTTng plug-in for Eclipse is used to visualize and analyze kernel traces produced by the LTTng tracer. Controlling the recording and fetching of an LTTng trace from Eclipse is currently a work in progress therefore, currently the plug-in can only be used to visualize existing LTTng traces. To record your own trace, you will need to install the LTTng kernel patches and the LTTv application (refer to the LTTng site for more information on how to record and fetch traces).

If not already installed, you can download and install the LTTng plug-ins from the Eclipse update site (search for LTTng under Linux Tools).


The rest of this section provides instructions on how to complete the LTTng for Eclipse installation and verify that it is correct.


Installing the LTTng parsing library

If you are using Ubuntu, you might want to use the LTTng PPA and avoid the detailed library installation. See the instalaltion video at [1] for more information.

The LTTng plug-in for Eclipse requires the installation of a third-party library, liblttngtrace, that is used to parse the binary LTTng traces. For the first Eclipse release, it was decided not to re-implemented that library in Java but to simply integrate it using JNI. As a consequence, you need to download, compile and install the parsing library.

This trace parsing library is not delivered as part of Eclipse (licensing stuff) and has to be downloaded from the LTTng site, compiled and then installed on your Linux system.

Currently, the Eclipse plug-in supports the following version of the LTTng trace format: 2.3, 2.5 and 2.6. Depending on which trace version(s) you wish to analyze, you will need to download the appropriate snapshot from the corresponding git repository (pick the "snapshot" link next to the latest commit):

LTTng trace format 2.3 parser
LTTng trace format 2.5 parser
LTTng trace format 2.6 parser

If you wish to stream LTTng traces (format 2.6 only), you will need to install the following library instead:

LTTng trace format 2.6 parser with live trace read


Build and install the snapshot(s) you downloaded. If you wish to install more than one version of the library, you will have to build them in format version sequence (e.g. build 2.6 last). For a plain library installation (in /usr/local/lib), you can execute the following commands for each version you wish to install:

$ cd liblttngtrace-2.n
$ ./autogen.sh --with-jni-interface --without-lttv && make
# make install

You will end up with something like this (the loader acts as a factory to pre-analyze the trace and load the correct parser library):

$ ls -l1 /usr/local/lib
total 1048
-rwxr-xr-x. 1 root root 210300 2010-06-07 11:24 liblttvtraceread-2.5.so
-rwxr-xr-x. 1 root root 211052 2010-06-07 11:25 liblttvtraceread-2.6.so
-rw-r--r--. 1 root root 316040 2010-06-07 11:25 liblttvtraceread.a
-rwxr-xr-x. 1 root root   1175 2010-06-07 11:25 liblttvtraceread.la
-rwxr-xr-x. 1 root root  93963 2010-06-07 11:24 liblttvtraceread_loader-2.5.so
-rwxr-xr-x. 1 root root  93963 2010-06-07 11:25 liblttvtraceread_loader-2.6.so
-rw-r--r--. 1 root root 125442 2010-06-07 11:25 liblttvtraceread_loader.a
-rwxr-xr-x. 1 root root   1224 2010-06-07 11:25 liblttvtraceread_loader.la
lrwxrwxrwx. 1 root root     30 2010-06-07 11:25 liblttvtraceread_loader.so -> liblttvtraceread_loader-2.6.so
lrwxrwxrwx. 1 root root     23 2010-06-07 11:25 liblttvtraceread.so -> liblttvtraceread-2.6.so

A number of configuration options are available to fine tune the library builds:

$ ./configure -- help
Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
...
  <more options>

Once the parsing libraries are installed, it is necessary to let Eclipse know where they sit otherwise the LTTng plugin won't be able to parse the traces. You basically have 3 options at this point:

a) Eclipse.ini

In the eclipse.ini (in the Eclipse folder), add the following line to the VM arguments:

-Djava.library.path=/usr/local/lib

b) LD_LIBRARY_PATH

Alternatively, you can specify the libraries location using LD_LIBRARY_PATH on the command line/script when you start your Eclipse.

> LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH" ./eclipse

c) Project trace library path

If you are customizing your own LTTng parsing libraries, you might have a number of variants sitting on your system and you might want to have the freedom to dynamically associate a parser to a project. The LTTng plugin allows you to do so (see Project View).

However, in this case, the RUNPATH of the library loader (liblttvtraceread_loader-X.Y.so) needs to be doctored a little so it can find the actual parsing libraries. This is done using the patchelf utility (available from here). Note that the library loader has to reside in the same directory as the parsing library (make install takes care of that).

> cd <wherever>
> patchelf --set-rpath '.:$ORIGIN' liblttvtraceread_loader-X.Y.so


Alternatively, you can tweak the library's makefile and set the RUNPATH directly.


Downloading Sample Traces

To verify that your installation is functional, you can download sample traces from GitHub. Each trace as its own characteristics and complexity. The size given refers to the size of the trace once "untarred", the actual tar size should be ~10 times smaller. To download a trace: click on the selected trace and then click on the "Downloads" button.

The following LTTng traces are provided as samples:

trace-2.5-15316 (360K) Very small trace (format 2.5 - 15316 events) made from a single LTTng marker. Recorded with a VirtualBox system with 1 processor.

trace-2.5-15471 (365K) Very small trace (format 2.5 - 15471 events) made from a single LTTng marker. Recorded with a VirtualBox system with 1 processor.

trace-2.6-22K (1.1M) Small trace (format 2.6 - 22K events).

(More sample traces to come)


Installation of the LTTng Agent

For the trace control version v0.x an LTTng agent has to be running on the remote system. To download the source code from the relevant git repository, use git clone git://git.dorsal.polymtl.ca/git/lttng-agent.git from a Linux command shell. Then go to directory lttng-agent/manual in the cloned git repository and type make. This will generate the LTTng Agent Manual in HTML format. Please follow the instructions in this manual for the manual installation of the agent.

To install an agent on Ubuntu using a PPA package follow the instruction of the web site LTTng PPA. However, this version won't always include the latest bugfixes. To get the latest version, the user should download the source code, compile and install the agent manually.

Installation Verification

Here are the quick steps to verify that your installation is functional. If needed, refer to the LTTng Perspective section for clarifications.

  • Start Eclipse
  • Open the LTTng perspective
  • Create an LTTng project
    • Right-click in the Project view and select "New Project"
    • Enter the name of your project (e.g. "MyLTTngProject")
    • The project will be created. It will contain 2 empty folders: "Traces" and "Experiments"
  • Import a sample trace
    • Right-click on the newly created project and select "Import Trace"
    • Navigate to the sample LTTng trace that you want to visualize
    • Note: you have to drill down into the trace directory to import the trace correctly (yes, we know that the Import function needs to be re-written...)
    • The newly imported trace should appear under the Traces folder
  • Visualize the trace
    • Expand the Traces folder
    • Double-click on the trace
    • The trace should load and the views be populated


Refer to LTTng Perspective for detailed description of the views and their usage.

LTTng Perspective

The LTTng perspective groups the LTTng specific views, namely:

The views are synchronized i.e. selecting an event, a timestamp, a time range, etc will update the other views accordingly.


LTTngPerspective.png


The perspective can be opened from the Eclipse Open Perspective dialog (Window > Open Perspective... > Other).


LTTngPerspectiveOpen.png


Project View

The project view is the standard Eclipse Project Explorer. LTTng being a tracing project, it is shown with a small "T" decorator in the upper right of the project folder icon.

ProjectExplorer.png

Tracing projects have two sub-folders: Traces which holds the individual traces, and Experiments which holds sets of traces that we want to correlate.


Creating a Project

A new LTTng project can be created using the LTTng Project dialog (File > New > Project... or the context menu)


LTTngProjectNewWizard.png


LTTngProjectNewDialog.png

At this point, you can press Finish and complete the project creation or you can press Next to provide a project-specific parser library path.


LTTngProjectNewDialog-2.png


Once the project is created, you can modify the parser library path at any time by selecting the project's property page (right-click on the project, Properties).


LTTngProjectProperty.png

Importing Traces in a Project

The Traces folder holds the set of traces available for experiments. Traces are imported in the LTTng project using an import wizard triggered by a context menu.

Note: An LTTng trace is actually a composite of multiple channel traces grouped under a folder. It is the folder that has to be imported.


LTTngTraceImportDialog.png


The Experiment folder holds the set of experiments of the project. Experiments are created from the Experiment context menu and are configured from the experiment's context menu.


LTTngTraceSelectDialog.png

Drag and Drop

Traces can be also be imported to a project by dragging from another tracing project and dropping to the project's trace folder. The trace will be copied and the trace type will be set.

Any resource can be dragged and dropped from a non-tracing project, and any file or folder can be dragged from an external tool, into a tracing project's trace folder. The resource will be copied or imported as a new trace, however the trace type will be unknown and need to be set manually by the user.

It is also possible to drop a trace, resource, file or folder into an existing experiment. If the item does not already exist as a trace in the project's trace folder, it will first be copied or imported, then the trace will be added to the experiment.

Events View

The Events view shows the basic trace data in chronological order in a tabular format.

When opened, the Events view displays the events of the currently selected trace or experiment.

The header displays the current trace (or experiment) name. The table displays the following fields:

  • Timestamp: the event timestamp
  • Trace: the trace that holds the event (e.g. experiments typically hold multiple traces)
  • Marker: the event type and localization
  • Content: the raw event content

The first row of the table is the header row a.k.a. the Search and Filter row.

EventsView.png

The highlighted event is the current event and is synchronized with the other views. If you select another event, the other views will be synchronized accordingly.

Searching and Filtering

Searching and filtering of events in the table can be performed by entering matching conditions in one or multiple columns in the header row (the first row below the column header).

To toggle between searching and filtering, click on the 'search' (TmfEventSearch.gif) or 'filter' (TmfEventFilter.gif) icon in the header row's left margin, or right-click on the header row and select Show Filter Bar or Show Search Bar in the context menu.

To apply a matching condition to a specific column, click on the column's header row cell, type in a regular expression and press the ENTER key. You can also enter a simple text string and it will be automatically be replaced with a 'contains' regular expression.

When matching conditions are applied to two or more columns, all conditions must be met for the event to match (i.e. 'and' behavior).

To clear all matching conditions in the header row, press the DEL key.

Searching

When a searching condition is applied to the header row, the table will select the next matching event starting from the top currently displayed event. Wrapping will occur if there is no match until the end of the trace.

All matching events will have a 'search match' icon in their left margin. Non-matching events will be dimmed.

TmfEvents-Search.png

Pressing the ENTER key will search and select the next matching event. Pressing the SHIFT-ENTER key will search and select the previous matching event. Wrapping will occur in both directions.

Press ESC to cancel an ongoing search.

Press DEL to clear the header row and reset all events to normal.

Filtering

When a filtering condition is entered in the head row, the table will clear all events and fill itself with matching events as they are found from the beginning of the trace.

A status row will be displayed before and after the matching events, dynamically showing how many matching events were found and how many events were processed so far. Once the filtering is completed, the status row icon in the left margin will change from a 'stop' to a 'filter' icon.

TmfEvents-Filter.png

Press ESC to stop an ongoing filtering. In this case the status row icon will remain as a 'stop' icon to indicate that not all events were processed.

Press DEL or right-click on the table and select Clear Filters from the context menu to clear the header row and remove the filtering. All trace events will be now shown in the table. Note that the currently selected event will remain selected even after the filter is removed.

You can also search on the subset of filtered events by toggling the header row to the Search Bar while a filter is applied. Searching and filtering conditions are independent of each other.

Bookmarking

Any event of interest can be tagged with a bookmark.

To add a bookmark, double-click the left margin next to an event, or right-click the margin and select Add bookmark.... Alternatively use the Edit > Add bookmark... menu. Edit the bookmark description as desired and press OK.

The bookmark will be displayed in the left margin, and hovering the mouse over the bookmark icon will display the description in a tooltip.

The bookmark will be added to the Bookmarks view. In this view the bookmark description can be edited, and the bookmark can be deleted. Double-clicking the bookmark or selecting Go to from its context menu will open the trace or experiment and go directly to the event that was bookmarked.

To remove a bookmark, double-click its icon, select Remove Bookmark from the left margin context menu, or select Delete from the Bookmarks view.

Bookmarks.png

Histogram View

The Histogram View displays the trace events distribution with respect to time. When streaming a trace, this view is dynamically updated as the events are received.


HistogramView.png


On the top left, there are two data controls:

  • Current Event (sec): Displays the timestamp of the currently selected event
  • Window Span (sec): Displays the current time range window size

Both control can be used to modify their respective value. After validation, the other controls and views will be synchronized and updated accordingly.


The large histogram, at the bottom, shows the event distribution over the whole trace or set of traces. It also has a smaller semi-transparent window, with a cross-hair, that shows the currently selected time range window. The time range window can be zoomed in/out by using the mouse wheel. It can also be selected by the mouse and dragged to another region of the trace.

The smaller histogram, on top right, corresponds to the currently selected time range window, a sub-range of the event set. Its size can also be zoomed in/out using the mouse wheel.

The x-axis of each histogram corresponds to the events timestamps. The timestamp of the first and the last event of the respective ranges is displayed. The y-axis of each histogram shows the minimum/maximum number of events in the corresponding histogram bars.

The dashed vertical magenta bar, on the right, shows the position of the last event. The dashed vertical red bar shows the relative position of the currently selected event. The current event can be changed by clicking on the histogram.

Hovering the mouse over an histogram bar pops up an information window that displays the start/end time of the corresponding bar as well as the number of events it represents.

In each histogram, the following keys are handled:

  • Left: Moves the current event to the previous non-empty bar
  • Right: Moves the current event to the next non-empty bar
  • Home: Displays the current event to the first histogram bar
  • End: Displays the current event to the last non-empty histogram bar


Control Flow View

The Control Flow View is applicable to kernel trace data. It displays the list processes in tabular format, sorted by PID, as well as their state transitions over time in a Gantt chart format.


ControlFlowView.png


The tool bar options from left to right are:

  • Home: reset the time window to the whole trace range.
  • Next Event: sets the current event to the next event of the selected process.
  • Previous Event: sets the current event to the previous event of the selected process.
  • Next Process: sets the selected process to the next down in the list.
  • Previous Process: sets the selected process to the next up in the list.
  • Zoom In: zooms in the trace (i.e. shortens the current time window) at the current location. This can also be accomplished by using the mouse wheel in the graphical part.
  • Zoom Out: zooms out the trace (i.e. widens the current time window) at the current location. This can also be accomplished by using the mouse wheel in the graphical part.
  • Filter: display only the processes that have a state transition in the current window.


The process table displays the following fields:

  • Process: the process name if know (UNNAMED otherwise)
  • Brand:
  • PID: the process ID
  • TGID:
  • PPID: the parent PID
  • CPU: the CPU/core the process is running on
  • Birth sec/nsec: the birth time of the process (0 if unknown) - seconds and nanoseconds
  • TRACE: the name of the trace that holds this process


The graphical part shows the state of the corresponding process a la Gantt: the color represents the process state and the length of the bar represents the time span during which the process was in that state. The header corresponds to a time axis. Note that depending on the scale or zooming factor, some pixels might represent more (in fact many, many more) than just one state.

Here's a list of possible colors and their corresponding process state:


LTTngControlFlowColorLegend.png


  • MODE_UNKNOWN - Process state is unknown: there is not enough information in the trace to determine the state. This usually occurs for processes already alive when tracing started and the system is unable to infer the original state from the following trace events.
  • USER_MODE - Running in user mode: the process is executing user source code or library calls (in user space).
  • SYSCALL - Running in system call: the process is doing a system call to the kernel and the running mode switched from process limited rights to super user mode.
  • IRQ - Servicing an IRQ: the currently running process is interrupted by an IRQ. As the IRQ does not change the currently running process (on some architectures it uses the same stack as the process), the IRQ state is shown in the state of the process. IRQ can be nested: a higher priority interrupt can interrupt a lower priority interrupt.
  • TRAP - Running in a trap: the kernel is running a trap that services a fault. The most frequent trap is the memory page fault trap, called every time a page is missing from physical memory.
  • WAIT - Waiting on I/O: the process is waiting for an I/O operation to complete before it can continue its execution.
  • UNNAMED - Mode unnamed: see MODE_UNKNOWN
  • WAIT_FORK - Waiting for fork: the process has just been created by its parent and is waiting for first scheduling.
  • WAIT_CPU - Waiting for CPU: the process is ready to run but waiting to get the CPU (a schedule in event).
  • EXIT - Process has exited: the process has exited, but still has the control of the CPU. It may happend if it has some tasks to do in the exit system call.
  • ZOMBIE - Zombie process: the process in in zombie state. This state happens when a process exits and then waits for the parent to wait for it (wait() or waitpid()).
  • SOFTIRQ - Running a soft IRQ handler: a SoftIRQ is normally triggered by an interrupt that whishes to have some work done very soon, but not "now". Doing the long part of a computation in a softirq that will be run just after the IRQ handler exits permits to do work while interrupts are enabled, without increasing the system latency.


Mouse Control

  • Mouse Wheel
    • Forward - Zoom In.
    • Backward - Zoom Out.
  • Mouse Drag
    • States Area - Moves the time window in the direction of the drag keeping current scale.
    • Time Scale Area - Changes the time scale to the time range from the start to the end of the drag action.
  • Mouse Hover
    • Hovering the mouse selection over a state will trigger a tooltip display with additional information, including the start and end times as well as the name of the state associated to the color.


LTTngProcessStateInfo.png


Resources View

The Resources View is applicable to kernel trace data. It displays the list system resources in tree format, grouped by trace, as well as their state transitions over time in a Gantt chart format.

ResourcesView.png

The tool bar options from left to right are:

  • Home: reset the time window to the whole trace range.
  • Next Event: sets the current event to the next event of the selected process.
  • Previous Event: sets the current event to the previous event of the selected process.
  • Next Process: sets the selected process to the next down in the list.
  • Previous Process: sets the selected process to the next up in the list.
  • Zoom In: zooms in the trace (i.e. shortens the current time window) at the current location. This can also be accomplished by using the mouse wheel in the graphical part.
  • Zoom Out: zooms out the trace (i.e. widens the current time window) at the current location. This can also be accomplished by using the mouse wheel in the graphical part.
  • Filter: display only the processes that have a state transition in the current window.


The resource tree displays the system resources whose state transitions were collected in the trace.

The graphical part shows the state of the corresponding resources a la Gantt: the color represents the resource state and the length of the bar represents the time span during which the resource was in that state. The header corresponds to a time axis. Note that depending on the scale or zooming factor, some pixels might represent more (in fact many, many more) than just one state.

Here's a list of possible colors and their corresponding process state:


LTTngResourcesColorLegend.png


Mouse Control

See Control Flow View


Statistics View

The Statistics View displays the various event counters that are collected when analyzing a trace. The data is organized per trace.


StatisticsView.png


Latency View

The Latency View is applicable to kernel trace data. The Latency View initiates the analysis of an LTTng trace to determine the elapsed time between 2 related events (the latency), and then to plot the distribution of the latencies.

LatencyView.png

The analysis is done to the current active time range of the LTTng perspective. The Latency View consists of 2 different graphs. The left graph shows the latency in respect to the event time (start event). The right graph shows the latency distribution.

Latency vs Time Graph

This graph shows the latency in respect of the event time of the first event of the 2 related events. The x-axis of the graph corresponds to the start event time in seconds. The y-axis shows the latency in milli-seconds. Hovering the mouse over an plotted item pops up an information window that displays the start/end event time of the corresponding area in the horizontal direction, the start/end latecny time of the corresponding area in vertical direction as well as the number of latencies that fall into this square.

LatencyGraphHover.png

Latency Distribution

This graph shows the latency distribution. The number of latency values that fall in the corresponding latency range are displayed. The x-axis of the distribution graph shows the latency in milli-seconds. The y-axis corresponds to the number of events (latency occurances). Hovering the mouse over a histogram bar pops up an information window that displays the number of events as well as the number of events it represents.

LatencyDistributionHover.png

Mouse Control

The mouse control is only applicaple for the Latency Distribution graph.

  • Mouse Wheel
    • Forward - Zoom in (in y-axix only).
    • Backward - Zoom out (in y-axis only).

After zooming in the zoom factor is displayed in brackets besides the title Latency Distribution. Additionally, besides the y-axis a warning icon is drawn to indicate that the maximum height is not displayed. Hovering over this icon pops up an information window about warning icon.

LatencyDistributionZoom.png

Tool Bar Menu

The latency view comes with serveral tool bar buttons and menu items for changing the bar width as well as for listing, adding and deleting of matching events.

LatencyToolbar.png

  • Increase bar width/height: To increase the bar width of the Latency Distribution and the square width and hight of the Latency vs Time graph, select the button Increase bar width/height. By default, the size is 2 pixels. The maximum size is 16.
  • Decrease bar width/height: To decrease the bar width of the Latency Distribution and the square width and hight of the Latency vs Time graph, select the button Decrease bar width/height. By default, the size is 2 pixels. The minimum size is 1.
  • List matching events: To display the current list of event pairs to be matched, select the button List matching events. A new dialog display will open with the list of matching events. To reset to the default list of matchin pairs, press button Reset to default pairs.

LatencyListMatchingDialog.png

  • Add matching events: To add a new pair of events to be matched, select the button Add matching events. A new display will open for selecting a new pair of events. First select the start event from the provided list on the left. The select an end event on the right side. After that, press Add. This will add a new entry to the list in the lower part of the display. After closing the dialog box, the trace will be parsed again using the new list of event pairs.

LatencyAddMatchingDialog.png

  • Delete matching events: To delete a pair of events to be matched, select the button Delete matching events. A new display will open for selecting of pairs of events. First select the event pair to be deleted, then press button Delete. After closing the dialog box, the trace will be parsed again using the new list of event pairs.

LatencyDeleteMatchingDialog.png

Note that the list of event pairs are presisted in the workspace and will be available after restart of Eclipse.

LTTng Tracer Control

The LTTng Tracer toolchains version v0.x requires and LTTng agent running on the remote host. The functions to control the LTTng tracer (e.g. start, pause, resume), either locally or remotely, are available from the Remote Systems view (See Remote Systems View, an RSE view).

LTTngControlRsePerspective.png

Remote Systems View

The LTTng tracer is controlled by the UI through an LTTng agent running on the tracer system.

Create a Remote Connection

A new connection to a remote system can be created using the new connection wizard of the Remote System Explorer. For this, a toolbar button exists on the top left corner of the Remote System view. Alternatively, this wizard can be started using File -> New -> Other ... -> Remote System Explorer -> Connection).

LTTngNewConnectionWizard1.png

For LTTng control, select connection type LTTng and press Next to define the connection.

LTTngNewConnectionWizard2.png

The Host name holds the IP address or DNS name of the remote system. The Connection name is the name to be displayed in the Remote Systems view. The Description is optional. Unless the user wants to use a specific Profile, the default profile is sufficient. The checkbox Verfiy host name enable or disables the verification of the remote system.

By default, when pressing "Finish" the new connection contains the following sub systems:

  • Files: This subsystem allows the user to browse the file system on the remote system. Amongst others the user has the possibility to open, create files, download or upload files.
  • Processes: This subsystem allows the user to browse and control processes on the remote system.
  • Shells: This subsystem allows the user to open command shells to issue commands on the remote system.
  • Terminals: This subsystem allows the user to open command terminals to issue commands on the remote system.
  • LTTng Tracing: With this subsystem the user can control the LTTng tracer on the remote system. It supports the control of kernel and userspace traces.

The user has the possibility to configure or omit the subsystems Files, Processes, Shells and Terminals by pressing "Next" and deselecting the relevant subsystem.

LTTngNewConnectionWizard3.png

LTTngNewConnectionWizard4.png

LTTngNewConnectionWizard5.png

LTTngNewConnectionWizard6.png


After finishing the new connection the Remote System view will be updated:

LTTngRemoteSystemsNewConnection.png

General

All actions are on elements in the Remote Systems view are accessible through context sensitive menu. By selecting a tree element and pressing the right mouse button a context sensitive menu is shown. Some of the actions are also available through toolbar buttons and toolbar menu.

Connect to Remote Host

The connection to the remote host can be established by selecting the connection in the tree, click the right mouse button and select Connect.

LTTngConnectToRemoteSystem.png

A login dialog will be opened, where the user needs to specify the user name and password.

LTTngLoginDialog.png

After logging in the LTTng Trace Control will retrieve LTTng Remote System information. First it will read the supported providers. The providers will be one or both of the following: "kernel" for kernel tracing and "ust" for user space tracing. After that the trace controls reads the currently available targets. For kernel tracing, there is only one target with ID "0". For user space tracing each target is the corresponding process ID of a user space application with user space tracing enabled. Without any traces created the Remote System view will look like this (after expanding the tree):

LTTngRemoteSystemsWithLTTng.png

Configure Kernel Markers

The marker of a given target can be configured by selecting the kernel target "0", clicking the right mouse button and selecting the Configure Markers menu item of the context sensitive menu.

LTTngConfigureMarkersAction.png

A new dialog box will show for the configuration of the relevant markers. All markers are listed in a table. The first column contains a checkbox to enable or disable a marker. The second column shows the name of the marker and all other columns display marker details.

LTTngConfigureMarkersDialog.png

Create a New Kernel Trace

A new trace can be created by selecting the kernel target "0", clicking the right mouse button and selecting the New Trace menu item of the context sensitive menu.

LTTngCreateNewTraceAction.png

A new dialog box will show for the configuration of the relevant parameter.

LTTngCreateNewTraceDialog.png

  • Name: holds the name of the trace.
  • Transport type: it is set by default to "relay" and cannot be changed.
  • Trace Location:
    • Target: means that the trace is stored on the remote system where the trace is collected.
    • Host: means that the trace is transfered to the local host where the eclipse application is running.
    • Trace path: holds the path where the trace shall be stored. The Browse button is only active for Local and can be used to open a dialog box to select the trace path directory.
  • Tracing project: defines the LTTng Tracing project where a streaming (host) trace will be added at trace start.
  • Number of lttd threads: defines the number of threads to be used by the ltt daemon during tracing.
  • Append: can be used to append the new trace to an existing trace.
  • Trace mode: can be either Normal or Flight Recorder. In normal mode, the subbuffers are constantly written while the trace is started. In flight recorder mode, only the latest subbuffers are written when the trace is stopped.

If all inputs are valid the Finish button will be active and can be used to create the trace. The LTTng trace control will setup the trace using the connection to the remote agent.

After creating a new trace, the Remote Systems view will be updated. When selecting the newly created trace in the tree, the buttons to start, stop and configure the trace will be enabled.

LTTngRemoteSystemConfiguredTrace.png

Configure a Trace and Trace Channels

Some of the trace parameters can be changed if the trace has not been previously started. A trace can be configured by selecting the relevant kernel trace, clicking the right mouse button and selecting the "Configure Trace" menu item of the context sensitive menu.

LTTngConfigureTraceAction.png

A new dialog box with a wizard page will show for the configuration of the relevant parameter. For explanation of the parameters see #Create_a_New_Kernel_Trace

LTTngConfigureTraceDialog.png

After entering valid parameters the Next and Finish buttons will be active. To finalize the configuration of the trace select the Finish button. To configure channel parameters select the Next button. A new wizard page will show for the configuration of channel parameters. For provider kernel the following display will show:

LTTngConfigureChannelsKernelDialog.png

All channels can be configured at once by clicking the right mouse button in the table and selecting one of the menu items of the context sensitive menu.

LTTngConfigureChannelsActions.png

For provider ust only the channel parameter Channel Timer can be configured and the following display will show after selecting the Next button of the Trace configuration page.

LTTngConfigureChannelsUstDialog.png

Please note that channel parameters cannot be retrieved from the LTTng agent. Therefore they cannot be displayed in the table unless they have been previously configured through Eclipse. However, this information will be lost after closing Eclipse. A question mark will indicate that the relevant value is not known and it will be ignored after selecting the Finish button.

  • Name column: holds the name of the channel.
  • Enabled column: a checkbox indicates whether the channel is enabled or not.
  • Buffer Override column: a checkbox indicates whether channel buffer overwrite is enabled or not.
  • Number of Sub Buffers column: holds the number of sub-buffers of the channel.
  • Sub Buffer size column: holds the size of the sub-buffer of the channel (in bytes).
  • Channel Timer column: holds the channel timer value (in milliseconds) for periodically flushing all collected data for the channel, whether the sub-buffers are full or not. The provider ust only supports a single timer for all channels, traces and targets it is monitoring.

Please note that the Configure Trace item of the context sensitive menu of a trace can be used to display the trace and channel parameters at any time. However, they can only be changed if the trace has not been previously started.

Start a Trace

A configured or paused trace can be started by selecting the kernel trace, clicking the right mouse button and selecting the Start menu item of the context sensitive menu. Alternatively, the start button in the toolbar can be used to start the trace. If the trace location is host (streaming trace), it will be automatically added to the tracing project that was selected at trace creation.

LTTngStartTraceAction.png

Pause a Trace

A started trace can be paused by selecting the relevant trace, clicking the right mouse button and selecting the Pause menu item of the context sensitive menu. Alternatively, the pause button in the toolbar can be used to pause the trace.

LTTngPauseTraceAction.png

Stop a Trace

A non-started trace can be stopped by selecting the relevant trace, clicking the right mouse button and selecting the Stop menu item of the context sensitive menu. Alternatively, the stop button in the toolbar can be used to stop the trace. This will destroy the trace on the remote system, all allocated memory will be deallocated and all relevant trace files will be finalized. This step is need to be able to import a trace into the LTTng Perspective.

LTTngStopTraceAction.png

Import a Trace in the LTTng Perspective

A streaming or stopped trace can be imported to the LTTng Perspective for viewing by selecting the relevant trace, clicking the right mouse button and selecting the Import To Project menu item of the context sensitive menu. A new dialog box will show to select the LTTng Project.

LTTngImportTraceAction.png

For local trace path (streamed to the host):

LTTngImportLocalTrace.png

For remote trace path (stored on the target):

LTTngImportRemoteTrace.png

  • Import to project: All existing LTTng tracing projects in the workspace are displayed in a table and the user has to select the destination. If no projects are available the user has to create a new project first (see #Project_View)
  • Trace name: specifies the name that the imported trace shall have in the LTTng project.
  • Link only: For a trace streamed to the host the user has the option to create a symbolic link to the trace path or, if the trace is stopped, to copy the trace into the LTTng project.

After pressing button Import the trace will be stored in the selected project. For a trace stored on the target, it will be downloaded from the remote system.

Delete a Trace

A stopped trace can be deleted by selecting the relevant trace, clicking the right mouse button and selecting the Delete menu item of the context sensitive menu. A new dialog box will show for the user to confirm that the selected trace should be deleted.

LTTngDeleteTraceAction.png

LTTngDeleteTraceConfirm.png

Property View

The Property view displays the properties of the selected resource in the tree. For traces it displays all configuration parameters.

LTTngTraceProperties.png

Limitations

The current implementation is under heavy development and some important features are still missing. Here is a list of the known bugs and missing features that are being worked on.

Performance

The current version takes 3-5 more time to fully parse a trace than the finely optimized C version (LTTv). However, once a trace/experiment is selected, the user has immediate feedback and can start navigating and visualizing the trace while the parsing takes place in the background.

Tracer control limitations (for LTTng Tracer Toolchain v0.x)

Due to errors in the LTTng remote agent, controlling of user space traces is currently not possible.

The LTTng agent doesn't allow to retrieve the trace state, i.e. whether a trace is started or stopped. Therefore, when reconnecting with Eclipse, the LTTng control is not able to display the correct state. Currently, the trace is considered as started.

For kernel traces, it's currently not possible to configure the channels. This will be added in the next release.

64-bits and non-x86 host architectures

The parser libraries have not been extensively tested for 64-bits and not tested at all on non-x86 architectures. The code is believed to work on a broad range of architectures but this cannot be certified for now.

Searches and Filters

There is currently no way to filter events following some criteria or to dynamically sort the different fields.

CTF Parser

Note: This feature is not available yet but will be delivered soon.

A Common Trace Format (CTF) parser is included in TMF, An example extension is supplied for LTTNG 2.0 User space tracing.

To load it, one needs to simply select the proper parser. (right click on a trace, Select Trace Type->Lttng 2.0->User Space Trace) as shown below

Ctf navigator.png

Then open the trace by right clicking on the the trace and clicking "Open".

Ctf open.png

The viewers that currently work with CTF traces are the events view and histogram.

Ctf opened no state system.png


References

Back to the top