Explore traces and spans from the Wavefront UI.

The Wavefront tracing UI enables exploration of your apps and services. Once your application is instrumented for tracing, you can examine traces, spans, and RED metrics, moving easily from one browser page to the next.

This page explains how you can examine trace data from our UI. You’ll learn how to query for traces, examine details, find potential problems, and more.

Choose Your Starting Point

Wavefront offers the following dashboards and browsers to view information on your applications and services and you can navigate from one to another to gather more information:

tracing user interfaces To choose your starting point for visualizing traces:
  1. In your web browser, go to your Wavefront cluster and log in.
  2. From the taskbar click Applications : tracing menu

Application Map (Beta)

The application map not only gives you an overview of how the applications and services are linked, it lets you, focus on a specific service, view Request, Error, and Duration (RED) metrics for each service and the edges in the application. You can also view traces for the services and edges and drill down from the application map.

application map

Let’s walk through the following scenario to get a quick overview of the application map.

ActionResult
Step 1: Search for applications
Click Tag, select application > beachshirts, and click Search.

You can search for applications by applying one or more filters, such as the application name, cluster, shard, or any other application tag.
Search for the beachshirts application using tags
Step 2: Hover over a service
Hover over the styling service of the beachshirts application. It highlights how the styling service communicates with the other services in the application.
Hover over the styling service
Step 3: Click on a service
Click on the styling service. Now, you can:
  • View Request, Error, and Duration (RED) metrics of the specific service.
  • View how a specific service communicates with the other services in an application when you click Focus on service.
  • Navigate to the Service dashboard when you click View Service dashboard.
  • Navigate to the Tracing browser when you click View traces for service.
  • See the components used by the service. The styling service uses the OpenTracing, Java, Dropwizard, and Jersey components.
Popup when you click on a service
Step 4: Focus on a service
Click on a service and then click Focus on service to focus on the styling service of the beachshirts application.

This will help you focus on a specific service when you have more than 10 services in your application.
Focus on the styling service
Step 5: Hover over an edge
Hover over the edge between the styling and shopping service. You see that they send requests to each other.

When you hover over an edge (the arrow that goes from one service to the other). It highlights the direction of the requests between the two services.
Edges are bidirectional if the two services send requests to each other.
Hover over the styling service
Step 6: Click on an edge
When you click on the edge between the styling and printing service, you can:
  • View Request, Error, and Duration (RED) metrics for the specific edge.
  • Navigate to the Tracing browser when you click View traces for this edge.
The pop up when you click an edge that is bidirectional

Application Status

Go to the Applications page for a top-level status overview of your instrumented applications.

app status

On the Applications page, you can:

  • View the status of all instrumented applications, or search for a particular application by applying filters.
    • Apply one or more filters to select application name, cluster, or shard and click Apply.
  • Inspect RED metrics to obtain a status summary for an application:
    • The total number of requests that are represented by the application’s spans.
    • The percentage of the application’s spans that contain errors.
    • The span duration (in milliseconds) at the 95th percentile across the application.
  • Sort the displayed applications by name or by a RED metric.
  • Click an application name for an overview of its services.

Service Status

When you select an application, you get an overview of its services.

app services

On the page for a particular application, you can:

  • Examine the services in the application, or search for a particular service by applying filters.
  • View the inventory of component frameworks that each service is built on.
  • Inspect RED metrics to obtain a status summary for a service:
    • The total number of requests that are represented by the service’s spans.
    • The percentage of the service’s spans that contain errors.
    • The span duration (in milliseconds) at the 95th percentile across the service.
  • Drill down from a service box:

Service Dashboard

When you click on a service’s Details, you can examine a dashboard of metrics to identify potential hot spots, and then drill down to the Traces browser.

Explore the Default Service Dashboard

The default, read-only dashboard for a service lets you explore that service, however, you can’t make changes to the dashboard.

examine services

On the dashboard for a particular service, you can:

  • Select the time and time zone in the task bar to customize the chart time properties.
  • Use the Jump To pulldown to select a dashboard section:
    • Select Overview to examine the RED metrics that are derived from all of the spans for the service. These metrics reflect the health of the service.
    • Select an individual component to examine metrics for just that component of the service. A component could be an instrumented framework (e.g., Jersey) or the runtime system (e.g., JVM).
    • Select System if your environment uses Telegraf and you want to view CPU usage, memory usage, and disk usage.
  • Filter the metrics based on the cluster, shard, or source.
  • Select Detailed View or Summarized View to change the level of detail for charts.
  • Examine the TopK charts to find out which operations are potential hot spots. The bars represent operations that execute in this component of the service.
  • Navigate to the Traces browser.
    • Click a bar on a histogram.
    • Click a bar on a TopK chart.
    • Click the vertical ellipsis in the top right of any chart, click Traces, and click a service.

Custom Service Dashboard

The standard dashboard for services is read-only. To create a customizable copy:

  1. Click Clone from the ellipsis menu.
  2. In the cloned dashboard, add your own charts or customize the RED metrics charts.

After you save the clone, you can find it by name from the Dashboards menu of the task bar, and you can use it to drill down to the Traces browser.

Troubleshooting

Don’t see data on the default service dashboard?

Starting with the next release (release 2020-26.x), the span.kind filter is introduced to the default service dashboard. As a result, if the spans from your OpenTracing application don’t have the span.kind point tag, you don’t see data on the default service dashboard. The OpenTracing SDK and Wavefront proxy versions listed below adds the span.kind tag to the spans by default. Therefore, use the recommended versions to see data on the default service dashboard.

SDK or Proxy Version
Wavefront proxy 7.0 or later
Java OpenTracing SDK v2.0 or later
Go OpenTracing SDK v0.9.0 or later
Python OpenTracing SDK v2.0.0 or later
C# OpenTracing SDK v2.0.0 or later
Java Tracing Agent v1.1.0 or later

Tracing Browser

In the Traces browser, you can explore the context and the details of your application’s traces.

  • Navigate from the service’s page to display traces for operations you selected.
  • Select Applications > Traces from the task bar to display an empty page that you populate by querying.

explore trace browser

From the Traces browser, you can:

  • Query for traces and view the query results in the traces list.
  • Select a trace in the list and:
  • Examine a trace’s percentile indicator to see how the trace’s duration compares to the durations of the other listed traces. You can toggle the panel size for the traces list, service map, or trace details.
  • Easily analyze your traces hierarchy and RED metrics by clicking the expand or restore icon.
    • Click the expand icon to expand the RED metrics and view them next to the trace list, and click the same icon again to restore to the default view. partial expand red metrics
    • Click the expand icon to expand the trace hierarchy and view them next to the trace list, and click the same icon again to restore to the default view. partial expand trace hierachy

Query for a List of Traces

In the Traces browser, you can use the query builder or query editor to query for traces that include spans for a particular operation.

You can view the results in a traces list.

explore traces list

Here’s how to get started:

  1. Add an Operation or Trace ID. See Get Started with Trace Queries for details.
  2. Click Add Filter and specify a filter to limit the results.
  3. (Optional) Advanced users can use Query Editor to limit the scope even further.

You can use the trace list to:

  • Sort the returned traces according to different criteria.
  • Click a trace to view its context in the service map panel and its details in the trace details panel.
  • View a trace’s percentile indicator to see how the trace’s duration compares to the durations of the other listed traces – and potentially find outliers.

Investigate the Service Map for a Trace

In the Traces browser, use the service map to investigate the services that contribute spans to a selected trace.

explore service map

In the service map panel, you can:

  • View the service dependencies, and follow the flow of request calls from service to service.
  • Click on a service to display the RED metrics that reflect the health of the service:
    • Request count, error count, and trace duration at the 95th percentile, over all traces (root spans) that originate in the service.
    • Line charts that indicate the general contour of the RED metrics.
    • Fine-tune the time window of the RED metrics charts, and see events.
  • Navigate to the selected service’s dashboard for more service details.
  • Scroll the service map to zoom in or out, and re-center the service map.
  • Look for nodes that are grayed out. These represent services that contribute spans to at least one trace in the list, but not to the trace that is currently selected.

Watch this video to see how a service map can help you pinpoint a performance bottleneck in a microservices architecture:

introduction to service maps

Examine Trace Details

In the Traces browser, use the trace details panel to examine the spans that belong to a selected trace. Some of these spans may represent operations executed by other services.

trace details

In the trace details panel, you can:

  • Examine the call hierarchy for a selected trace.
  • Click any span to expand and see more detail.
  • Explore the trace’s critical path. This is an end-to-end sequence of blocking spans, where each blocking span must complete before the trace can complete.

Drill Down Into Spans and View Metrics and Span Logs

In the trace details panel, you can view details about a particular span in the trace. If that span came from another service, you can navigate to the dashboard for that service.

trace span details

Span details include:

  • Application tags. These are the application, service, cluster, and shard, as selected by the trace query.
  • Other tags, including the trace ID.
  • A clickable link to the corresponding dashboard that lets you examine the metrics associated with the call.

If your spans have been instrumented to show span logs, you can click Logs to fetch the logs and drill down to examine them. We don’t show span logs by default for better browser performance.

A Closer Look at Critical Paths

The trace details panel uses an orange line to show the critical path through a trace. You can think of the critical path as the end-to-end combination of spans that are the most blocking. These spans represent the sequence of operations that must complete before the trace itself can complete.

Analyzing the critical path of a trace can help you determine which operations took the most time, and can help you decide which operations to try to optimize.

We use the following rules to determine which spans to include in a critical path (in order of applicability):

  1. Ignore asynchronous spans (spans tagged with followFrom).
  2. Ignore spans that end before their parent starts.
  3. If a child span continues after its parent, ignore that continuation period.
  4. Choose longer spans over shorter siblings.
  5. Choose later spans over earlier spans.
  6. Choose child spans instead of their parent spans.