Explore traces and spans from the Wavefront UI.

The Wavefront tracing UI enables exploration of your apps and services. Once your application is instrumented for tracing, you can examine traces, spans, and RED metrics, moving easily from one browser page to the next.

This page explains how you can examine trace data from our UI. You’ll learn how to query for traces, examine details, find potential problems, and more.

Choose Your Starting Point

Wavefront offers the following dashboards and browsers to view information on your applications and services and you can navigate from one to another to gather more information:

tracing user interfaces To choose your starting point for visualizing traces:
  1. In your web browser, go to your Wavefront cluster and log in.
  2. From the taskbar, click Applications : tracing menu

Application Status

It is important to get an overview of the services and applications that send data to Wavefront, understand the health of each service, and troubleshoot when your applications or services run into issues. You can get an overview and see the overall health of each application using the application map, table view, and grid view.

Application Map

The application map gives you an overview of how the applications and services are linked, lets you focus on a specific service, view Request, Error, and Duration (RED) metrics for each service and the tracing traffic in the application. You can also, drill down to the Service Dashboard and Tracing Browser.

This video highlights the application map features and settings:

See the application map:

  1. In your web browser, go to your Wavefront cluster and log in.
  2. From the taskbar, click Applications > Application Status and select the Application Map icon ( icon to click to get the application map view )
  3. Optionally, use icon to click to get the link to get a link and share what you’re seeing right now (NON-LIVE display) with other users.

Let’s walk through the following scenario to get a quick overview of the application map.

ActionResult
Step 1: Search for applications
Click Tag, select application > beachshirts, and click Search.

You can search for applications by applying one or more filters, such as the application name, cluster, shard, or any other application tag.
Search for the beachshirts application using tags
Step 2: Customize the application map view
You can customize how you see your applications and services on the application map using the settings icon.
  • Service layout: View the services in the default, concentric, circle, or grid layout. Chose the layout that helps you understand how your services are linked.
  • Isolated Services: These are services that don't interact with any other services or applications.
  • External Service: These are external applications or services, such as AWS services or Database services, your application communicates with. You can group these services too. If you want to group all the database services and view it as a single external service, select Group External Services.
  • Show Service Labels: When you have many services in an application, the service names on the application map look cluttered. To get a clear view of your application and services, disable the Show Service Labels option.
Step 3: Hover over a service
Hover over the styling service of the beachshirts application. It highlights how the styling service communicates with the other services in the application.
Hover over the styling service
Step 4: Click on a service
Click on the styling service. Now, you can:
  • View Request, Error, and Duration (RED) metrics of the specific service.
  • View how a specific service communicates with the other services in an application when you click Focus on service.
  • Navigate to the Service Dashboard when you click View Service Dashboard.
  • Navigate to the Traces Browser when you click View traces for service.
  • See the components used by the service. The styling service uses the OpenTracing, Java, Dropwizard, and Jersey components.
Popup when you click on a service
Step 5: Focus on a service
Click on a service and then click Focus on service to focus on the styling service of the beachshirts application.

This will help you focus on a specific service when you have many services in your application.
Focus on the styling service
Step 6: Hover over a tracing traffic
Hover over the tracing traffic between the styling and shopping service. You see that they send requests to each other.

When you hover over a tracing traffic (the arrow that goes from one service to the other). It highlights the direction of the requests between the two services.
Tracing traffic is bidirectional if the two services send requests to each other.
Hover over the styling service
Step 7: Click on a tracing traffic
When you click on the tracing traffic between the styling and printing service, you can:
  • View Request, Error, and Duration (RED) metrics for the specific edge.
  • Navigate to the Traces Browser when you click View traces for this traffic.
The pop up when you click a tracing traffic that is bidirectional
Step 8: Update the legend
Click the settings icon and select Error or Duration. These settings can be configured by each user and apply to the table view, and grid view too.
  • Error Percentage: Update the legend to highlight the data based on the error percentage. Select Error % from the dropdown and customize the values. The values need to be in ascending order.
  • Duration: Update the legend to highlight the data based on the duration. Select Duration from the dropdown menu and customize the values. The values need to be in ascending order and in milliseconds.
Shows the settings to update the legend for the error %. You need to select error % from the drop down and then add the values in ascending order.

Table View

View the list of applications and services. You can see the Request, Error, and Duration (RED) metrics at a glance and sort the data.

See the table view:

  1. In your web browser, go to your Wavefront cluster and log in.
  2. From the taskbar, click Applications > Application Status and select the Table View icon ( icon to click to get the table view )

the image shows the table view of all the applications that send data to Wavefront. It has helpers to show you what to do with each UI section. For example, how to filter applications or services, change the table settings or the legend settings, and how to change back to the application map view or the grid view

Using the table view, you can:

  • Examine the applications and services, or search for a particular application or service by applying filters.
  • Click the name of the service to drill down to the Service Dashboard.
  • Sort data:
    • Sort the application and service names alphabetically.
    • Sort the table in the ascending or descending order of the RED metrics.
  • See the change (Δ value) in the RED metrics based on the time you selected for Compare.
    For example, if you select week ago from the Compare drop-down, the Δ value indicate the change in RED metrics since the data was recorded a week ago.
    shows the compare option on the table view. The drop down has the values, off (if selected doesn't show the change in value), 2 hours ago, day ago, week ago, and month ago.
  • Inspect RED metrics to obtain a status summary of a service.
    Table Data Description
    Request Rate The request rate of the service.
    Δ Request Rate The difference between:
    • The current request rate
    • The request rate at the Compare option time.
    This difference is also shown as a percentage.
    Error % The percentage of the service's spans that contain errors.
    Δ Error % The difference between:
    • The current error percentage
    • The error percentage at the Compare option time.
    Duration (P95) The span duration at the 95th percentile across the service.
    Δ Duration (P95) The difference between:
    • The current duration (P95)
    • The duration (P95) at the Compare option time.
    This difference is also shown as a percentage.
    Component See the components used by the service. For example, the beachshirts application's styling service uses the OpenTracing, Java, Dropwizard, and Jersey components.
  • Update the legend by clicking the settings icon. These settings can be configured by each user and apply to the application map, and grid view too.
    Error Percentage
    Update the legend to highlight the data based on the error percentage. Select Error % from the dropdown and customize the values. The values need to be in ascending order.
    Shows the settings to update the legend for the error %. You need to select error % from the drop down and then add the values in ascending order.
    Duration
    Update the legend to highlight the data based on the duration. Select Duration from the dropdown menu and customize the values. The values need to be in ascending order and in milliseconds.
    Shows the settings to update the legend for the duration. You need to select duration from the drop down and then add the values in ascending order.
  • Click the settings icon to customize the table view:
    • Group the services by the application or ungroup the services.
    • Add or remove columns by selecting or deselecting items from the table settings options.
    Shows the settings to customize the table view. Select or deselect the settings to customize the table.
  • Use icon to click to get the link to get a link and share what you’re seeing right now (NON-LIVE display) with other users.

Grid View

When you select an application, you get an overview of its services.

See the grid view:

  1. In your web browser, go to your Wavefront cluster and log in.
  2. From the taskbar, click Applications > Application Status and select the Grid View icon ( icon to click to get the table view )

Shows how the offline traces look once you upload the JSOn file that has the imported trace details.

On the page for a particular application, you can:

  • Examine the services in the application, or search for a particular service by applying filters.
  • View the inventory of component frameworks that each service is built on.
  • Inspect RED metrics to obtain a status summary for a service:
    • The request rate of the service.
    • The percentage of the service’s spans that contain errors.
    • The span duration at the 95th percentile across the service.
  • Drill down from a service box:
  • Update the legend by clicking the settings icon. These settings can be configured by each user and apply to the application map, and table view too.
    Error Percentage
    Update the legend to highlight the data based on the error percentage. Select Error % from the dropdown and customize the values. The values need to be in ascending order.
    Shows the settings to update the legend for the error %. You need to select error % from the drop down and then add the values in ascending order.
    Duration
    Update the legend to highlight the data based on the duration. Select Duration from the dropdown menu and customize the values. The values need to be in ascending order and in milliseconds.
    Shows the settings to update the legend for the duration. You need to select duration from the drop down and then add the values in ascending order.

Service Dashboard

Use the Service Dashboard identify potential hot spots, and then drill down to the Traces Browser.

See the Service Dashboard:

  • Option 1:
    1. In your web browser, go to your Wavefront cluster and log in.
    2. From the taskbar, click Applications > Service Dashboard.
  • Option 2: Drill down to the Service Dashboard from the application map, table view, or grid view.

Explore the Default Service Dashboard

The default, read-only dashboard for a service lets you explore that service, however, you can’t make changes to the dashboard.

examine services

On the dashboard for a particular service, you can:

  • Select the time and time zone in the task bar to customize the chart time properties.
  • Use the Jump To pulldown to select a dashboard section:
    • Select Overview to examine the RED metrics that are derived from all of the spans for the service. These metrics reflect the health of the service.
    • Select an individual component to examine metrics for just that component of the service. A component could be an instrumented framework (e.g., Jersey) or the runtime system (e.g., JVM).
    • Select System if your environment uses Telegraf and you want to view CPU usage, memory usage, and disk usage.
  • Filter the metrics based on the cluster, shard, or source.
  • Select Detailed View or Summarized View to change the level of detail for charts.
  • Examine the TopK charts to find out which operations are potential hot spots. The bars represent operations that execute in this component of the service.
  • Use icon to click to get the link to get a link and share what you’re seeing right now (NON-LIVE display) with other users.
  • Navigate to the Traces Browser.
    • Click a bar on a histogram.
    • Select a region of the histogram chart and click Search Traces to view the traces for the selected duration.
    • Click a bar on a TopK chart.
    • Click the vertical ellipsis in the top right of any chart, click Traces, and click a service.

Custom Service Dashboard

The standard dashboard for services is read-only. To create a customizable copy:

  1. Click Clone from the ellipsis menu.
  2. In the cloned dashboard, add your own charts or customize the RED metrics charts.

After you save the clone, you can find it by name from the Dashboards menu of the task bar, and you can use it to drill down to the Traces Browser.

Save Charts in the Service Dashboard

View queries used in the charts of the default Service Dashboard and save these charts to your dashboard.

  1. Click the chart name to view the chart in edit mode and to view the query used in the chart.
  2. Click Save.
  3. Save the chart to a dashboard:
    • To save to an existing dashboard, start typing the name of the dashboard, select a dashboard, and click Insert.
    • Click Save to New Dashboard, enter the dashboard name and URL, and click Create. Specify only the URL string; do not include https://.
  4. When the target dashboard opens in edit mode, click and drag the chart to the location of your choice and click Save at the top.
Save the chart to a dashboard

Troubleshooting

Don’t see RED metrics or see incorrect RED metrics on your charts?

Starting with release 2020-26.x, the span.kind filter is introduced to the default Service Dashboard. As a result, if the spans from your OpenTracing application don’t have the span.kind point tag, the RED metrics you see on the default Service Dashboard will be incorrect, or you will not see RED metrics on your charts.

The OpenTracing SDK and Wavefront proxy versions listed below add the span.kind tag to the spans. Use the recommended versions to see accurate data on the default Service Dashboard.

SDK or Proxy Version
Wavefront proxy 7.0 or later
Java OpenTracing SDK v2.1.1 or later
Go OpenTracing SDK v0.9.0 or later
Python OpenTracing SDK v2.0.0 or later
C# OpenTracing SDK v2.0.0 or later
Java Tracing Agent v1.2.0 or later

Traces Browser

In the Traces Browser, you can explore the context and the details of your application’s traces.

See the Traces Browser:

  • Option 1:
    1. In your web browser, go to your Wavefront cluster and log in.
    2. From the taskbar, click Applications > Traces.
  • Option 2: Drill down to the Traces Browser from the Service Dashboard, application map, table view, or grid view.

explore trace browser

From the Traces Browser, you can:

  • Query for traces and view the query results in the traces list.
  • Select a trace in the list and:
  • Examine a trace’s percentile indicator to see how the trace’s duration compares to the durations of the other listed traces. You can toggle the panel size for the traces list, service map, or trace details.
  • Export traces by clicking import tracing icon, save the JSON file, and view them later using Offline Traces.
  • Easily analyze your traces hierarchy and RED metrics by clicking the expand or restore icon.
    • Click the expand icon to expand the RED metrics and view them next to the trace list, and click the same icon again to restore to the default view. partial expand red metrics
    • Click the expand icon to expand the trace hierarchy and view them next to the trace list, and click the same icon again to restore to the default view. partial expand trace hierachy
  • Use icon to click to get the link to get a link and share what you’re seeing right now (NON-LIVE display) with other users.

Query for a List of Traces

In the Traces Browser, you can use the query builder or query editor to query for traces that include spans for a particular operation.

You can view the results in a traces list.

explore traces list

Here’s how to get started:

  1. Add an Operation or Trace ID. See Get Started with Trace Queries for details.
  2. Click Add Filter and specify a filter to limit the results.
  3. (Optional) Advanced users can use Query Editor to limit the scope even further.

You can use the trace list to:

  • Sort the returned traces according to different criteria.
  • Click a trace to view its context in the service map panel and its details in the trace details panel.
  • View a trace’s percentile indicator to see how the trace’s duration compares to the durations of the other listed traces – and potentially find outliers.

Investigate the Service Map for a Trace

In the Traces Browser, use the service map to investigate the services that contribute spans to a selected trace.

explore service map

In the service map panel, you can:

  • View the service dependencies, and follow the flow of request calls from service to service.
  • Click on a service to display the RED metrics that reflect the health of the service:
    • Request count, error count, and trace duration at the 95th percentile, over all traces (root spans) that originate in the service.
    • Line charts that indicate the general contour of the RED metrics.
    • Fine-tune the time window of the RED metrics charts, and see events.
  • Navigate to the selected service’s dashboard for more service details.
  • Scroll the service map to zoom in or out, and re-center the service map.
  • Look for nodes that are grayed out. These represent services that contribute spans to at least one trace in the list, but not to the trace that is currently selected.

Watch this video to see how a service map can help you pinpoint a performance bottleneck in a microservices architecture:

introduction to service maps

Examine Trace Details

In the Traces Browser, use the trace details panel to examine the spans that belong to a selected trace. Some of these spans may represent operations executed by other services.

trace details

In the trace details panel, you can:

  • Examine the call hierarchy for a selected trace.
  • Click any span to expand and see more detail.
  • Explore the trace’s critical path. This is an end-to-end sequence of blocking spans, where each blocking span must complete before the trace can complete.

Drill Down Into Spans and View Metrics and Span Logs

In the trace details panel, you can view details about a particular span in the trace. If that span came from another service, you can navigate to the dashboard for that service.

trace span details

Span details include:

  • Application tags. These are the application, service, cluster, and shard, as selected by the trace query.
  • Other tags, including the trace ID.
  • A clickable link to the corresponding dashboard that lets you examine the metrics associated with the call.

If your spans have been instrumented to show span logs, you can click Logs to fetch the logs and drill down to examine them. We don’t show span logs by default for better browser performance.

A Closer Look at Critical Paths

The trace details panel uses an orange line to show the critical path through a trace. You can think of the critical path as the end-to-end combination of spans that are the most blocking. These spans represent the sequence of operations that must complete before the trace itself can complete.

Analyzing the critical path of a trace can help you determine which operations took the most time, and can help you decide which operations to try to optimize.

We use the following rules to determine which spans to include in a critical path (in order of applicability):

  1. Ignore asynchronous spans (spans tagged with followFrom).
  2. Ignore spans that end before their parent starts.
  3. If a child span continues after its parent, ignore that continuation period.
  4. Choose longer spans over shorter siblings.
  5. Choose later spans over earlier spans.
  6. Choose child spans instead of their parent spans.

View Traces Offline

You can export traces from Wavefront, save them locally as JSON files, and view them later using the Offline Traces.

  1. Export Traces:
    • Export the traces via the Traces Browser and save the JSON file.
    • Export traces via the API.
      • Use the Wavefront Swagger UI. See API Documentation (Wavefront Instance) for details on navigating to the Wavefront Swagger UI.
        Example: UI image showing where the API is on the Wavefront Swagger UI.
      • Use a curl command that has the /api/v2/chart/api URL.
        Example:
        curl -X GET --header "Accept: application/json" --header "Authorization: Bearer <Wavefront_Token>" "https://<Tenant_Name>.wavefront.com/api/v2/chart/api?q=limit(100%2C%20traces(spans(%22beachshirts.shopping.*%22)))&s=1601894248&g=d&view=METRIC&sorted=false&cached=true&useRawQK=false"
        
  2. Upload the JSON file or feed the JSON response from the API to offline traces, and view trace data. You can only upload one JSON file at a time.
    1. Click Applications > Offline Traces.
    2. Click Upload JSON, select the JSON file you saved that has the imported trace data, and click Open.
      Now, you see the trace data you imported. Shows how the offline traces look once you upload the JSOn file that has the imported trace details.