The Wavefront tracing UI enables exploration of your apps and services. Once your application is instrumented for tracing, you can examine traces, spans, and RED metrics, moving easily from one browser page to the next.
This page explains how you can examine trace data from our UI. You’ll learn how to query for traces, examine details, find potential problems, and more.
Choose Your Starting Point
To choose your starting point for visualizing traces:
- In your web browser, go to your Wavefront cluster and log in.
From the task bar:
- Select Applications > Application Status to view the status of your instrumented applications and drill down from there.
- Select Applications > Traces to start by querying for traces and drill down from there.
View Application Status
Go to the Applications page for a top-level status overview of your instrumented applications.
On the Applications page, you can:
- View the status of all instrumented applications, or search for a particular application by applying filters.
- Apply one or more filters to select application name, cluster, or shard and click Apply.
- Inspect RED metrics to obtain a status summary for an application:
- The total number of requests that are represented by the application’s spans.
- The percentage of the application’s spans that contain errors.
- The span duration (in milliseconds) at the 95th percentile across the application.
- Sort the displayed applications by name or by a RED metric.
- Click an application name for an overview of its services.
View the Services of an Application
When you select an application, you get an overview of its services.
On the page for a particular application, you can:
- Examine the services in the application, or search for a particular service by applying filters.
- View the inventory of component frameworks that each service is built on.
- Inspect RED metrics to obtain a status summary for a service:
- The total number of requests that are represented by the service’s spans.
- The percentage of the service’s spans that contain errors.
- The span duration (in milliseconds) at the 95th percentile across the service.
- Drill down from a service box:
- Click Details to go to the dashboard for that service.
- Click All Traces to explore the traces that originate in that service.
Examine Service Metrics and Drill Down
When you click on a service’s Details, you can examine a dashboard of metrics to identify potential hot spots, and then drill down to the Traces browser.
Explore the Default Service Metrics Dashboard
The default, read-only dashboard for a service lets you explore that service, however, you can’t make changes to the dashboard.
On the dashboard for a particular service, you can:
- Select the time and timezone in the task bar to customize the chart time properties.
- Use the Jump To pulldown to select a dashboard section:
- Select Overview to examine the RED metrics that are derived from all of the spans for the service. These metrics reflect the health of the service.
- Select an individual component to examine metrics for just that component of the service. A component could be an instrumented framework (e.g., Jersey) or the runtime system (e.g., JVM).
- Select System if your environment uses Telegraf and you want to view CPU usage, memory usage, and disk usage.
- Filter the metrics based on the cluster, shard, or source.
- Select Detailed View or Summarized View to change the level of detail for charts.
- Examine the TopK charts to find out which operations are potential hot spots. The bars represent operations that execute in this component of the service.
- Navigate to the Traces browser:
- Select See All … Traces to display all traces that include a span from this service component.
- Click a bar in a TopK chart to display just the traces that include spans for the selected operation.
Custom Service Metrics Dashboard
The standard dashboard for service metrics is read-only. To create a customizable copy:
- Click Clone from the ellipsis menu.
- In the cloned dashboard, add your own charts or customize the RED metrics charts. (Use the ts_countersum function to display RED metrics.)
After you save the clone, you can find it by name from the Dashboards menu of the task bar, and you can use it to drill down to the Traces browser.
In the Traces browser, you can explore the context and the details of your application’s traces.
- Navigate from the service’s page to display traces for operations you selected.
- Select Applications > Traces from the task bar to display an empty page that you populate by querying.
From the Traces browser, you can:
- Query for traces and view the query results in the traces list.
- Select a trace in the list and:
- Examine a trace’s percentile indicator to see how the trace’s duration compares to the durations of the other listed traces.
You can toggle the panel size for the traces list, service map, or trace details.
Query for a List of Traces
In the Traces browser, you can use the query builder or query editor for query for the traces that include spans for a particular operation.
You can view the results in a traces list.
Here’s how to get started:
- Start typing in the Operations field and select an operation (or type a name in the search box).
- Click Add Filter and specify a filter to limit the results.
- (Optional) Advanced users can use Query Editor to limit the scope even further.
You can use the trace list to:
- Sort the returned traces according to different criteria.
- Click a trace to view its context in the service map panel and its details in the trace details panel.
- View a trace’s percentile indicator to see how the trace’s duration compares to the durations of the other listed traces – and potentially find outliers.
Investigate the Service Map for a Trace
In the Traces browser, use the service map to investigate the services that contribute spans to a selected trace.
In the service map panel, you can:
- View the service dependencies, and follow the flow of request calls from service to service.
- Click on a service to display the RED metrics that reflect the health of the service:
- Request count, error count, and trace duration at the 95th percentile, over all traces (root spans) that originate in the service.
- Line charts that indicate the general contour of the RED metrics.
- Navigate to the selected service’s dashboard for more service details.
- Scroll the service map to zoom in or out, and re-center the service map.
- Look for nodes that are grayed out. These represent services that contribute spans to at least one trace in the list, but not to the trace that is currently selected.
Watch this video to see how a service map can help you pinpoint a performance bottleneck in a microservices architecture:
Examine Trace Details
In the Traces browser, use the trace details panel to examine the spans that belong to a selected trace. Some of these spans may represent operations executed by other services.
In the trace details panel, you can:
- Examine the call hierarchy for a selected trace.
- Click any span to expand and see more detail.
- Explore the trace’s critical path. This is an end-to-end sequence of blocking spans, where each blocking span must complete before the trace can complete.
Drill Down Into Spans and View Metrics and Span Logs
In the trace details panel, you can view details about a particular span in the trace. If that span came from another service, you can navigate to the dashboard for that service.
Span details include:
- Application tags. These are the application, service, cluster, and shard, as selected by the trace query.
- Other tags, including the trace ID.
- A clickable link to the corresponding dashboard that lets you examine the metrics associated with the call.
If your spans have been instrumented to show span logs, you can click Logs to fetch the logs and drill down to examine them. We don’t show span logs by default for better browser performance.
A Closer Look at Critical Paths
The trace details panel uses an orange line to show the critical path through a trace. You can think of the critical path as the end-to-end combination of spans that are the most blocking. These spans represent the sequence of operations that must complete before the trace itself can complete.
Analyzing the critical path of a trace can help you determine which operations took the most time, and can help you decide which operations to try to optimize.
We use the following rules to determine which spans to include in a critical path (in order of applicability):
- Ignore asynchronous spans (spans tagged with
- Ignore spans that end before their parent starts.
- If a child span continues after its parent, ignore that continuation period.
- Choose longer spans over shorter siblings.
- Choose later spans over earlier spans.
- Choose child spans instead of their parent spans.