Trace - Microservice Monitoring and Debugging

We are happy to announce Trace, a microservice monitoring and debugging tool that empowers you to get all the metrics you need when operating microservices. Trace both comes as a free, open source tool and as a hosted service.

UPDATE: This article mentions Trace, RisingStack’s Node.js Monitoring platform several times. On 2017 October, Trace has been merged with Keymetrics’s APM solution. Click here to give it a try!

Why Trace for microservice monitoring?

Debugging and monitoring microservices can be really challenging:

• no stack trace, hard to debug
• easy to lose track of services when dealing with a lot
• bottleneck detection

Key Features

Trace solves these problems by adding the ability to

• do distributed stack traces,
• topology view for your services,
• and alerting for overwhelmed services,
• third-party service monitoring (coming soon),
• trace heterogeneous infrastructures with languages like Java, PHP or Ruby (coming soon).

How It Works

We want to monitor the traffic of our microservices. To be able to do this, we have to access each HTTP request-response pairs to get and set information. With wrapping the http core module's request function and the Server.prototype object, we can sniff all the information we need.

Trace is mostly based on the Google Dapper white paper - so we implemented the ServerReceive, ServerSend, ClientSend, ClientReceive events for monitoring the lifetime of a request.

In the example above, we want to catch the very first incoming request: SR (A): Server Receive. The http.Server will emit a request event, with http.IncomingMessage and a http.ServerResponse with the signature of

function (request, response) { }

In the wrapper, we can record every information we want, like timing, the source, the requested path, or even the whole HTTP header for further investigation.

In Trace, one of the fundamental features is tracking the whole transaction in microservice architectures. Luckily we can do it, by setting a request-id header on the outgoing requests.

If our service has to call another service before it can send the response to its caller, we have to track this kind of request-response pairs, spans as well. A span always comes from http.request by calling an endpoint. By wrapping the http.request function, we can do the same as in the http.Server.prototype with one minor difference: here we want to pair the corresponding request and response, and assign a span-id to it.

However, the request-id will just pass through the span. In order to store the generated request-id, we use Continuation-Local Storage: after a request arrived and we generated the request-id, we store it in CLS, so when we try to call another service we can just get it back.

Create reporters

After you set up the collector by simply requiring it in your main file:

require('@risingstack/trace');

You can select a reporting method to process the collected data. You can use:

• our Trace servers to see the transactions, your topology and services,
• Logstash,
• or any other custom reporter (see later).

You have to provide a trace.config.js config file, where you can declare the reporter. If you just want to see the collected data, you can use Logstash with the following config file:

/**
* Trace example config file for using with Logstash
*/

var reporters = require('@risingstack/trace/lib/reporters');
var config = {};

config.appName = 'Example Service Name';

config.reporter = reporters.logstash.create({
  type: 'tcp',
  host: 'localhost',
  port: 12201
});

module.exports = config;

If you start Logstash with the following command, every collected packet information will be displayed in the terminal:

logstash -e 'input { tcp { port => 12201 } } output { stdout {} }'

Also, this approach can be really powerful when you want to tunnel these metrics into different systems, like ElasticSearch or just store them on S3.

Adding custom reporters

If you want to use the collector with your custom reporter, you have to provide your own implementation of the reporter API. The only required method is a send method with the collected data and a callback as parameters.

function CustomReporter (options) {
  // init your reporter
}

CustomReporter.prototype.send = function (data, callback) {
  // implement the data sending,
  // don't forget to call the callback after the data sending has ended
};

function create(options) {
  return new CustomReporter(options);
}

module.exports.create = create;

Use the Trace collector with Trace servers

If you want to enjoy all the benefits of our Trace service, you need to create an account first. After your API Key has been generated, you can use it in your config file:

/**
* Trace example config file for using with Trace servers
*/

var config = {};

config.appName = 'Example Service Name';

config.reporter = require('@risingstack/trace/lib/reporters').trace.create({
  apiKey: 'YOUR-APIKEY',
  appName: config.appName
});

module.exports = config;

Adding Trace to your project

To use the Trace collector as a dependency of your project, use:

npm install --save @risingstack/trace

Currently, Trace supports [email protected], [email protected] and [email protected].

Trace-as-a-Service

If you don't want run your own infrastructure for storing and displaying microservice metrics we provide microservice monitoring as a service as well. This is Trace: