Hi There

I am going to show you how to integrate Prometheus with Rails application. We will promethues to collect metrics from our application and we are going to display them in Grafana.

You might be wondering what is Prometheus and Grafana? Let’s start with Prometheus.

Prometheus is an open-source monitoring and alerting toolkit.
It is a time-series database and a pull-based monitoring system.
It is used to collect metrics from our application and store them in a time-series database.
It also provides a query language to query the metrics.
And many more features. You can check the link here to know more about Prometheus.

Why do we need Prometheus? The answer is simple. These days we are building complex systems where we need to monitor our application and we need to know how our application is performing. If you don’t want to spend too much money on monitoring tools then Prometheus is the best choice. It is open source and it supports plenty of languages and frameworks.

Now let’s talk about Grafana. Grafana is an open-source platform for monitoring and observability. It is used to visualize the metrics collected from different data sources. It can be Prometheus but there many others. Grafan provides a powerful and elegant way to create, explore, and share dashboards. The UX might be a little bit complex but once you get used to it, it’s very easy to use. You can check the link here to know more about Grafana.

Ok, enough of the introduction. Let’s start with the simple app.

Firstly, let’s create a rails app but in the api mode. We don’t need any views for this app.

rails new o11y-rails --api -T

We are going to use puma, sidekiq and prometheus_exporter gem for this app. Add the following gems to your Gemfile.

# Gemfile
gem 'puma'
gem 'sidekiq'
gem 'prometheus_exporter'

We will collect metrics for puma and sidekiq because these are the most common gems used in the rails application.

Ok, once you added the gems to your Gemfile, we will configure the prometheus_exporter gem. Firstly, we need to create an initializer file for the prometheus_exporter gem. For purpose of this tutorial I will keep it simple.

Create a file


config/initializers/prometheus_exporter.rb

and add the following code.


require "prometheus_exporter/middleware"

Rails.application.middleware.unshift PrometheusExporter::Middleware

More here Rails Usage

What is prometheus_exporter gem doing?

It collects the metrics from the application and stores them in the memory.
It provides a way to expose the metrics to the prometheus server.
There is also a standalone server provided by the gem to expose the metrics to the prometheus server. You can run it locally using the following command.

prometheus_exporter

It will return the following output.

Starting prometheus exporter on localhost:9394

So if you start the rails server and prometheus_exporter server then you can go to http://localhost:9394/metrics you will see the metrics collected by the prometheus_exporter.

http://localhost:9394/metrics

As you can see we have some puma and sidekiq metrics there. In order to achieve this, we need to add some code to puma config and sidekiq initializer.

Add the following code to the puma config file.


# config/puma.rb
require "prometheus_exporter/instrumentation"
if !PrometheusExporter::Instrumentation::Puma.started?
  PrometheusExporter::Instrumentation::Puma.start
  PrometheusExporter::Instrumentation::ActiveRecord.start(
    custom_labels: {type: "puma_worker"}
    config_labels: [:database, :host]
  )
end

This will instrument the puma server but also the active record.

Ok, now let’s add the code to the sidekiq initializer. The full config can look like this:


# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = {url: "redis://redis:6379/0"}

  require "prometheus_exporter/instrumentation"
  config.server_middleware do |chain|
    chain.add PrometheusExporter::Instrumentation::Sidekiq
  end
  config.death_handlers << PrometheusExporter::Instrumentation::Sidekiq.death_handler
  config.on :startup do
    PrometheusExporter::Instrumentation::Process.start type: "sidekiq"
    PrometheusExporter::Instrumentation::SidekiqProcess.start
    PrometheusExporter::Instrumentation::SidekiqQueue.start
    PrometheusExporter::Instrumentation::SidekiqStats.start
  end
end

Sidekiq.configure_client do |config|
  config.redis = {url: "redis://redis:6379/0"}
end

Link: Sidekiq Metrics

This will instrument all stuff related to sidekiq. There are useful metrics like queue latency, queue size, etc. You can check the link above to know more about the metrics.

Once we have the setup ready, we can also add dummy endpoint to simulate some load on the server and inside sidekiq.


# config/routes.rb
resources :hello, only: [:index]


# app/controllers/hello_controller.rb
class HelloController < ApplicationController
  def index
    1000.times do
      HelloJob.perform_async("world", 3)
    end
    render json: {message: "Hello, world!"}
  end
end


# app/jobs/hello_job.rb
class HelloJob
  include Sidekiq::Job

  def perform(name, times = 1)
    sleep 5
    # raise error randomly
    raise "Error" if rand(1..100) < 10

    logger.info { {message: "Performing work"} }
    logger.info { {message: "Hello, #{name}! " * times} }
  end
end

Now we have a simple rails app with puma and sidekiq configured to collect metrics. Let’s make our life simple and start all services using docker-compose.


version: "3.8"
services:
  web:
    build: .
    command: bundle exec rails server -p 3000 -b '0.0.0.0'
    depends_on:
      - prometheus_exporter
    volumes:
      - .:/o11y-rails
    ports:
      - "3000:3000"
    links:
      - redis
      - prometheus_exporter
  sidekiq:
    build: .
    command: bundle exec sidekiq
    volumes:
      - .:/o11y-rails
    links:
      - redis
  redis:
    image: "redis:latest"
    ports:
      - "6379:6379"
  prometheus_exporter:
    image: discourse/prometheus_exporter
    build:
      context: ../../docker-images/prometheus_exporter/
      dockerfile: ../../docker-images/prometheus_exporter/Dockerfile
      args:
        RUBY_VERSION: "3.2.3"
    ports:
      - "9394:9394"

    volumes:
      - ./tmp/prometheus_exporter:/tmp
  prometheus:
    depends_on:
      - prometheus_exporter
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"

  grafana:
    image: grafana/grafana-oss
    ports:
      - "3001:3000"
    volumes:
      - grafana-data:/var/lib/grafana
volumes:
  prom_data:
  grafana-data:

For the prometheus server we need to add config so it can scrape the metrics from the prometheus_exporter server. Create a file prometheus.yml and add the following code.


global:
  scrape_interval: 5s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - prometheus:9090
  - job_name: "o11y-rails"
    scrape_interval: 5s
    metrics_path: "/metrics"
    static_configs:
      - targets: ["prometheus_exporter:9394"]

Now we have everything ready to start the services. Run the following command to start the services. So docker-compose up.

You can visist:

http://localhost:3000/hello to simulate some load on the server and sidekiq.
http://localhost:9394/metrics to see the metrics collected by the prometheus_exporter.
http://localhost:9090 to see the prometheus server running with UI
http://localhost:3001 to see the grafana server running with UI.

Once you go to localhost:9090 you will see promethues UI and you can query the metrics. For example, let’s say that we want to see how many requests we have have for each endpoint.

In order to group them by the endpoint, you can use the following query.

sum(ruby_http_requests_total{job="o11y-rails"}) by (controller)

This will return the number of requests for each endpoint and the result will look like this.

PromQL is a powerful query language and you can do a lot of things with it. You can check the link here to know more about PromQL.

Having know the very basicis now, let’s move to the Grafana and add data source. What is data source?

Data source is a connection to the database or any other data source which provides the data to the Grafana. In our case, the data source is the prometheus server.

Steps:

Go to http://localhost:3001 and login with the default credentials. (admin:admin)
Click on the hamburger icon on the left side and then click on the Data Sources.
Click on the Add data source button.
Select Prometheus from the list of data sources.
Add the URL of the prometheus server. In our case it’s *http://localhost:9090
Click on the Save & Test button. All done.

Creating a new dashboard and visualizations is also simple. Having this data exported from the rails app, I’ve created this dummy dashboard.

Adding a simple panel is mostly about selecting the data source and writing the PromQL query.

Hope you find this useful. If you have any questions, feel free to ask me.

Happy hacking!