Lynn Chang's Blog

How to Benchmark my code in Ruby or Rails

Photo by Steven Lelham on Unsplash

How to Benchmark my code in Ruby or Rails

Lynn Chang's photo
Lynn Chang
·

4 min read

Recently, I had a job to fix the slow code. Before diving into fixing the slow code, I used Scout APM to identify the performance bottlenecks, which helped me make sure where the slow code is.

To prove the new optimization solution is faster, in addition to checking the before-after response time in the log file, I think I need a more efficient tool to evaluate the solution.

Benchmarking

Benchmarking is a way of measuring the performance of the code. It can answer the question like 'Which one is faster, A or B?'

Ruby has a benchmarking tool called Benchmark in its standard library. Benchmark module provides methods to measure and report the time used to execute Ruby code.

How to use Ruby Benchmark module?

You can add a Ruby file first.

$ touch test.rb

You should require 'benchmark' to make sure you can use it. Next, put the code that you would like to test into the x.report { #put your code here } block.

require 'benchmark'

n = 100
Benchmark.bm do |x|
  x.report('A') { for i in 1..n; a = '1'; end }
  x.report('B') { n.times do   ; a = '1'; end }
  x.report('C') { 1.upto(n) do ; a = '1'; end }
end

Go back to the terminal and execute the file:

We can see the C plan is faster than the A or B plan.

$ ruby test.rb
       user     system      total        real
A  0.000026   0.000005   0.000031 (  0.000024)
B  0.000018   0.000001   0.000019 (  0.000018)
C  0.000016   0.000001   0.000017 (  0.000017)

This report shows the user CPU time, system CPU time, the sum of the user and system CPU times, and the elapsed real time. The unit of time is seconds. (Source)

Another enhancement benchmarking gem - benchmark-ips

When searching the benchmarking tool, I found out another interesting gem called benchmark-ips.

Feature

An iterations per second enhancement to Benchmark.

What's the difference between the traditional Benchmark library and benchmark-ips?

Benchmark/ips will report the number of iterations per second for a given block of code. When analyzing the results, notice the percent of standard deviation which tells us how spread out our measurements are from the average. A high standard deviation could indicate the results having too much variability.

benchmark-ips uses standard deviation which is the concept of Statistics to analyze the result. The benefit is that we can literally make sure the result is significant/extreme significant/no significant difference.

In other words, take the result above for example:

C plan seems to be faster than the A or B plan, but it might not be the fact.

# n = 100
$ ruby test.rb
       user     system      total        real
A  0.000026   0.000005   0.000031 (  0.000024)
B  0.000018   0.000001   0.000019 (  0.000018)
C  0.000016   0.000001   0.000017 (  0.000017)

Let's use benchmark-ips to analyze the result:

require "benchmark/ips"

n = 100
Benchmark.ips do |x|
  x.report('A') { for i in 1..n; a = "1"; end }
  x.report('B') { n.times do  ; a = "1"; end }
  x.report('C') { 1.upto(n) do ; a = "1"; end }

  x.compare!
end

There is something interesting:

  • B plan seems to iterate more times per second than the A or C plan.
Warming up --------------------------------------
                   A    11.038k i/100ms
                   B    11.868k i/100ms
                   C    12.166k i/100ms
Calculating -------------------------------------
                   A    125.949k (± 3.9%) i/s -    629.166k in   5.003441s
                   B    126.662k (± 2.7%) i/s -    640.872k in   5.063567s
                   C    125.172k (± 6.3%) i/s -    632.632k in   5.076965s

Comparison:
                   B:   126662.1 i/s
                   A:   125949.4 i/s - same-ish: difference falls within error
                   C:   125172.4 i/s - same-ish: difference falls within error

From the perspective of Statistics, there is not a meaningful difference between the three plans. When I change n=100 to n=10000, the result is the same. I feel very surprised about the result. Very. (Please correct me if I misunderstood. 🙏)

How to Benchmark the code in Rails

It's simple. Just add require_relative '../../config/environment' so that you can use the model class you have built in your Rails App.

require 'benchmark/ips'
require_relative '../../config/environment'

transaction_ids = [1, 2, #....]
transactions = Transaction.where(id: transaction_ids)

Benchmark.ips do |x|
  x.report('new') do
    transactions.map(&:some_column).compact
  end

  x.report('old') do
    transaction_records = transactions.
      includes(:a_table, :b_table, :c_table).order(created_at: :desc)

    transaction_records.map(&:some_column).compact
  end

  x.compare!
end

Reference

Ruby