Recently, I had a job to fix the slow code. Before diving into fixing the slow code, I used Scout APM to identify the performance bottlenecks, which helped me make sure where the slow code is.
To prove the new optimization solution is faster, in addition to checking the before-after response time in the log file, I think I need a more efficient tool to evaluate the solution.
Benchmarking
Benchmarking is a way of measuring the performance of the code. It can answer the question like 'Which one is faster, A or B?'
Ruby has a benchmarking tool called Benchmark in its standard library. Benchmark module provides methods to measure and report the time used to execute Ruby code.
How to use Ruby Benchmark module?
You can add a Ruby file first.
$ touch test.rb
You should require 'benchmark' to make sure you can use it. Next, put the code that you would like to test into the x.report { #put your code here } block.
require 'benchmark'
n = 100
Benchmark.bm do |x|
x.report('A') { for i in 1..n; a = '1'; end }
x.report('B') { n.times do ; a = '1'; end }
x.report('C') { 1.upto(n) do ; a = '1'; end }
end
Go back to the terminal and execute the file:
We can see the C plan is faster than the A or B plan.
$ ruby test.rb
user system total real
A 0.000026 0.000005 0.000031 ( 0.000024)
B 0.000018 0.000001 0.000019 ( 0.000018)
C 0.000016 0.000001 0.000017 ( 0.000017)
This report shows the user CPU time, system CPU time, the sum of the user and system CPU times, and the elapsed real time. The unit of time is seconds. (Source)
Another enhancement benchmarking gem - benchmark-ips
When searching the benchmarking tool, I found out another interesting gem called benchmark-ips.
Feature
An iterations per second enhancement to Benchmark.
What's the difference between the traditional Benchmark library and benchmark-ips?
Benchmark/ips will report the number of iterations per second for a given block of code. When analyzing the results, notice the percent of standard deviation which tells us how spread out our measurements are from the average. A high standard deviation could indicate the results having too much variability.
benchmark-ips uses standard deviation which is the concept of Statistics to analyze the result. The benefit is that we can literally make sure the result is significant/extreme significant/no significant difference.
In other words, take the result above for example:
C plan seems to be faster than the A or B plan, but it might not be the fact.
# n = 100
$ ruby test.rb
user system total real
A 0.000026 0.000005 0.000031 ( 0.000024)
B 0.000018 0.000001 0.000019 ( 0.000018)
C 0.000016 0.000001 0.000017 ( 0.000017)
Let's use benchmark-ips to analyze the result:
require "benchmark/ips"
n = 100
Benchmark.ips do |x|
x.report('A') { for i in 1..n; a = "1"; end }
x.report('B') { n.times do ; a = "1"; end }
x.report('C') { 1.upto(n) do ; a = "1"; end }
x.compare!
end
There is something interesting:
- B plan seems to iterate more times per second than the A or C plan.
Warming up --------------------------------------
A 11.038k i/100ms
B 11.868k i/100ms
C 12.166k i/100ms
Calculating -------------------------------------
A 125.949k (± 3.9%) i/s - 629.166k in 5.003441s
B 126.662k (± 2.7%) i/s - 640.872k in 5.063567s
C 125.172k (± 6.3%) i/s - 632.632k in 5.076965s
Comparison:
B: 126662.1 i/s
A: 125949.4 i/s - same-ish: difference falls within error
C: 125172.4 i/s - same-ish: difference falls within error
From the perspective of Statistics, there is not a meaningful difference between the three plans. When I change n=100 to n=10000, the result is the same. I feel very surprised about the result. Very. (Please correct me if I misunderstood. 🙏)
How to Benchmark the code in Rails
It's simple. Just add require_relative '../../config/environment' so that you can use the model class you have built in your Rails App.
require 'benchmark/ips'
require_relative '../../config/environment'
transaction_ids = [1, 2,
transactions = Transaction.where(id: transaction_ids)
Benchmark.ips do |x|
x.report('new') do
transactions.map(&:some_column).compact
end
x.report('old') do
transaction_records = transactions.
includes(:a_table, :b_table, :c_table).order(created_at: :desc)
transaction_records.map(&:some_column).compact
end
x.compare!
end
Reference
