Building Rails for scale
Dinshaw Gobhai | dgobhai@constantcontact.com
@tallfriend
dev-setup.md
1 ## Install prerequisites
2 brew install mysql
... ...
1002
1003 ## Run the app
1004 rails s
Don't do it.
Don't over architect.
Don't prematurely engineer.
Don't solve problems you don't have yet.
iRule to direct requests by :account_id
8 Torquebox instances (96 app servers)
1 Redis Server
1 Memcached Server
1 MySQL master, 1 slave
SLA: < 500ms
Actual: ~ 44s
---
(3M / 500) * 44s = 3 days
(3B / 500) * 44s = 8.4 years
Ruby-prof/Dtrace
Benchmark
def count
Benchmark.measure "Count people" do
ContactsSelector.count_people(params)
end
end
Preload, Eagerload, Includes and Joins
.preload(:association)
Separate queries for associated tables.
.eager_load(:association)
One query with all associations 'LEFT OUTER' joined.
.includes(:association)
Picks one of the above.
.joins(:association)
One query with all associations 'INNER' joined.
https://www.youtube.com/watch?v=ShPAxNcLm3o
.where(
"author.name = ? and posts.active = ?",
"Jane", true
)
Post
.joins(:comments)
.joins(Comments.joins(:author).join_sources)
.where(
Author[:name].eq('Jane')
.and(Post[:active].eq(true))
)
SLA: < 500ms
Actual: ~ 26s
Rails anti-pattern?
Multiple-column index is faster than multiple indexs
class Contact > ActiveRecord::Base
primary_key [:account_id, :contact_id]
...
end
Remember a previous method lookup, directly at call site.
class Foo
def do_someting
puts 'foo!'
end
end
# first time does a full lookup of .do_something
# stores "puts 'foo!'" at the call site, or 'in-line'
foo = Foo.new
foo.do_something
# second time
# knows to just run "pust 'foo!'"
bar = Foo.new
bar.do_something
class Foo
def do_something
puts 'foo!'
end
end
class Bar
def do_something
puts 'bar?'
end
end
# Worst case scenario for Monomorphic
[Foo.new, Bar.new, Foo.new, Bar.new].each do |obj|
obj.do_something
end
case obj.class
when Foo; puts 'foo!'
when Bar; puts 'bar?'
else # lookup ...
end
github.com/charliesome/...rubys-method-cache
# composite_primary_keys/relation.rb
def add_cpk_support
class << self
include CompositePrimaryKeys::ActiveRecord::Batches
include CompositePrimaryKeys::ActiveRecord::Calculations
...
# patch
class CompositePrimaryKeys::ActiveRecord::Relation < ActiveRecord::Relation
include CompositePrimaryKeys::ActiveRecord::Batches
include CompositePrimaryKeys::ActiveRecord::Calculations
...
class ActiveRecord::Relation
def self.new( klass, ... )
klass.composite? ?
CompositePrimaryKeys::ActiveRecord::Relation.new : self
end
Tables still too big - millions of Contacts per cell
MySQL Hash partitioning by :account_id
SLA: < 500ms
Actual: ~ 4s
DB: 12ms
Skip your ORM
string = params[:country]
country = COUNTRY_CODE_MAP.detect do |k,v|
[k.downcase, v.downcase].include? string.downcase
end
string = params[:country]
country = COUNTRY_CODE_MAP.detect do |k,v|
k.casecmp(string) == 0 || v.casecmp(string) == 0
end
def contact_ids
@contact_ids ||= params[:ids].split(',')
end
def subscriber_confirmation
@subscriber_confirmation ||= expensive_lookup
end
def subscriber_confirmation
@subscriber_confirmation ||= Rails.cache.fetch(
"subscriber_confirmation:#{Current.account}",
:expires_in => LONG_TIME_IN_SECONDS
) { | key| expensive_lookup(key) }
end
class SomeModel
def funky_method
unless complicated_action_succeeds
raise FunkyError, 'Something Funky Failed'
end
end
end
class SomeController
def some_action
response = SomeModel.new(params).funky_method
render status: 200, json: response
rescue FunkyError => e
render status: 400, json: {errors: 'Funky Failboat!'}
end
end
class SomeModel
def funky_method
unless complicated_action_succeeds
{ error: 'Yeah, that happens' }
end
end
end
class SomeController
def some_action
response = SomeModel.new(params).funky_method
status = response[:error] ? 400 : 200
render status: status, json: response
end
end
Rails.logger.debug "This is expensive #{some_expensive_method}"
big_array.each do |x|
Rails.logger.debug 'processing row'
...
end
if Rails.logger.debug?
Rails.logger.debug "This is expensive #{some_expensive_method}"
end
Rails.logger.debug "processing #{big_array.size} rows"
big_array.each do |x|
...
end
def all_contact_ids
Contact.all.map &:contact_id
end
def all_contact_ids
Contact.all.pluck(:contact_id)
end
class Contact
has_many :addresses, dependent: :destroy
def self.remove_bad_contacts
self.destroy_all(bad_contact: true)
end
end
class Contact
has_many :addresses, dependent: :destroy
scope :bad_contacts, -> { where(bad_contact: true) }
def self.remove_bad_contacts
Address.delete_all contact_id: self.bad_contacts.pluck(:contact_id)
self.bad_contacts.delete_all
end
end
class Contact < ActiveRecord::Base
validates :first_name, uniqueness: {scope: :last_name}
end
class Contact < ActiveRecord::Base
around_save :check_uniqueness
def check_uniqueness
yield
rescue ActiveRecord::RecordNotUnique => exception
errors.add :base, "contact is not unique"
raise ActiveRecord::RecordInvalid.new(self)
end
end
# db migration
add_index :contacts, [:firstname, :lastname], :unique => true
Dinshaw Gobhai | dgobhai@constantcontact.com
@tallfriend
github.com/dinshaw/meet-the-slas
Alex 'the beast' Berry
Andre 'the log grepper' Zelenkovas
Tom 'the meeting man' Beauvais