Optimizing asset bundling and serving with Rails
We spend a lot of time optimizing the front end experience at GitHub. With that said, our asset (css, javascript, images) packaging and serving has evolved to be the best…
We spend a lot of time optimizing the front end experience at GitHub. With that said, our asset (css, javascript, images) packaging and serving has evolved to be the best setup I’ve seen out of any web application I’ve worked on in my life.
Originally, I was going to package what we have up into a plugin, but realized that much of our asset packaging is specific our particular app architecture and choice of deployment strategy. If you haven’t read up on our deployment recipe _read it now_. I cannot stress enough how awesome it is to have 14 second no downtime deploys. In any case, you can find the relevant asset bundling code in this gist
Benefits of our asset bundling
- Users never have to wait while the server generates bundle caches, ever. With default Rails bundling, each time you deploy, each request until your server generates the bundle has to wait for the bundle to finish. This makes your site pause for about 30s after each deploy.
- We can use slower asset minifiers (such as YUI or Google Closure) without consequence to our users.
- Adding new stylesheets or javascripts is as easy as creating the file. No need to worry about including a new file in every layout file.
- Because we base our ASSET_ID off our git modified date, we can deploy code updates without forcing users to lose their css/js cache.
- We take full advantage of image caching with SSL while eliminating the unauthenticated mixed content warnings some browsers throw.
Our asset bundling is comprised of several different pieces:
- A particular css & js file structure
- Rails helpers to include css & js bundles in production and the corresponding files in development.
- A rake task to bundle and minify css & javascript as well as the accompanying changes to deploy.rb to make it happen on deploy
- Tweaks to our Rails environment to use smart ASSET_ID and asset servers
CSS & JS file layout
Our file layout for CSS & JS is detailed in the README for Javascript, but roughly resembles something like this:
public/javascripts
|-- README.md
|-- admin
| |-- date.js
| `-- datePicker.js
|-- common
| |-- application.js
| |-- jquery.facebox.js
| `-- jquery.relatize_date.js
|-- dev
| |-- jquery-1.3.2.js
| `-- jquery-ui-1.5.3.js
|-- gist
| `-- application.js
|-- github
| |-- _plugins
| | |-- jquery.autocomplete.js
| | `-- jquery.truncate.js
| |-- application.js
| |-- blob.js
| |-- commit.js
`-- rogue
|-- farbtastic.js
|-- iui.js
`-- s3_upload.js
I like this layout because:
- It allows me to namespace specific files to specific layouts (gist, github.com, iPhone, admin-only layouts, etc) and share files between apps (common).
- I can lay out files however I want within each of these namespaces, and reorganize them at will.
Some might say that relying on including everything is bad practice — but remember that web-based javascript is almost exclusively onDOMReady or later. That means that there is no dependency order problems. If you run into dependency order issues, you’re writing javascript wrong.
Rails Helpers
To help with this new bundle strategy, I’ve created some Rails helpers to replace your standard stylesheet_link_tag
and javascript_include_tag
. Because of the way we bundle files, it was necessary to use custom helpers. As an added benefit, these helpers are much more robust than the standard Rails helpers.
Here’s the code:
require 'find'
module BundleHelper
def bundle_files?
Rails.production? || Rails.staging? || params[:bundle] || cookies[:bundle] == "yes"
end
def javascript_bundle(*sources)
sources = sources.to_a
bundle_files? ? javascript_include_bundles(sources) : javascript_include_files(sources)
end
# This method assumes you have manually bundled js using a rake command
# or similar. So there better be bundle_* files.
def javascript_include_bundles(bundles)
output = ""
bundles.each do |bundle|
output << javascript_src_tag("bundle_#{bundle}", {}) + "n"
end
output
end
def javascript_include_files(bundles)
output = ""
bundles.each do |bundle|
files = recursive_file_list("public/javascripts/#{bundle}", ".js")
files.each do |file|
file = file.gsub('public/javascripts/', '')
output << javascript_src_tag(file, {}) + "n"
end
end
output
end
def javascript_dev(*sources)
output = ""
sources = sources.to_a
sources.each do |pair|
output << javascript_src_tag(Rails.development? ? "dev/#{pair[0]}" : pair[1], {})
end
output
end
def stylesheet_bundle(*sources)
sources = sources.to_a
bundle_files? ? stylesheet_include_bundles(sources) : stylesheet_include_files(sources)
end
# This method assumes you have manually bundled css using a rake command
# or similar. So there better be bundle_* files.
def stylesheet_include_bundles(bundles)
stylesheet_link_tag(bundles.collect{ |b| "bundle_#{b}"})
end
def stylesheet_include_files(bundles)
output = ""
bundles.each do |bundle|
files = recursive_file_list("public/stylesheets/#{bundle}", ".css")
files.each do |file|
file = file.gsub('public/stylesheets/', '')
output << stylesheet_link_tag(file)
end
end
output
end
def recursive_file_list(basedir, extname)
files = []
basedir = RAILS_ROOT + "/" + basedir
Find.find(basedir) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?.
Find.prune
else
next
end
end
files << path.gsub(RAILS_ROOT + '/', '') if File.extname(path) == extname
end
files.sort
end
end
Our application.html.erb
now looks something like this:
<%= javascript_dev ['jquery-1.3.2', "#{http_protocol}://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"] %>
<%= javascript_bundle 'common', 'github' %>
This includes jQuery and all javascript files under public/javascripts/common
and public/javascripts/github
(recursively). Super simple and we probably won’t need to change this for a very long time. We just add files to the relevant directories and they get included magically.
For pages that have heavy javascript load, you can still use the regular javascript_include_tag
to include these files (we keep them under the public/javascripts/rogue
directory).
Bundle rake & deploy tasks
The javascript_bundle
and stylesheet_bundle
helpers both assume that in production mode, there’ll be a corresponding bundle file. Since we are proactively generating these files, you need to create these manually on each deploy.
RAILS_ROOT ||= ENV["RAILS_ROOT"]
namespace :bundle do
task :all => [ :js, :css ]
task :js do
compression_method = "closure"
require 'lib/js_minimizer' if compression_method != "closure"
closure_path = RAILS_ROOT + '/lib/closure_compressor.jar'
paths = get_top_level_directories('/public/javascripts')
targets = []
paths.each do |bundle_directory|
bundle_name = bundle_directory.gsub(RAILS_ROOT + '/public/javascripts/', "")
files = recursive_file_list(bundle_directory, ".js")
next if files.empty? || bundle_name == 'dev'
target = RAILS_ROOT + "/public/javascripts/bundle_#{bundle_name}.js"
if compression_method == "closure"
`java -jar #{closure_path} --js #{files.join(" --js ")} --js_output_file #{target} 2> /dev/null`
else
File.open(target, 'w+') do |f|
f.puts JSMinimizer.minimize_files(*files)
end
end
targets << target
end
targets.each do |target|
puts "=> bundled js at #{target}"
end
end
task :css do
yuipath = RAILS_ROOT + '/lib/yuicompressor-2.4.1.jar'
paths = get_top_level_directories('/public/stylesheets')
targets = []
paths.each do |bundle_directory|
bundle_name = bundle_directory.gsub(RAILS_ROOT + '/public/stylesheets/', "")
files = recursive_file_list(bundle_directory, ".css")
next if files.empty? || bundle_name == 'dev'
bundle = ''
files.each do |file_path|
bundle << File.read(file_path) << "n"
end
target = RAILS_ROOT + "/public/stylesheets/bundle_#{bundle_name}.css"
rawpath = "/tmp/bundle_raw.css"
File.open(rawpath, 'w') { |f| f.write(bundle) }
`java -jar #{yuipath} --line-break 0 #{rawpath} -o #{target}`
targets << target
end
targets.each do |target|
puts "=> bundled css at #{target}"
end
end
require 'find'
def recursive_file_list(basedir, ext)
files = []
Find.find(basedir) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?. # Skip dot directories
Find.prune
else
next
end
end
files << path if File.extname(path) == ext
end
files.sort
end
def get_top_level_directories(base_path)
Dir.entries(RAILS_ROOT + base_path).collect do |path|
path = RAILS_ROOT + "#{base_path}/#{path}"
File.basename(path)[0] == ?. || !File.directory?(path) ? nil : path # not dot directories or files
end - [nil]
end
end
Throw this into lib/tasks/bundle.rake
_and the corresponding YUI & Closure jars_ and then run rake bundle:all
to generate your javascript. You can customize this to use the minifying package of your choice.
To make sure this gets run on deploy, you can add this to your deploy.rb:
namespace :deploy do
desc "Shrink and bundle js and css"
task :bundle, :roles => :web, :except => { :no_release => true } do
run "cd #{current_path}; RAILS_ROOT=#{current_path} rake bundle:all"
end
end
after "deploy:update_code", "deploy:bundle"
Tweaks to production.rb
The last step in optimizing your asset bundling for deploys is to tweak your production.rb config file to make asset serving a bit smarter. The relevant bits in our file are:
config.action_controller.asset_host = Proc.new do |source, request|
non_ssl_host = "http://assets#{source.hash % 4}.github.com"
ssl_host = "https://assets#{source.hash % 4}.github.com"
if request.ssl?
if source =~ /.js$/
ssl_host
elsif request.headers["USER_AGENT"] =~ /(Safari)/
non_ssl_host
else
ssl_host
end
else
non_ssl_host
end
end
repo = Grit::Repo.new(RAILS_ROOT)
js = repo.log('master', 'public/javascripts', :max_count => 1).first
css = repo.log('master', 'public/stylesheets', :max_count => 1).first
ENV['RAILS_ASSET_ID'] = js.committed_date > css.committed_date ? js.id : css.id
There’s three important things going on here.
First— If you hit a page using SSL, we serve all assets through SSL. If you’re on Safari, we send all CSS & images non-ssl since Safari doesn’t have a mixed content warning.
It is of note that many people suggest serving CSS & images non-ssl to Firefox. This was good practice when Firefox 2.0 was standard, but now that Firefox 3.0 is standard (and obeys cache-control:public as it should) there is no need for this hack. Firefox does have a mixed content warning (albeit not as prominent as IE), so I choose to use SSL.
Second— We’re serving assets out of 4 different servers. This fakes browsers into downloading things faster and is generally good practice.
Third— We’re hitting the git repo on the server (note our deployment setup) and getting a sha of the last changes to the public/stylesheets
and public/javascripts
directory. We use that sha as the ASSET_ID (the bit that gets tacked on after css/js files as ?sha-here).
This means that if we deploy a change that only affects app/application.rb
we don’t interrupt our user’s cache of the javascripts and stylesheets.
Conclusion
What all of this adds up to is that our deploys have almost no frontend consequence unless they intend to (changing css/js). This is huge for a site that does dozens of deploys a day. All browser caches remain the same and there isn’t any downtime while we bundle up assets. It also means we’re not afraid to deploy changes that may only affect one line of code and some minor feature.
All of this is not to say there isn’t room for improvement in our stack. I’m still tracking down some SSL bugs, and always trying to cut down on the total CSS, javascript and image load we deliver on every page.
Tags:
Written by
Related posts
Celebrating the GitHub Awards 2024 recipients 🎉
The GitHub Awards celebrates the outstanding contributions and achievements in the developer community by honoring individuals, projects, and organizations for creating an outsized positive impact on the community.
New from Universe 2024: Get the latest previews and releases
Find out how we’re evolving GitHub and GitHub Copilot—and get access to the latest previews and GA releases.
Bringing developer choice to Copilot with Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro, and OpenAI’s o1-preview
At GitHub Universe, we announced Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro, and OpenAI’s o1-preview and o1-mini are coming to GitHub Copilot—bringing a new level of choice to every developer.