How GitHub converts previously encrypted and unencrypted columns to ActiveRecord encrypted columns
This post is the second part in a series about ActiveRecord::Encryption that shows how GitHub upgrades previously encrypted and unencrypted columns to ActiveRecord::Encryption.
![](https://github.blog/wp-content/uploads/2022/11/Group-4-1.png?resize=1600%2C850)
Background
In the first post in this series, we detailed how we designed our easy‐to‐use column encryption paved path. We found during the rollout that the bulk of time and effort was spent in robustly supporting the reading and upgrading of previous encryption formats/plaintext and key rotation. In this post, we’ll explain the design decisions we made in our migration plan and describe a simplified migration pattern you can use to encrypt (or re-encrypt) existing records in your Rails application.
We have two cases for encrypted columns data migration–upgrading plaintext or previously encrypted data to our new standard and key rotation.
Upon consulting the Rails documentation to see if there was any prior art we could use, we found the previous encryptor strategy but exactly how to migrate existing data is, as they say, an “exercise left for the reader.”
Dear reader, lace up your sneakers because we are about to exercise. 👟
To convert plaintext columns or columns encrypted with our deprecated internal encryption library, we used ActiveRecord::Encryption
’s previous encryptor strategy, our existing feature flag mechanism and our own type of database migration called a transition. Transitions are used by GitHub to modify existing data, as opposed to migrations that are mainly used to add or change columns. To simplify things and save time, in the example migration strategy, we’ll rely on the Ruby gem, MaintenanceTasks.
Previous encryptor strategy
ActiveRecord::Encryption
provides as a config option config.active_record.encryption.support_unencrypted_data
that allows plaintext values in an encrypted_attribute to be read without error. This is enabled globally and could be a good strategy to use if you are migrating only plaintext columns and you are going to migrate them all at once. We chose not to use this option because we want to migrate columns to ActiveRecord::Encryption
without exposing the ciphertext of other columns if decryption fails. By using a previous encryptor, we can isolate this “plaintext mode” to a single model.
In addition to this, GitHub’s previous encryptor uses a schema validator and regex to make sure that the “plaintext” being returned does not have the same shape as Rails encrypted columns data.
Feature flag strategy
We wanted to have fine-grained control to safely roll out our new encryption strategy, as well as the ability to completely disable it in case something went wrong, so we created our own custom type using the ActiveModel::Type API, which would only perform encryption when the feature flag for our new column encryption strategy was disabled.
A common feature flag strategy would be to start a feature flag at 0% and gradually ramp it up to 100% while you observe and verify the effects on your application. Once a flag is verified at 100%, you would remove the feature flag logic and delete the flag. To gradually increase a flag on column encryption, we would need to have an encryption strategy that could handle plaintext and encrypted records both back and forth because there would be no way to know if a column was encrypted without attempting to read it first. This seemed like unnecessary additional and confusing work, so we knew we’d want to use flagging as an on/off switch.
While a feature flag should generally not be long running, we needed the feature flag logic to be long running because we want it to be available for GitHub developers who will want to upgrade existing columns to use ActiveRecord::Encryption
.
This is why we chose to inverse the usual feature flag default to give us the flexibility to upgrade columns incrementally without introducing unnecessary long‐running feature flags. This means we set the flag at 100% to prevent records from being encrypted with the new standard and set it to 0% to cause them to be encrypted with our new standard. If for some reason we are unable to prioritize upgrading a column, other columns do not need to be flagged at 100% to continue to be encrypted on our new standard.
We added this logic to our monkeypatch of ActiveRecord::Base::encrypts
method to ensure our feature flag serializer is used:
Code sample 1
self.attribute(attribute) do |cast_type|
GitHub::Encryption::FeatureFlagEncryptedType.new(cast_type: cast_type, attribute_name: attribute, model_name: self.name)
end
Which instantiates our new ActiveRecord Type that checks for the flag in its serialize method:
Code sample 2
# frozen_string_literal: true
module GitHub
module Encryption
class FeatureFlagEncryptedType < ::ActiveRecord::Type::Text
attr_accessor :cast_type, :attribute_name, :model_name
# delegate: a method to make a call to `this_object.foo.bar` into `this_object.bar` for convenience
# deserialize: Take a value from the database, and make it suitable for Rails
# changed_in_place?: determine if the value has changed and needs to be rewritten to the database
delegate :deserialize, :changed_in_place?
, to: :cast_type
def initialize(cast_type:, attribute_name:, model_name:)
raise RuntimeError, "Not an EncryptedAttributeType" unless cast_type.is_a?(ActiveRecord::Encryption::EncryptedAttributeType)
@cast_type = cast_type
@attribute_name = attribute_name
@model_name = model_name
end
# Take a value from Rails and make it suitable for the database
def serialize(value)
if feature_flag_enabled?("encrypt_as_plaintext_#{model_name.downcase}_#{attribute_name.downcase}")
# Fall back to plaintext (ignore the encryption serializer)
cast_type.cast_type.serialize(value)
else
# Perform encryption via active record encryption serializer
cast_type.serialize(value)
end
end
end
end
end
A caveat to this implementation is that we extended from ActiveRecord::Type::Text
which extends from ActiveModel::Type:String, which implements changed_in_place?
by checking if the new_value
is a string, and, if it is, does a string comparison to determine if the value was changed.
We ran into this caveat during our roll out of our new encrypted columns. When migrating a column previously encrypted with our internal encryption library, we found that changed_in_place?
would compare the decrypted plaintext value to the encrypted value stored in the database, always marking the record as changed in place as these were never equal. When we migrated one of our fields related to 2FA recovery codes, this had the unexpected side effect of causing them to all appear changed in our audit log logic and created false-alerts in customer facing security logs. Fortunately, though, there was no impact to data and our authentication team annotated the false alerts to indicate this to affected customers.
To address the cause, we delegated the changed_in_place?
to the cast_type
, which in this case will always be ActiveRecord::Encryption::EncryptedAttributeType
that attempts to deserialize the previous value before comparing it to the new value.
Key rotation
ActiveRecord::Encryption
accommodates for a list of keys to be used so that the most recent one is used to encrypt records, but all entries in the list will be tried until there is a successful decryption or an ActiveRecord::DecryptionError is raised. On its own, this will ensure that when you add a new key, records that are updated after will automatically be re-encrypted with the new key.
This functionality allows us to reuse our migration strategy (see code sample 5) to re-encrypt all records on a model with the new encryption key. We do this simply by adding a new key and running the migration to re-encrypt.
Example migration strategy
This section will describe a simplified version of our migration process you can replicate in your application. We use a previous encryptor to implement safe plaintext support and the maintanence_tasks gem to backfill the existing records.
Set up ActiveRecord::Encryption
and create a previous encryptor
Because this is a simplified example of our own migration strategy, we recommend using a previous encryptor to restrict the “plaintext mode” of ActiveRecord::Encryption
to the specific model(s) being migrated.
Set up ActiveRecord::Encryption
by generating random key set:
bin/rails db:encryption:init
And adding it to the encrypted Rails.application.credentials using:
bin/rails credentials:edit
If you do not have a master.key, this command will generate one for you. Remember never to commit your master key!
Create a previous encryptor. Remember, when you provide a previous strategy, ActiveRecord::Encryption
will use the previous to decrypt and the current (in this case ActiveRecord’s default encryptor) to encrypt the records.
Code sample 3
app/lib/encryption/previous_encryptor.rb
# frozen_string_literal: true
module Encryption
class PreviousEncryptor
def encrypt(clear_text, key_provider: nil, cipher_options: {})
raise NotImplementedError.new("This method should not be called")
end
def decrypt(previous_data, key_provider: nil, cipher_options: {})
# JSON schema validation
previous_data
end
end
end
Add the previous encryptor to the encrypted column
Code sample 4
app/models/secret.rb
class Secret < ApplicationRecord
encrypts :code, previous: { encryptor: Encryption::PreviousEncryptor.new }
end
The PreviousEncryptor
will allow plaintext records to be read as plaintext but will encrypt all new records up until and while the task is running.
Install the Maintenance Tasks gem and create a task
Install the Maintenance Tasks gem per the instructions and you will be ready to create the maintenance task.
Create the task.
bin/rails generate maintenance_tasks:task encrypt_plaintext_secrets
In day‐to‐day use, you shouldn’t ever need to call secret.encrypt
because ActiveRecord handles the encryption before inserting into the database, but we can use this API in our task:
Code sample 5
app/tasks/maintenance/encrypt_plaintext_secrets_task.rb
# frozen_string_literal: true
module Maintenance
class EncryptPlaintextSecretsTask < MaintenanceTasks::Task
def collection
Secret.all
end
def process(element)
element.encrypt
end
…
end
end
Run the Maintenance Task
Maintenance Tasks provides several options to run the task, but we use the web UI in this example:
Verify your encryption and cleanup
You can verify encryption in Rails console, if you like:
And now you should be able to safely remove your previous encryptor leaving the model of your newly encrypted column looking like this:
Code sample 6
app/models/secret.rb
class Secret < ApplicationRecord
encrypts :code
end
And so can you!
Encrypting database columns is a valuable extra layer of security that can protect sensitive data during exploits, but it’s not always easy to migrate data in an existing application. We wrote this series in the hope that more organizations will be able to plot a clear path forward to using ActiveRecord::Encryption
to start encrypting existing sensitive values.
Tags:
Written by
Related posts
![](https://github.blog/wp-content/uploads/2024/08/1200x630-Blog-Banner-Light-Mode-2.png?resize=400%2C212)
The ultimate guide to developer happiness
Five actionable tips and strategies to supercharge developer happiness—and a more innovative workplace.
![A landscape image with a simple dark background featuring two security-themed shapes in the top left and bottom right corners. The foreground text is centered and reads "Strategies for supporting your neurodivergent employees."](https://github.blog/wp-content/uploads/2024/07/neurocats-header.png?resize=400%2C212)
How GitHub supports neurodiverse employees (and how your company can, too)
Teams with neurodivergent employees can be up to 30 percent more productive. Discover tips on how best to support them in your workplace.
![](https://github.blog/wp-content/uploads/2023/12/Productivity-DarkMode-2-1.png?resize=400%2C212)
How we improved availability through iterative simplification
Solving and staying ahead of problems when scaling up a system of GitHub’s size is a delicate process. Here’s a look at some of the tools in GitHub’s toolbox, and how we’ve used them to solve problems.