From 48k lines of code to 10—the story of GitHub’s JavaScript SDK
Learn about the legacy, architecture, and methods used to reduce 48k lines of code to 10 as we take a deep dive into GitHub’s Javascript SDK.
Gregor is the maintainer of the JavaScript Octokit. He’s a seasoned open source maintainer with a passion for automating repetitive tasks and lowering the barrier for contributors of all kinds and backgrounds. Aside from Octokit, Gregor is a maintainer of Probot, nock, and semantic-release. Outside of work, he spends time taking care of his triplets Nico, Ada, and Kian. You can read more from Gregor on the DEV community and on Twitter.
@octokit/rest
wasn’t originally created by GitHub. Instead, @bkeepers decided to adopt the package that was the most popular back in 2017: github. In this post, we’ll cover the story of @octokit/rest
—the official JavaScript SDK for GitHub’s REST APIs.
The legacy
Later renamed to @octokit/rest
, the github module was one of the oldest projects in the Node ecosystem, with its first commit from June 2010. That was the time of Node v0.1 and package.json didn’t exist because the npm registry was still being made.
In 2017, GitHub hired me to turn github into the official JavaScript SDK for GitHub’s APIs, for both Node.js and browsers. You can even see my first commit from September 2017. At the time, the project had close to 16 thousand lines of code (LOC) spread across three JavaScript files, one huge JSON file, and two files for TypeScript/Flow type definitions.
➜ rest.js git:(50720c8) wc -l lib/*
120 lib/error.js
3246 lib/index.d.ts
905 lib/index.js
3232 lib/index.js.flow
17 lib/promise.js
7995 lib/routes.json
143 lib/util.js
15658 total
The adoption
The primary goal of the project was maintainability. The library’s core piece was a huge routes.json file with nearly eight thousand LOC, and it was supposed to define all of GitHub’s REST API endpoints. The file was maintained manually, and endpoints were only added or fixed once someone realized it was missing or incorrect (check out an example).
With this in mind, instead of manually maintaining the definitions in routes.json, I created a script to scrape GitHub’s REST API documentation nightly and turn it into a machine-readable JSON representation: octokit/routes
. If the script found changes, @octokit/rest
received a pull request to update the routes.json file. After merging, a new version was released automatically. Thanks to the automation, the routes.json file was now guaranteed to cover all of GitHub’s REST API endpoints, and it consists of 10,275 LOC. The accompanying TypeScript definitions grew to over 26,700 LOC.
The architecture
Once the completeness and maintainability were taken care of, I focused on another project goal: Decomposability.
The JavaScript Octokit is meant for all JavaScript runtime environments, some of which have strict constraints. For example, bundle size is significant for browser usage. Instead of a single, monolithic library which implements all REST API endpoints, authentication strategies, and recommended best practices (such as pagination), it’s important to give users the choice to use lower-level libraries. That way users can make their own choices about trade-offs between bundle size and features.
Here’s an overview of the architecture I came up with in January 2018:
The result of the internal refactoring into the new architecture looked like this:
Note that this output was simplified for readability.
➜ rest.js git:(f7c9f86) wc -l index.* lib/**/*.{js,json}
31 index.js
3474 index.d.ts
3441 index.js.flow
101 lib/endpoint/ # 4 files
162 lib/request/ # 3 files
83 lib/plugins/authentication/ # 3 files
130 lib/plugins/endpoint-methods/ # 4 files
130 lib/plugins/pagination/ # 11 files
58 lib/parse-client-options.js
10628 lib/routes.json
18238 total
Over the next six months, I refactored the code and started extracting some of the modules:
@octokit/endpoint
: Turns API REST endpoint options into generichttp
request options@octokit/request
: Sends parameterized requests to GitHub’s APIs with sensible defaults in browsers and Nodebefore-after-hook
: API used to hook into the request lifecycle
After using plugins internally for roughly six months, I announced the plugin API in November 2018 with version 16 and moved most of the library’s module to internal plugins that I could extract in the future.
The new internal code architecture was now looking like this:
Note that this output was simplified for readability:
➜ rest.js git:(01763bf) wc -l index.* plugins/**/*.{js,json} lib/**/*.js
14 index.js
26714 index.d.ts
110 lib/ # 6 files
86 plugins/authentication/ # 3 files
77 plugins/pagination/ # 3 files
39 plugins/register-endpoints/ # 3 files
108 plugins/validate/ # 2 files
10275 plugins/rest-api-endpoints/routes.json
37423 total
Later, I created @octokit/core.js
: the new Octokit JavaScript core library that @octokit/rest
and all other Octokit libraries will be based on moving forward. Most of its logic was extracted from @octokit/rest
, without all deprecated features. I didn’t use it right away within @octokit/rest in order to avoid breaking changes.
As @octokit/core
was free of any legacy code and had a lower cost for breaking changes, I experimented with decomposing the different means of authentication, too. The results are separate packages for each authentication strategy—all listed in @octokit/auth
‘s README. If you’d like to learn more about GitHub’s authentication strategies, check out my series on GitHub API Authentication.
@octokit/core
and the separate authentication libraries would replace all the code in lib/* and plugins/authentication/*
. All that was left were three plugins that I extracted into their own modules:
The validate plugin became obsolete thanks to TypeScript’s code validation at compile time. There was no longer a need for validating request parameters in the client. That resulted in a significant reduction of code and bundles size. For example, here’s the current definition for the octokit.checks.create()
method:
{
checks: {
create: {
headers: { accept: "application/vnd.github.antiope-preview+json" },
method: "POST",
params: {
actions: { type: "object[]" },
"actions[].description": { required: true, type: "string" },
"actions[].identifier": { required: true, type: "string" },
"actions[].label": { required: true, type: "string" },
completed_at: { type: "string" },
conclusion: {
enum: [
"success",
"failure",
"neutral",
"cancelled",
"timed_out",
"action_required"
],
type: "string"
},
details_url: { type: "string" },
external_id: { type: "string" },
head_sha: { required: true, type: "string" },
name: { required: true, type: "string" },
output: { type: "object" },
"output.annotations": { type: "object[]" },
"output.annotations[].annotation_level": {
enum: ["notice", "warning", "failure"],
required: true,
type: "string"
},
"output.annotations[].end_column": { type: "integer" },
"output.annotations[].end_line": { required: true, type: "integer" },
"output.annotations[].message": { required: true, type: "string" },
"output.annotations[].path": { required: true, type: "string" },
"output.annotations[].raw_details": { type: "string" },
"output.annotations[].start_column": { type: "integer" },
"output.annotations[].start_line": { required: true, type: "integer" },
"output.annotations[].title": { type: "string" },
"output.images": { type: "object[]" },
"output.images[].alt": { required: true, type: "string" },
"output.images[].caption": { type: "string" },
"output.images[].image_url": { required: true, type: "string" },
"output.summary": { required: true, type: "string" },
"output.text": { type: "string" },
"output.title": { required: true, type: "string" },
owner: { required: true, type: "string" },
repo: { required: true, type: "string" },
started_at: { type: "string" },
status: { enum: ["queued", "in_progress", "completed"], type: "string" }
},
url: "/repos/:owner/:repo/check-runs"
}
}
}
Starting with v17, the definition looks like this 😎:
{
checks: {
create: [
"POST /repos/{owner}/{repo}/check-runs",
{ mediaType: { previews: ["antiope"] } }
]
}
}
Finally, all I needed to do was to put the previously extracted code back together. As promised, 10 LOC:
import { Octokit as Core } from "@octokit/core";
import { requestLog } from "@octokit/plugin-request-log";
import { paginateRest } from "@octokit/plugin-paginate-rest";
import { restEndpointMethods } from "@octokit/plugin-rest-endpoint-methods";
import { VERSION } from "./version";
export const Octokit = Core
.plugin([requestLog, paginateRest, restEndpointMethods])
.defaults({ userAgent: `octokit-rest.js/${VERSION}` });
The tests
Every single line of code was changed between version 16 and the upcoming version 17 of @octokit/rest
. The only way to be confident that no new bugs were introduced is to run extensive tests.
When adopting the module back in 2017, no tests existed for our use case, but there were usage examples. The first thing I did was turn these usage examples into integration tests. And because the JavaScript Octokit SDK was meant to be the start of a suite of SDKs across all popular languages, I created octokit/fixtures
—a language-agnostic, auto-updating set of http
mocks for common use cases.
For the remaining logic specific to @octokit/rest
, I created integration tests until we reached 100% test coverage. The tests will fail today if coverage drops below that.
While working on the migration to version 17 with its 10 LOC, I continued to run the tests of version 16 against the new version, with the exception of tests for deprecated APIs. At the same time, too many tests aren’t helpful either. Once I got all tests to pass for version 17, I reviewed all existing tests and removed any that no longer belong in @octokit/rest
. Some of the tests were moved into the plugins, @octokit/core
or @octokit/request
. Now, all that’s left are a few smoke tests and the scenario tests using @octokit/fixtures
.
The future
@octokit/rest
used to be the Node.js library for GitHub’s REST APIs. Starting with v17, it will be the JavaScript library implementing all best practices for the @octokit/rest
APIs, including pagination, throttling and automated request retries. It will support all existing and future authentication strategies and even GraphQL requests, as that is part of @octokit/core
.
And finally, we’d like to say thanks to Fabian Jakobs, Mike de Boer, and Joe Gallo who created and maintained the github module before it became @octokit/rest
.
Written by
Related posts
GitHub Enterprise Cloud with data residency: How we built the next evolution of GitHub Enterprise using GitHub
How we used GitHub to build GitHub Enterprise Cloud with data residency.
The ultimate guide to developer happiness
Five actionable tips and strategies to supercharge developer happiness—and a more innovative workplace.
How GitHub supports neurodiverse employees (and how your company can, too)
Teams with neurodivergent employees can be up to 30 percent more productive. Discover tips on how best to support them in your workplace.