Gregor is the maintainer of the JavaScript Octokit. He’s a seasoned open source maintainer with a passion for automating repetitive tasks and lowering the barrier for contributors of all kinds and backgrounds. Aside from Octokit, Gregor is a maintainer of Probot, nock, and semantic-release. Outside of work, he spends time taking care of his triplets Nico, Ada, and Kian. You can read more from Gregor on the DEV community and on Twitter.

@octokit/rest wasn’t originally created by GitHub. Instead, @bkeepers decided to adopt the package that was the most popular back in 2017: github. In this post, we’ll cover the story of @octokit/rest—the official JavaScript SDK for GitHub’s REST APIs.

The legacy

Later renamed to @octokit/rest, the github module was one of the oldest projects in the Node ecosystem, with its first commit from June 2010. That was the time of Node v0.1 and package.json didn’t exist because the npm registry was still being made.

In 2017, GitHub hired me to turn github into the official JavaScript SDK for GitHub’s APIs, for both Node.js and browsers. You can even see my first commit from September 2017. At the time, the project had close to 16 thousand lines of code (LOC) spread across three JavaScript files, one huge JSON file, and two files for TypeScript/Flow type definitions.

➜  rest.js git:(50720c8) wc -l lib/*  
     120 lib/error.js
    3246 lib/index.d.ts
     905 lib/index.js
    3232 lib/index.js.flow
      17 lib/promise.js
    7995 lib/routes.json
     143 lib/util.js
   15658 total

The adoption

The primary goal of the project was maintainability. The library’s core piece was a huge routes.json file with nearly eight thousand LOC, and it was supposed to define all of GitHub’s REST API endpoints. The file was maintained manually, and endpoints were only added or fixed once someone realized it was missing or incorrect (check out an example).

With this in mind, instead of manually maintaining the definitions in routes.json, I created a script to scrape GitHub’s REST API documentation nightly and turn it into a machine-readable JSON representation: octokit/routes. If the script found changes, @octokit/rest received a pull request to update the routes.json file. After merging, a new version was released automatically. Thanks to the automation, the routes.json file was now guaranteed to cover all of GitHub’s REST API endpoints, and it consists of 10,275 LOC. The accompanying TypeScript definitions grew to over 26,700 LOC.

The architecture

Once the completeness and maintainability were taken care of, I focused on another project goal: Decomposability.

The JavaScript Octokit is meant for all JavaScript runtime environments, some of which have strict constraints. For example, bundle size is significant for browser usage. Instead of a single, monolithic library which implements all REST API endpoints, authentication strategies, and recommended best practices (such as pagination), it’s important to give users the choice to use lower-level libraries. That way users can make their own choices about trade-offs between bundle size and features.

Here’s an overview of the architecture I came up with in January 2018:

The result of the internal refactoring into the new architecture looked like this:
Note that this output was simplified for readability.

➜  rest.js git:(f7c9f86) wc -l index.* lib/**/*.{js,json}
      31 index.js
    3474 index.d.ts
    3441 index.js.flow

     101 lib/endpoint/ # 4 files
     162 lib/request/ # 3 files

      83 lib/plugins/authentication/ # 3 files
     130 lib/plugins/endpoint-methods/ # 4 files
     130 lib/plugins/pagination/ # 11 files

      58 lib/parse-client-options.js
   10628 lib/routes.json

   18238 total

Over the next six months, I refactored the code and started extracting some of the modules:

• @octokit/endpoint: Turns API REST endpoint options into generic http request options
• @octokit/request: Sends parameterized requests to GitHub’s APIs with sensible defaults in browsers and Node
• before-after-hook: API used to hook into the request lifecycle

After using plugins internally for roughly six months, I announced the plugin API in November 2018 with version 16 and moved most of the library’s module to internal plugins that I could extract in the future.

The new internal code architecture was now looking like this:
Note that this output was simplified for readability:

➜  rest.js git:(01763bf) wc -l index.* plugins/**/*.{js,json} lib/**/*.js
      14 index.js
   26714 index.d.ts

     110 lib/ # 6 files

      86 plugins/authentication/ # 3 files
      77 plugins/pagination/ # 3 files
      39 plugins/register-endpoints/ # 3 files
     108 plugins/validate/ # 2 files
   10275 plugins/rest-api-endpoints/routes.json
   37423 total

Later, I created @octokit/core.js: the new Octokit JavaScript core library that @octokit/rest and all other Octokit libraries will be based on moving forward. Most of its logic was extracted from @octokit/rest, without all deprecated features. I didn’t use it right away within @octokit/rest in order to avoid breaking changes.

As @octokit/core was free of any legacy code and had a lower cost for breaking changes, I experimented with decomposing the different means of authentication, too. The results are separate packages for each authentication strategy—all listed in @octokit/auth‘s README. If you’d like to learn more about GitHub’s authentication strategies, check out my series on GitHub API Authentication.

@octokit/core and the separate authentication libraries would replace all the code in lib/* and plugins/authentication/*. All that was left were three plugins that I extracted into their own modules:

• @octokit/plugin-rest-endpoint-methods
• @octokit/plugin-paginate-rest
• @octokit/plugin-request-log

The validate plugin became obsolete thanks to TypeScript’s code validation at compile time. There was no longer a need for validating request parameters in the client. That resulted in a significant reduction of code and bundles size. For example, here’s the current definition for the octokit.checks.create() method:

{
  checks: {
    create: {
      headers: { accept: "application/vnd.github.antiope-preview+json" },
      method: "POST",
      params: {
        actions: { type: "object[]" },
        "actions[].description": { required: true, type: "string" },
        "actions[].identifier": { required: true, type: "string" },
        "actions[].label": { required: true, type: "string" },
        completed_at: { type: "string" },
        conclusion: {
          enum: [
            "success",
            "failure",
            "neutral",
            "cancelled",
            "timed_out",
            "action_required"
          ],
          type: "string"
        },
        details_url: { type: "string" },
        external_id: { type: "string" },
        head_sha: { required: true, type: "string" },
        name: { required: true, type: "string" },
        output: { type: "object" },
        "output.annotations": { type: "object[]" },
        "output.annotations[].annotation_level": {
          enum: ["notice", "warning", "failure"],
          required: true,
          type: "string"
        },
        "output.annotations[].end_column": { type: "integer" },
        "output.annotations[].end_line": { required: true, type: "integer" },
        "output.annotations[].message": { required: true, type: "string" },
        "output.annotations[].path": { required: true, type: "string" },
        "output.annotations[].raw_details": { type: "string" },
        "output.annotations[].start_column": { type: "integer" },
        "output.annotations[].start_line": { required: true, type: "integer" },
        "output.annotations[].title": { type: "string" },
        "output.images": { type: "object[]" },
        "output.images[].alt": { required: true, type: "string" },
        "output.images[].caption": { type: "string" },
        "output.images[].image_url": { required: true, type: "string" },
        "output.summary": { required: true, type: "string" },
        "output.text": { type: "string" },
        "output.title": { required: true, type: "string" },
        owner: { required: true, type: "string" },
        repo: { required: true, type: "string" },
        started_at: { type: "string" },
        status: { enum: ["queued", "in_progress", "completed"], type: "string" }
      },
      url: "/repos/:owner/:repo/check-runs"
    }
  }
}

Starting with v17, the definition looks like this 😎:

{
  checks: {
    create: [
      "POST /repos/{owner}/{repo}/check-runs",
      { mediaType: { previews: ["antiope"] } }
    ]
  }
}

Finally, all I needed to do was to put the previously extracted code back together. As promised, 10 LOC:

import { Octokit as Core } from "@octokit/core";
import { requestLog } from "@octokit/plugin-request-log";
import { paginateRest } from "@octokit/plugin-paginate-rest";
import { restEndpointMethods } from "@octokit/plugin-rest-endpoint-methods";

import { VERSION } from "./version";

export const Octokit = Core
  .plugin([requestLog, paginateRest, restEndpointMethods])
  .defaults({ userAgent: `octokit-rest.js/${VERSION}` });

The tests

Every single line of code was changed between version 16 and the upcoming version 17 of @octokit/rest. The only way to be confident that no new bugs were introduced is to run extensive tests.

When adopting the module back in 2017, no tests existed for our use case, but there were usage examples. The first thing I did was turn these usage examples into integration tests. And because the JavaScript Octokit SDK was meant to be the start of a suite of SDKs across all popular languages, I created octokit/fixtures—a language-agnostic, auto-updating set of http mocks for common use cases.

For the remaining logic specific to @octokit/rest, I created integration tests until we reached 100% test coverage. The tests will fail today if coverage drops below that.

While working on the migration to version 17 with its 10 LOC, I continued to run the tests of version 16 against the new version, with the exception of tests for deprecated APIs. At the same time, too many tests aren’t helpful either. Once I got all tests to pass for version 17, I reviewed all existing tests and removed any that no longer belong in @octokit/rest. Some of the tests were moved into the plugins, @octokit/core or @octokit/request. Now, all that’s left are a few smoke tests and the scenario tests using @octokit/fixtures.

The future

@octokit/rest used to be the Node.js library for GitHub’s REST APIs. Starting with v17, it will be the JavaScript library implementing all best practices for the @octokit/rest APIs, including pagination, throttling and automated request retries. It will support all existing and future authentication strategies and even GraphQL requests, as that is part of @octokit/core.

And finally, we’d like to say thanks to Fabian Jakobs, Mike de Boer, and Joe Gallo who created and maintained the github module before it became @octokit/rest.

Written by

Gregor Martynus

@gr2m