The problem

While developing a package that is used by clients on different versions of a product and php, I encountered the problem of GitHub API limits during functional testing.

To test changes in pull requests, I use Jenkins with a large test matrix. The number of tests, different versions of the product, plus different versions of php - we get a really large matrix of functional tests.

After the next addition of a new version of the product to the matrix, the tests began to fall randomly with the error “Could not authenticate against github.com”.

Why does it happen?

The point is that I pull the tested package and a couple of its dependencies from GitHub.

Jenkins at one of the stages generates the following composer.json, substituting the needed version of the product and php:

{
  "name": "vendor/tested-package",
  "description": "The example",
  "version": "1.2.3",
  "repositories": {
    "tested-package-repo": {
      "type": "vcs",
      "url": "https://github.com/vendor/tested-package.git"
    },
    "dependency-1": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-1.git"
    },
    "dependency-2": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-2.git"
    },
    "dependency-3": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-3.git"
    },
    "composer-repo.vendor.org": {
      "type": "composer",
      "url": "composer-repo.vendor.org"
    }
  },
  "require": {
    "php": "7.2",
    "vendor/product": "1.10.22",
    "vendor/tested-package": "dev-develop as 1.2.3",
    "vendor/dependency-1": "dev-develop as 2.0.1",
    "vendor/dependency-2": "dev-develop as 2.0.2",
    "vendor/dependency-3": "dev-develop as 2.0.3"
  }
}

In the next stage, Jenkins runs the composer update command and then runs the functional test.

Composer uses the GitHub API to get information from repositories. My limit is 5000 API requests per hour. But considering the number of tests, product versions and php - I started to get beyond this limit.

How to check GitHub API limits?

To check how many GitHub API requests you have left, you can run the following command in the console:

curl -H "Accept: application/vnd.github+json" -H "Authorization: token <your-token>" https://api.github.com/rate_limit

<your-token> needs to be replaced with your GitHub token that you use in the auth.json file.

The example of a response from GitHub:

{
  "resources": {
    "core": {
      "limit": 5000,
      "used": 5001,
      "remaining": 0,
      "reset": 1662134609
    },
    "search": {
      "limit": 30,
      "used": 0,
      "remaining": 30,
      "reset": 1662133328
    },
    "graphql": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1662136868
    },
    "integration_manifest": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1662136868
    },
    "source_import": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1662133328
    },
    "code_scanning_upload": {
      "limit": 1000,
      "used": 0,
      "remaining": 1000,
      "reset": 1662136868
    },
    "actions_runner_registration": {
      "limit": 10000,
      "used": 0,
      "remaining": 10000,
      "reset": 1662136868
    },
    "scim": {
      "limit": 15000,
      "used": 0,
      "remaining": 15000,
      "reset": 1662136868
    },
    "dependency_snapshots": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1662133328
    }
  },
  "rate": {
    "limit": 5000,
    "used": 5001,
    "remaining": 0,
    "reset": 1662134609
  }
}

We are interested in this part:

...
"rate": {
    "limit": 5000,
    "used": 5001,
    "remaining": 0,
    "reset": 1662134609
}
...
  • limit - how many requests are given per hour
  • used - how many requests were used
  • remaining - how many requests are left
  • reset - time when reset will occur

In this example, you can see that the limit has been reached.

Solving the problem with the GitHub API limits

Using no-api

One way is to add the “no-api” option to each GitHub repo in composer.json:

{
  "name": "vendor/tested-package",
  "description": "The example",
  "version": "1.2.3",
  "repositories": {
    "tested-package-repo": {
      "type": "vcs",
      "url": "https://github.com/vendor/tested-package.git",
      "no-api": true
    },
    "dependency-1": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-1.git",
      "no-api": true
    },
    "dependency-2": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-2.git",
      "no-api": true
    },
    "dependency-3": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-3.git",
      "no-api": true
    },
    "composer-repo.vendor.org": {
      "type": "composer",
      "url": "composer-repo.vendor.org"
    }
  },
  "require": {
    "php": "7.2",
    "vendor/product": "1.10.22",
    "vendor/tested-package": "dev-develop as 1.2.3",
    "vendor/dependency-1": "dev-develop as 2.0.1",
    "vendor/dependency-2": "dev-develop as 2.0.2",
    "vendor/dependency-3": "dev-develop as 2.0.3"
  }
}

Using this option composer does not use the GitHub API to get information from the repository, instead it clones the repository and reads the needed information from the local repository.

This approach solves the problem with API limits, but it can take 2-3 times longer to run tests, because cloning is slower than using the API.

Using your own composer repository

You can set up your own composer repository on the Jenkins machine and use it to store tested packages and their dependencies.

What I’ve done.

Before the composer.json generation stage, I added one more satage - packaging and publishing the tested package and its dependencies to my composer repository.

After these changes composer.json looks like this:

{
  "name": "vendor/tested-package",
  "description": "The example",
  "version": "1.2.3",
  "repositories": {
    "test-repo": {
      "type": "composer",
      "url": "127.0.0.1"
    }
    "composer-repo.vendor.org": {
      "type": "composer",
      "url": "composer-repo.vendor.org"
    }
  },
  "require": {
    "php": "7.2",
    "vendor/product": "1.10.22",
    "vendor/tested-package": "1.2.3",
    "vendor/dependency-1": "2.0.1",
    "vendor/dependency-2": "2.0.2",
    "vendor/dependency-3": "2.0.3"
  }
}

The problem with GitHub API limits has been resolved without affecting the tests execution time.

To spin up your own composer repository, you can use, for example, Satis

If you have any other ideas to solve this problem - please, write in the comments.