Functional tests and GitHub API limits

The problem

While developing a package that is used by clients on different versions of a product and PHP, I encountered the problem of GitHub API limits during functional testing.

To test changes in pull requests, I use Jenkins with a large test matrix. With the number of tests, different versions of the product, and different versions of PHP, we get a really large matrix of functional tests.

After adding a new version of the product to the matrix, the tests began to fail randomly with the error “Could not authenticate against github.com”.

Why does it happen?

The point is that I pull the package under test and a couple of its dependencies from GitHub.

During one of the stages, Jenkins generates the following composer.json by substituting the necessary version of the product and PHP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


{
  "name": "vendor/tested-package",
  "description": "The example",
  "version": "1.2.3",
  "repositories": {
    "tested-package-repo": {
      "type": "vcs",
      "url": "https://github.com/vendor/tested-package.git"
    },
    "dependency-1": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-1.git"
    },
    "dependency-2": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-2.git"
    },
    "dependency-3": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-3.git"
    },
    "composer-repo.vendor.org": {
      "type": "composer",
      "url": "composer-repo.vendor.org"
    }
  },
  "require": {
    "php": "7.2",
    "vendor/product": "1.10.22",
    "vendor/tested-package": "dev-develop as 1.2.3",
    "vendor/dependency-1": "dev-develop as 2.0.1",
    "vendor/dependency-2": "dev-develop as 2.0.2",
    "vendor/dependency-3": "dev-develop as 2.0.3"
  }
}

In the next stage, Jenkins runs the composer update command and then runs the functional test.

Composer uses the GitHub API to get information from repositories. My limit is 5000 API requests per hour. But considering the number of tests, product versions, and PHP - I started to go beyond this limit.

How to check GitHub API limits?

To check how many GitHub API requests you have left, you can run the following command in the console:

1

curl -H "Accept: application/vnd.github+json" -H "Authorization: token <your-token>" https://api.github.com/rate_limit

<your-token> needs to be replaced with your GitHub token that you use in the auth.json file.

An example of a response from GitHub:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64


{
  "resources": {
    "core": {
      "limit": 5000,
      "used": 5001,
      "remaining": 0,
      "reset": 1662134609
    },
    "search": {
      "limit": 30,
      "used": 0,
      "remaining": 30,
      "reset": 1662133328
    },
    "graphql": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1662136868
    },
    "integration_manifest": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1662136868
    },
    "source_import": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1662133328
    },
    "code_scanning_upload": {
      "limit": 1000,
      "used": 0,
      "remaining": 1000,
      "reset": 1662136868
    },
    "actions_runner_registration": {
      "limit": 10000,
      "used": 0,
      "remaining": 10000,
      "reset": 1662136868
    },
    "scim": {
      "limit": 15000,
      "used": 0,
      "remaining": 15000,
      "reset": 1662136868
    },
    "dependency_snapshots": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1662133328
    }
  },
  "rate": {
    "limit": 5000,
    "used": 5001,
    "remaining": 0,
    "reset": 1662134609
  }
}

We are interested in this part:

1
2
3
4
5
6
7
8


...
"rate": {
    "limit": 5000,
    "used": 5001,
    "remaining": 0,
    "reset": 1662134609
}
...

limit - how many requests are given per hour
used - how many requests were used
remaining - how many requests are left
reset - time when the reset will occur

In this example, you can see that the limit has been reached.

Solving the problem with GitHub API limits

Using no-api

One way is to add the no-api option to each GitHub repo in composer.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
  "name": "vendor/tested-package",
  "description": "The example",
  "version": "1.2.3",
  "repositories": {
    "tested-package-repo": {
      "type": "vcs",
      "url": "https://github.com/vendor/tested-package.git",
      "no-api": true
    },
    "dependency-1": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-1.git",
      "no-api": true
    },
    "dependency-2": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-2.git",
      "no-api": true
    },
    "dependency-3": {
      "type": "vcs",
      "url": "https://github.com/vendor/dependency-3.git",
      "no-api": true
    },
    "composer-repo.vendor.org": {
      "type": "composer",
      "url": "composer-repo.vendor.org"
    }
  },
  "require": {
    "php": "7.2",
    "vendor/product": "1.10.22",
    "vendor/tested-package": "dev-develop as 1.2.3",
    "vendor/dependency-1": "dev-develop as 2.0.1",
    "vendor/dependency-2": "dev-develop as 2.0.2",
    "vendor/dependency-3": "dev-develop as 2.0.3"
  }
}

Using this option, composer does not use the GitHub API to get information from the repository; instead, it clones the repository and reads the needed information from the local repository.

This approach solves the problem with API limits, but it can take 2-3 times longer to run tests because cloning is slower than using the API.

Using your own composer repository

You can set up your own composer repository on the Jenkins machine and use it to store tested packages and their dependencies.

What I’ve done.

Before the composer.json generation stage, I added one more stage - packaging and publishing the tested package and its dependencies to my composer repository.

After these changes, composer.json looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
  "name": "vendor/tested-package",
  "description": "The example",
  "version": "1.2.3",
  "repositories": {
    "test-repo": {
      "type": "composer",
      "url": "127.0.0.1"
    }
    "composer-repo.vendor.org": {
      "type": "composer",
      "url": "composer-repo.vendor.org"
    }
  },
  "require": {
    "php": "7.2",
    "vendor/product": "1.10.22",
    "vendor/tested-package": "1.2.3",
    "vendor/dependency-1": "2.0.1",
    "vendor/dependency-2": "2.0.2",
    "vendor/dependency-3": "2.0.3"
  }
}

The problem with GitHub API limits has been resolved without affecting the tests’ execution time.

To spin up your own composer repository, you can use, for example, Satis

If you have any other ideas to solve this problem - please, write in the comments.