Managing Github: via Terraform to a custom Ansible solution

We have 350+ people and 400+ repositories on Github. Each turnip can have several admins, and they do what they see fit – naturally, it happens that one person does not know what the other is doing. When we got tired of looking at the torment of others in the infrastructure and adding / removing people manually, we decided that we would go to centralized management, Infrastructure as Code.

And we chose Terraform as the platform.

“I have cubes with the letters O, P, A …”

Everything looked smooth on paper. Terraform is popular, it shouldn’t be hard to find people who know it. It has a state, and TF brings resources to match – we can always be sure that the real configuration is exactly as described. And there is no need to climb the Web UI anymore – I looked at the config and saw everything.

We hit the github limits. TF reads first all, and then changes what you want. With our size, this took about 20 minutes, and before the next change we had to wait an hour – we ran into the Github limits on the number of API calls.

To address constraint issues, we have divided the entire control into six parts:

1. Members of the organization.
2. Repositories.
3. Teams.
4. Composition of teams.
5. Command repositories.
6. Collaborators.

Typical operations are now performed in two rounds. To add a new developer, we launch Terraform with different parameters: 1 and 4. To add a new repository, we execute 2 and 5. It began to take quite a long time: launch TF once, come back in a few minutes, restart it. Return again – answer the author of the request that everything is done. Or not done, if somewhere in the config or pull request an error crept in. Once they brought PR, where in several places instead of English c Russian was written from… It took a long time to catch …

And the syntax didn’t work. The description of anything is quite verbose. Here’s an example:

resource "github_membership" "membership_for_юзер" {
    username = "юзер"
    role     = "member"
}

resource "github_team" "team_команда" {
    name           = "команда"
    description    = ""
    privacy        = "closed"
    parent_team_id = "123456"
}

resource "github_team_membership" "team_команда_юзер_membership" {
    team_id  = "${data.terraform_remote_state.teams.team_команда_id}"
    username = "юзер"
    role     = "member"
}

resource "github_repository" "репа" {
    name          = "репа"
    description   = ""
    homepage_url  = ""
    has_projects  = false
    has_wiki      = true
    has_issues    = true
    has_downloads = true
    private       = true
    archived      = false
    topics        = ["yii", "school", "mobile"]
}

resource "github_team_repository" "team_команда_repo_репа" {
    team_id    = "${data.terraform_remote_state.teams.team_команда_id}"
    repository = "${data.terraform_remote_state.repos.repo_репа_name}"
    permission = "push"
}

resource "github_repository_collaborator" "репа_юзер_collaborator" {
    repository = "репа"
    username   = "юзер"
    permission = "admin"
}

Given that such things are usually copied and pasted, after which something change, you can easily do it something not change or remove unnecessary. Put a minus instead of an underline – and no one will notice. The cost of an error is minutes of waiting. Imagine debugging with minutes between iterations …

You cannot refer to resources by name, you can only by id. Resources – repositories – are described in one file, and variables with their id in another. Users and teams are similar. Also in different places are the parameters of the repository itself and the list of commands with access to it. Collaborators are somewhere in third place. A common question is who has access to this turnip? Try to put everything together.

The “looked at the config and saw everything” approach did not work. Command Repositories are a many-to-many relationship. Everything in one file, thousands of lines. How to sort such a list? By repositories? By commands? No way. New entries are added to the end, then to the middle. Collecting a complete list of who has access to a particular turnip is a separate task.

Months after TF was introduced, it was especially fun to know that a repository of some kind was made quietly manually. And when now it is necessary to give someone the rights, we cannot do it. Terraform doesn’t know anything about it! Of course, this problem can also be solved: delete the turnip and make it again using TF, or somehow reinitialize TF itself. But…

Christmas trees, needles, why is it so difficult!

Adding a person to an organization is just one API call. To give a command to the repository is the same. Finally, when Terraform just began to fall on the management of the composition of teams with the words that it wants to remove 800 resources, add 801 and for some reason cannot do it, we sat down to figure out how it could be in an ideal world.

• Changes are applied pointwise.
• Simple syntax, understandable without reading the documentation. Without further ado like resource, value and without type identifiers 123456, which is not clear where to get it.
• All parameters of an entity – for example, a repository – are described in one place.
• One turnip / group / organization – one file.

Translated to YAML

Organization

skyeng:
  name: Skyeng
  admin:
    - aleksandr.sergeich

  member:
    - andrey.vadimych
    - denis.andreich
    - mikhail.leonidych
    - vladimir.nickolaich

Group

qa-team:
  privacy: secret

  maintainer:
    - denis.andreich

  member:
    - andrey.vadimych
    - mikhail.leonidych
    - vladimir.nickolaich

Repository

alerta:
  description: >-
    Alerta monitoring system
  homepage: https://alerta.io

  teams:
    admin:
      - admin-team

    push:
      - dev-team
      - qa-team

  collaborators:
    direct:
      - denis.andreich

    outside:
      - william.shakespeare

I wanted to use a ready-made solution, but did not find it – and wrote our own

Probably TF is a cool thing, but for us it turned out to be a Procrustean bed … And we went to implement our own vision in Ansible, which we actively use to manage infrastructure.

It works in a matter of seconds: typical changes are just a few calls, for large projects – several dozen. Easy to do CI / CD. Parameters of something are collected in one file: changes are local, they are easy to track. And now it really turns out to look into the file and see everything. The link to the code is at the end, but for now a couple of examples.

Now you can create a new repository or update an existing one like this:

ansible-playbook gitwand.yml
    -e github_repos__state=present
    -e github_repos__include=my_repo

To do something with the group – like this:

ansible-playbook gitwand.yml
    -e github_teams__state=present
    -e github_teams__include=my_team

If you need to update all groups, then we simply do not specify the parameter github_teams__include…

And a little side effect. We have LDAP, and wherever possible, we take users and user groups from it. A person’s login consists of a first and last name, and this is better than a nickname invented by someone in a stormy youth and understandable only to its owner. Now we have the opportunity to use the same names instead of nicknames on Github.

If you want to try our solution

It lies here…