Taking control of TODOs in PHP code

The problem of the complexity of managing TODO/FIXME comments in code is very old. It is more than 50 years old. It got to the point that some projects began to prohibit them with the words: “either fix it right away or don’t create garbage”. Here is how you can automate their management and turn garbage into a useful tool.

TL;DR;

1. We take a tool that will register new TODO/FIXME in the issue/bug tracker. For example, in Jira.

2. We take a tool that will remind you to remove such comments from the code when the tasks associated with them are completed.

3. We apply them to different scenarios of using these tools.

Now for the details.

Introduction / Problems

Every more or less experienced developer has either encountered TODO/FIXME and similar notes in the code or written them themselves (for the sake of brevity, we will simply call them TODO). Sometimes you come to a project and there are 100500 such notes. Some are already 5+ years old and no one does anything with them. Most of them have already been forgotten.

Over the decades since the problem emerged, many automation tools have emerged. Now we have CI and various code quality control tools. It would be a sin not to take advantage of this. You can set up the transfer of new TODOs from the code to a bug tracking system, for example, JIRA. Then set up control over the removal of these marks from the code when the tasks are completed.

Someone might object: “it will be making noise and you will get a lot of garbage tickets”. To which I want to clarify. This is expected behavior. Moreover, it is desired.

Let's imagine a situation. A developer submits a merge request with 10+ new TODOs. The merge request is reviewed and everyone agrees that the identified problems are in the code, they need to be fixed, but not now… Who should remember about this? And it's okay if there is a team lead on the project and he will remember these problems for some time and, perhaps, set a task. But what if the process for accepting merge requests is built in such a way that they can be accepted without a team lead? Or there is no team lead on the project, but there is a project manager who does not look at the code? In any case, no matter who manages the project, it is necessary to allocate resources to perform the work, plan it, and link it to the implementation of business tasks. This means that a ticket needs to be created anyway. So let them automatically fly into the ticket tracking system in the backlog. With configured labels, components, priority, assigned to the one who should disassemble them, and other goodies. And then the magic of management will happen with them.

Another issue is implementing this approach on old projects with a huge number of old and possibly outdated notes. More on that later.

By implementing this approach, we can start stimulating developers to create TODO in the code. Why? It's simple. For example, a business asks not to spend resources on writing tests for a while. Or to implement the simplest and fastest solution, even if it's crooked. And the developer, in the process of implementation, sees a block of code that requires refactoring. Or only positive flow processing is intentionally implemented. He warns the business that the tech debt will grow and boldly places TODO in the code. Then, when the time comes to deal with the tech debt, we talk to the business not about an abstract tech debt, the size of which no one really knows, but about specific tickets with a description, links to parent tickets, estimates and priorities. And this, you must agree, is a completely different conversation.

Registering TODO in the bug tracker

To my surprise, I didn't find a ready-made tool for transferring TODOs from code to a bug tracker. I had to write. And I did this with an eye on the chosen tool for monitoring completed TODOs (more on that below).

With a ready-made tool this is done simply:

1. Installed.

2. Configured.

3. Added a job to CI.

Below, a little more detail on each point.

Installation

Installation is simple. Add the package “aeliot/todo-registrar” in dev-dependency composer. Later it is planned to pack into PHAR, so that there will be fewer problems with dependencies in projects. An article about this is planned a little later.

Configuration

Create the file “.todo-registrar.dist.php” in the root of the project. It can be in any other place, but then when calling the script you will need to pass the path to the config. This file must return an instance of the class \Aeliot\TodoRegistrar\Config.

An example can be seen in the package root.

Basic config

The basic config is quite simple:

1. You must specify an array of tags by which TODO will be searched, if it differs from the basic set. By default, TODO and FIXME support is configured. However, you can configure tracking of any of your own tags. Tag search is case-insensitive.

2. Pass the configured Finder, responsible for searching files. If you have ever configured PHP CS Fixer – here is identical.

3. Specify the type of bug tracker and configs for it. Instead of an enum with the selected registrar type, you can pass your registrar factory. The interface there is as simple as possible. This greatly simplifies the use of custom solutions.

Issue tracker specific settings

The configuration of the recorder itself is a little more complicated.

At the moment, of the popular bug/issue trackers, only JIRA is supported. The rest are planned for development. If you wish, you can help with this.

Here you need to specify:

1. JIRA API connection parameters. Authorization is possible both by token (recommended) and using login and password.

2. project code in JIRA

3. general ticket settings. Such as:

   • Type of ticket being created

   • Labels common to all tickets

   • Components common to all tickets

   • Priorities, etc.

Fine-tuning the tickets created

There are several tools here to customize different aspects.

Firstlyusing multi-line comments allows you to highlight the title and body of tickets.

Secondlytracker-specific settings. For example, the JIRA logger allows you to specify labels, components, priority, and type of tickets to create.

Thirdyou can define assignee, both in the main registrar config, and through the @ symbol immediately after the tag (// TODO@my.assignee: ... ).

Fourthlyyou can have multiple configs for different parts of the code or for different tags.

Fifthlyinline config. You can directly in the text of the comment add or redefine various ticket parameters, applicable to a specific TODO.

Inline config

An interesting feature of the component is the inline config. It allows you to redefine the basic ticket settings and/or supplement them directly in the comment text. For example, with labels or links to other tickets. Specify components. Redefine the type of ticket being created and its priority. Almost anything you want. A very flexible tool.

Now implemented “EXTRAS”. This is a simplified format similar to js-object or JSON only without quotes, containing the root field “EXTRAS”. Covers most needs in customization.

The system expects no more than one inline config per TODO.

{EXTRAS: {linkedIssues: XX-123}}

{EXTRAS: {linkedIssues: [XX-123, XX-234]}}

{EXTRAS: {linkedIssues: {child_of: XX-123, is_blocked_by: [XX-234, YY-567]}}}

{EXTRAS: {labels: [label-a, label-b], components: [component-a, component-b]}}

As can be seen from the examples, fields that assume the presence of multiple values ​​can be passed either an array of values ​​or a single string.

For linked tickets, you can specify the binding type. Only due to the inline config format limitation, they must be one word. Therefore, all symbols except Latin letters and numbers are changed to an underscore.

The order of applying configs

1. @assignee in tag bindings.

2. EXTRAS

3. Then the general JIRA registrar config.

Setting up a job on CI

This is a very important point. Whether the IDs of the created tickets will be saved in the code depends on the correctness of its settings. And whether there will be duplicate tickets.

Scheme

First, we select one branch in which we will monitor the appearance of new TODO in the code. This should be a fairly stable branch in which development is not carried out. At the same time, it should be as close to development as possible, so that the tech debt gets into the accounting as early as possible. As a rule, this is “develop”. You should not use release branches. Several release branches can exist in parallel, and tickets can migrate from release to release at the request of the business. This can lead to the re-creation of tickets under different numbers and merge conflicts associated with this.

Secondly, after running the script, if there are changes in the project code, you should create a new branch using a template. For example, “todo-registrar” or “todo-registrar-”. And commit the changes there. This is important when re-running. See below.

Third, create a merge request from the new branch to the controlled one.

Restart

The previous points describe creating commits and merge requests, but there is something else you need to do to avoid creating duplicate tickets. Before running the main script, it is important to check for at least one open merge request in the controlled branch from the branch created according to the selected template. If there is one, you should skip the step of finding new TODOs and registering them.

Don't worry about something “not getting through” because of an open merge request. Merging it will trigger a re-check of the branch.

Example for GitLab

The system was tested on Gitlab. For Github it is done similarly. The main thing is to follow the scheme given above.

'TODO registrar':
    #...
    allow_failure: true
    script:
        - |
            export MR_TARGET_BRANCH=develop
            export TR_NEW_BRANCH=todo-registrar-$(echo "$RANDOM$RANDOM$RANDOM" | base64 | head -c 8; echo)
            export TR_MR_EXISTS=$(bash scripts/mr_check_existing.sh $GITLAB_TOKEN $MR_TARGET_BRANCH "todo-registrar-[[:alnum:]]{8}")
            echo "Matching of existing result: $TR_MR_EXISTS"
            if [[ "0" == "$TR_MR_EXISTS" ]]; then
                echo "NO opened MR exists!"
                git config --global user.email "cibot.pluscard@aventusit.eu"
                git config --global user.name "Pluscard CI Bot"
                git config remote.origin_ci.url >&- || git remote add origin_ci https://oauth2:${GITLAB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git
                git checkout -b $TR_NEW_BRANCH
                vendor/bin/todo-registrar
                git add .
                export TR_TITLE="TODO-REGISTRAR: automated registering of new TODOs "
                export TR_COMMITED=$(git commit -m "$TR_TITLE")
                if [[ $TR_COMMITED ]]; then
                    git push origin_ci $TR_NEW_BRANCH
                    bash scripts/mr_create.sh $GITLAB_TOKEN $MR_TARGET_BRANCH "$TR_NEW_BRANCH" "$TR_TITLE"
                else
                    echo "No changes, nothing to commit!"
                fi
            else
                echo "Registering of TODOs is skip case opened MR exists!"
            fi
    only:
        - develop

For it to work, you need to create two more scripts.

#!/bin/bash

if [ $# -ne 3 ]; then
  echo "Usage: $0 <MR_PRIVATE_TOKEN> <MR_TARGET_BRANCH> <MR_SOURCE_BRANCH>"
  exit 1
fi

PRIVATE_TOKEN=$1
PROJECT_ID=${CI_PROJECT_ID}
SOURCE_BRANCH=$3
TARGET_BRANCH=$2
GITLAB_API_URL="https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests"
MR_EXIST=0
REGEX="\"source_branch\":\s*\"$SOURCE_BRANCH\""

# shellcheck disable=SC2034
for page in {1..15}
do
  MR_LIST=$(curl -sS --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "$GITLAB_API_URL?target_branch=$TARGET_BRANCH&state=opened&page=$page")

  if [ "$MR_LIST" == "[]" ]; then
    break
  fi
  if [[ "$MR_LIST" =~ $REGEX ]]; then
    MR_EXIST=1
    break
  fi
done

echo $MR_EXIST

#!/bin/bash

if [ $# -ne 4 ]; then
  echo "Usage: $0 <MR_PRIVATE_TOKEN> <MR_TARGET_BRANCH> <MR_SOURCE_BRANCH> <MR_TITLE>"
  exit 1
fi

GITLAB_API_URL="https://gitlab.com/api/v4"
PRIVATE_TOKEN=$1
PROJECT_ID=${CI_PROJECT_ID}
SOURCE_BRANCH=$3
TARGET_BRANCH=$2
TITLE=$4

curl -sS --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "$GITLAB_API_URL/projects/$PROJECT_ID/merge_requests" \
  --form "source_branch=$SOURCE_BRANCH" \
  --form "target_branch=$TARGET_BRANCH" \
  --form "title=$TITLE" \
  --form "description=Automated merge request from CI/CD pipeline"

Control over the removal of irrelevant TODO

The plan is as follows:

1. We put the package staabm/phpstan-todo-by

2. The package provides a rule for PHPStan. If it is not installed, then install it.

3. Add a rule from this package to the PHPStan config.

4. We configure the connection to the issue tracker (JIRA) in the PHPStan config.

5. Setting up a job on Gitlab CI.

If you use a lot of rules for PHPStan and the job has been running for a long time, it may be useful to set up a separate job that will only run the rules of this package. This can speed up the pipeline execution due to parallel processes.

This job can be run on any branch. And it should be allowed to fail. Typically, the ticket for which the code is written is closed after passing the review and testing, and at this stage of the open merge request, the script cannot control the closure of the ticket, since it is still usually open.

Limitation of the selected package:

In the current implementation, this package controls only 3 tags: TODO, FIXME, XXX. Therefore, the XXX tag should be used as a custom tag. Or, after registering tickets, change the custom tags to one of those supported by this component. Perhaps, with a script. Or contribute the ability to configure the tracked tags. Or use another solution. Fortunately, there are alternatives.

Use cases

Implementation into an old project

On older projects, some preliminary preparation may be required to avoid receiving too many tickets at once, sometimes junk ones.

Estimate how many TODO/FIXME you have in your project now. How painful it will be to start with default settings. Perhaps, during this analysis, you will delete some of the outdated comments. If it is still painful to start with what is left, then there are several scenarios:

1. Adding a suffix to existing tags. To turn TODO into TODO-OLD, or something similar. And immediately set up tracking of common tags. Later, you can remove this suffix from some comments so that they are processed by the script. Perhaps in several approaches.

2. Set up tracking for only one tag.

3. Set up custom tag tracking. For example, TODO-JIRA. And explain to all developers which tag to use. In any IDE, you can also set up custom tag highlighting. Limitations: There may be problems with doctrine/annotation if the @ sign is before the tag. You may need to set up doctrine to ignore custom tags.

4. Set up monitoring of only part of the project.

I recommend the first and second scenarios.

Conducting project analysis

It's become much more fun to review projects.

• Put the package in place.

• Configured.

• You sit, read the code and bomb the tutushki, using any convenient tag.

No need to switch between windows and fill in 100500 fields for each comment in the issue tracker. Everything will fly into JIRA itself. Then, using the ticket number, it will be easy to find the place that requires attention.

The work of developers has been simplified

Now, when faced with the need to change something, the developer does not suffer with the question “is it possible to fix this within the framework of my task? And how?” For some, this takes a colossal amount of time. Now, set TODO and move on. And you can be sure that no one will forget about this task. That is, the developer is not completely relieved of responsibility for the quality of the code, but the pressure is reduced. Therefore, tickets are implemented faster.

Tracking technical debt

Now it doesn't matter for what reason the developer leaves tech debt. Whether the business asked to speed up or at his own request. You just leave comments in the code with a link to the current task in the inline config and related tickets for each item fly into Jira.

Post scriptum

I am very interested in similar solutions for tracking and registering TODO in files: yaml, twig, html, xml, javascript, css, less, flatter. I would be grateful for links to them.

Fresh ideas, comments and contributors to the project TODO Register are also welcome.