Legacy PHP-FPM on Kubernetes
Each scaling strategy has its pros and cons. Why not start combining them and leaving only the positives? Let’s talk about this – how to do it well when you can’t just bury it – in the context of PHP.
The article was written based on my report at Saint HighLoad++ 2023.
My name is Igor Titov, I’m a systems engineer at Garage Eight. I’ll tell you how we transferred a scary old monster – a legacy PHP-FPM application – to Kubernetes and how many hedgehogs we ate in the process.
The international IT product company Garage Eight has been developing an entire ecosystem of financial products and services in foreign markets for 11 years. By 2023, the company has hundreds of thousands of active users in 183 countries, as well as more than 300 microservices.
Start. What is PHP-FPM?
PHP-FPM (PHP FastCGI Process Manager) is simply a fast process manager for PHP. It was written in 2004 and is still supported today. It comes with 1000 and 1 setup guide. Separately, I would like to note the possibility of creating dynamic, static and “on-call” process pools, which can be used depending on the incoming load, the desired response time of the application, when necessary, reducing the amount of consumed computing resources, or vice versa – improving the user experience in the application due to its speed , which we actively used in the company. There are also many other settings, for example, query execution time, restrictions on the number of open files, the number of child processes, etc.
When working with PHP-FPM, it is possible to process statics on the balancer side, for example Nginx, and send dynamics to PHP-FPM.
It is on this process manager that our old PHP-FPM application runs.
Problems
Historically, at Garage Eight, microservices were deployed in “vanilla” Docker; we didn’t even use Docker Compose.
These were dozens of virtual servers with Docker, on which we raised microservice containers using Ansible and monitored their availability. Scaling was carried out through the deployment of new servers with Docker and reconfiguration of Nginx. The worst disadvantage was that traffic came in periodically, it was necessary to quickly expand. How did we do it?
Let’s imagine that we have a cloud load balancer and a pair of Nginx, with the help of which we send traffic to virtual machines with containers in which our application is running. These containers also have their own Nginx, which processes static data and transfers dynamics to PHP-FPM for further processing.
To expand, we added new virtual machines, deployed containers with PHP-FPM, Nginx on them and configured balancers.
System engineers had to oversee all this. They answered, incl. for assessing incoming traffic, monitoring it and processing rules.
The introduction of Kubernetes made it possible to greatly simplify the task of scaling.
But besides scaling there were other problems:
A container crash with Nginx did not cause a container crash with PHP-FPM. This also had to be monitored – additional metrics and rules had to be added to the monitoring system.
The application daemons were located in the same container and could desperately fight among themselves for computing power.
All the monoliths that were written at the startup stage had a problem with the fact that they deployed their own self-written Ansible roles. Deploying to Kubernetes solved these problems using the universal Helm.
The application was built right during deployment. It was necessary to download all the dependencies, assemble the image, build and launch the container.
Migrating a Legacy PHP-FPM application to Kubernetes
This is what migrating an application to Kubernetes looked like, in general terms.
1. Pool division
PHP-FPM pools were divided into different deployments:
RPC
PHP-Daemons,
API,
Static
etc.
Separately, it is worth noting the crowns. These were ordinary Linux crowns, which were stored in Crontab and launched according to a schedule. They had something like Docker exec and so on. When moving to Kubernetes, the developers rewrote them into Cronjobs and allocated a separate Node pool for them so that they did not affect other services in any way. Used for Cronjobs spec.concurrencyPolicyto control how many instances can be launched for each of them. By making crons the essence of Cronjobs, developers could immediately use them without additional tools. We received Cronjob results via Prometheus and quickly set up alerts for falling crowns in the alert manager.
2. Migrations
Migrations during the deployment of this monster of steel will be performed through hooks on pre-upgrade and pre-instal. Anyone familiar with Kubernetes knows that under the hood these are Kubernetes Jobs.
Please note that you need to properly configure timeouts for migrations, because migrations can be really long. Therefore, we increased the time using activeDeadlineSeconds.
3. Command execution console
Previously, we simply went into the Docker container and did something, now we have made a separate deployment with a container where you can go and do something in the cli. Everything is still simple, but done as a separate deployment.
4. Statics
Statics is also a separate deployment with a container that we built in advance, collected an image and placed it. Static requests are handled by Nginx, as I mentioned above.
What problems did you encounter?
AMQProxy
After migrating the application to Kubernetes under load, we almost managed to install RabbitMQ. It turned out that we forgot about AMQ-Proxy. PHP does not like to break connections with RabbitMQ, so each node must have an additional container with AMQP. We had to add it to the pods along with the legacy PHP application as a sidecar container.
It is necessary not only to add, but also to properly manage sidecar containers, because Kubernetes does not support the complex life cycle of sidecar containers. At that time, there was a big Merge Request, which said, now we will be able to support sidecar containers in Kubernetes v1.19: lifetime, order of start and stop, etc. In fact, the Merge Request was rejected, and the management of sidecar containers comes down to timeouts. We configured AMQP to stop only after PHP-FPM stopped, so as not to lose data in RabbitMQ.
502nd on production due to premature stopping of pods
We migrated the application and began periodically receiving 502s on production. The error occurred during deployment. It turned out that it was enough to add a small timeout to the preStop hook to make this problem go away. This is how our application responded that we needed to wait a little before stopping. As a result, the team came to the conclusion that we wait 5 seconds, then another 10, and only then execute the Graceful shutdown command for PHP-FPM.
We also forgot about internal timeouts:
It should be noted that these threshold values also affect the correct shutdown of pods with PHP-FPM.
In addition, there is a Graceful shutdown timeout for pods. By default it is 30 seconds. If your application does not stop within this time, then Kubernetes will disintegrate it. We have also increased this time.
As a result, there was a stepwise change in timeouts and we got this picture:
We just wait 15 seconds, confident that during this time our application will accurately process all remaining requests, this wait is less than the maximum request execution time in PHP, then we wait until PHP-FPM accurately completes all processes and only at the end we allow Kubernetes to destroy our under. In this case, we avoid confusion with timeouts; no one will kill the application earlier.
502 on production due to autoscaling
But the problems didn’t end there. Something went wrong and the 502 errors in production continued. The culprit was the Horizontal Pod Autoscaler (HPA):
We connected it in the process when we were solving problems with timeouts. It allows the application to quickly scale under load.
At first glance, in its configuration indicated above, everything is simple, but let’s imagine that Horizontal Pod Autoscaler expanded the deployment to 60 replicas. During deployment, we reduce their number to 20, as a result we get a mountain of 502, and then we begin to autoscaling again to 60 according to the scaling rules.
This is what it looked like in Grafana:
We responded and corrected the behavior when scaling:
It allows us to properly manage the Horizontal Pod Autoscaler: if autoscaling is enabled, we use the autoscaling parameters rather than replicas.
As an alternative and a crutch, you can deploy to the maximum number of replicas (set the values in Replicas in ReplicaSet and Max Replicas in Horizontal Pod Autoscaler to be the same), and after deployment Horizontal Pod Autoscaler will set the number of replicas depending on the load.
However, the documentation says that it is wrong to do this.
Hybrid scaling
There is detailed information on Horizontal Pod Autoscaler documentationwhich describes a safe procedure for moving from a ReplicaSet to a Horizontal Pod Autoscaler.
Please note that the recipe from the documentation may not work – this happened to us. Therefore, for such procedures of your PHP-FPM application or any legacy application, when moving to Horizontal Pod Autoscaler, it is better to select the routine maintenance window.
In Horizontal Pod Autoscaler version 2, we added the ability to control scaling behavior, allowing you to smoothly or vice versa stepwise increase or decrease the number of replicas in Deployment.
Additionally, we added dynamic pools to the php-fpm configuration.
Now it is possible to grow a little inside the pod and not request Horizontal Pod Autoscaler immediately, but request it only at the moment when there is too much traffic.
Finally, we have added Cluster Autoscaler to Google on almost all pools. At the same time, we can grow horizontally in clusters.
In fact, we got hybrid scaling in the legacy application:
That is, we can vertically pull ourselves up using PHP-FPM processes, handle a sharp surge in traffic, and if this is not enough, we will enable Hozontal Pod Autoscaler and begin to grow in the number of replicas in Deployment. And if this is not enough, we can grow in clusters. In addition, we can predict how much traffic there will be with the help of analysts and scale in advance using manual methods. For example, in Horizontal Pod Autoscaler, make 240 replicas instead of 140.
Problems after the transfer
— Nginx-matryoshka.
The provider has Nginx on load balancers, our load balancers are also Nginx, Nginx in Ingress and Nginx in our PHP-FPM application. It turns out that when debugging, you need to unwind a long chain of Nginx and their configurations in order to understand what is happening.
– Kubernetes probes.
The application is very old, it was written only on the knees, and is almost no longer supported. All readiness tests coincide with liveness tests. It would seem that we are simply checking the availability of the TCP port; if it is available, then traffic can be sent. But in reality this is not always the case – the availability of a port does not mean the availability of the application to fulfill user requests.
We have startup trials that, using timeouts, simply wait for the under to rise. Yes, this is wrong, you can write healthcheck, which will respond to any request by URL with 200 – which is also not entirely correct.
To finally solve everything, you can add working healthcheck endpoints or simply warm up the application in postStart, write the results to a file, and then use the startup test to go to this file and check the “warm-up” results in it.
Bottom line
I would like those who are migrating or migrating legacy applications to Kubernetes to take advantage of our experience and be able to avoid mistakes that we could not avoid. And, like us, we were able to detect the incorrect behavior of the application before it got to production.
Adviсe:
→ Gracefully shutdown processes in containers.
→ Remember, Sidecar containers can be tricky. You need to be able to complete them correctly, otherwise you may simply lose some of the real user data or data for analytics.
→ Remember about timeouts at all levels. For example, Kubernetes can stop your container before the process actually completes.
→ Remember about the conflict between ReplicaSet and Horizontal Pod Autoscaler, that when switching from ReplicaSet to Horizontal Pod Autoscaler there is a chance that all pods will collapse to just one, and this is “not very good”.
→ Do more monitoring, alerting, logging. This will allow you to solve problems.
Many thanks to the Rubik’s Cube team from Garage Eight, who worked hard on the legacy application, and to our system engineers, who know when to stop the process so that someone else doesn’t do it at a time when you don’t expect it.
Mission accomplished
Terminating…
