We have configured 3 instances behind a load balancer powering our backend running php on them. All three instances have same configurations. Everything was working fine but suddenly one of the three instances started taking much more CPU Utilisation than the other two. On investigating on NewRelic I found out that the load is not being divided equally on all three instances. On one of the VMs php-fpm process is taking 5x the CPU% while catering to only 1/4th of the total traffic.
On running HTOP, I can clearly see that the php-fpm process is taking way more CPU usage than on the other VMs running the same application with same configurations and catering more traffic.
Earlier, everything was symmetrical. All three instances divided the traffic equally and the php-fpm process didn’t take this much CPU usage(as in the case of other 2 VMs).
So during times of high traffic the CPU Utilisation is spiking to 90-100% on that particular VM meanwhile the others are running at 10%.
I can’t seem to wrap my head around what could have gone wrong and why is that VM behaving weirdly since I did not change any configuration.