How to deal with high load on serversInder
High load causes lots of problem on your servers. It’s one of those problems which is equivalent to server down and effects your service level agreement along with your company’s reputation at stake. Given the competitive hosting industry market someone can really ill afford to let their servers face high load and server down issues.
The downside of high load is customers are getting effected with slow loading times, sometimes almost inaccessible and various errors popping up on the websites with database related errors. Moreover, you start getting complains from your end customers who are running mission critical applications running live transactions on website. Customers find email issues and multiple errors unaware of what exactly is going on thus interrupting their normal routine businesses. It’s quite frustrating when you start receiving complains about websites not running and customers questioning server uptime guarantee etc. However, the real situation when your system admins find out that high load is the reason customers are unable to access their websites.
So how do you deal with high load issue on your server?
As a server expert we have been dealing with high load issues every day and interestingly we find some of the major causes of high load are spamming, eCommerce cron jobs, high disk usage processes like sql queries and high IO wait on disk.
Here are some of the major causes of high load which we have noticed for high load on servers.
- Overloaded server
- Hard disk performance
- Process & Cron job scheduling
- Database performance.
- Viruses and malware’s
Very common reason for high load is overloaded server with too many accounts, Spamming causes lot of problems with your server, your mail server starts using high resources, IPs getting blacklisted in spamhaus databases, Server IP reputation is affected thus causing bounce back issues with major providers like google and Hotmail. At the same time because of bounce back messages your mail queue is stacked with millions of emails in the queue.
Cron jobs can also cause high load for example. Magento application based cron jobs which executes quite frequently to update stock products.
Database server like mysql can also be a reason for high load because some websites which executed multiple JOIN queries are taking too much time to execute thus causing high load.
Backup process is running and taking too much of time to complete.
Systematic approach to optimize your server to avoid high load issues.
1.) Choosing the right Server Hardware
Your server hardware plays a major role in your server’s performance. You need to identify how many accounts/domains you are going to host on your server which you are looking to order.
A good server with 8 cores, 32GB RAM and SSD drives should be able to handle 200 to 300 accounts quite comfortably. Do not overload your server with too many accounts as you will fall short of resources. Optimization only helps when you have enough resources available on your server. Make use of cloud Linux kernel which helps to limit resource per account basis.
2.) Being Selective about Hard disk Drives.
Server hard disk should be running in good condition, Check the dmesg log and check Disk Input/Output performance, Observe IO Wait parameters. There are many commands to measure disk activity. For example, the following iostat command.
iostat -p sda
The iostat command helps monitor system input/output device loading by observing the time the devices are active in relation to their average transfer rates.
Another command which is mostly used is iotop
The iotop command is top-like utility for looking at disk I/O. It gathers I/O usage information provided by the Linux kernel so that you can get an idea which processes are most demanding in terms in disk I/O.
I will recommend consider using SSD (Solid State drives ) as the performance is much better than normal SATA disk. You can learn more about SSD disks on various forums and blogs. Today most of the servers are coming with SSD drives and are little expensive then normal disk servers.
3.) Plan your Backup Scheduling.
Schedule the backup process to run during off peak hours as this will help to complete the backup faster since resources on the server is mostly free. You can identify your server usage patterns and identify the correct time for your backups.
Monitor the backup log report and see if the backup is completed in time. In cPanel servers you will get notification from LFD as shown below.
The backup process completed.
Start Time: Monday, January 13, 2020 at 8:30:01 PM UTC
End Time: Monday, January 13, 2020 at 8:43:43 PM UTC
Run Time: 13 minutes and 42 seconds
Or you can check the backup logs from “/usr/local/cpanel/logs/cpbackup/filename.log”
Cron Scheduling. Find out the active cron process which are running too frequently. We have noticed mostly ecommerce based websites like Magento, Joomla and Opencart have high frequency cron jobs which eat up all the resources. Consult with the website owner and plan the cron jobs accordingly.
4.) Don’t miss to track the mail server.
Well this issue has been discussed on various blogs, forums and many other articles on how to detect spamming on your server. There are various types of command and tools available to detect spamming. Well, we are not going to discuss about that as our topic is little different here.
Major problem here is when your server is running on high load you won’t be able to access the shell. So what do you do now ? The best approach will be add monitoring alerts in your monitoring tool as you can see in the below image.
The best strategy is proactively monitor the mailing activity. We particularly use Nagios mail plugins to detect relay alerts on server. You can choose whatever system you’ve got in place, but you cannot really let spamming happen without getting traced. The moment we receive alert we should be able to take right actions. Proactive spam monitoring is the right strategy to adopt.
5.) Give considerable thought on your Database Performance.
High database usage is also one of the reason for high load activity on the server. You can install Mytop which is an open source and free monitoring tool for MySQL and MariaDB databases. It is much similar in look and feel of the most famous Linux system monitoring tool called top.
Mytop tool provides a command-line shell interface to monitor real time MySQL/MariaDB threads, queries per second, process list and performance of databases and gives an idea for the database administrator to better optimize the server to handle heavy load. specially to find out JOIN related queries, queries which have high wait time etc.
6.) Identify security risks like Viruses and Malware’s.
Servers hosting insecure websites like WordPress, e-commerce application can be infected with viruses and malwares. Reason could be related with websites using older versions, insecure third party plugins etc. Such insecure websites can be exploited for doing outgoing DDOS attack resulting in high CPU usage and network traffic. Ensure that your websites are fully secured with latest version, patched and apply proper security measures on the server. Install firewalls, antivirus software’s and web application firewall ( WAF ) on servers.
So are you dealing with high load on your server ? Our experience and expertise in managing and monitoring high load server is tested and we do the right things to systematically optimize the server for optimal performance.