Memory leakage - what can cause this?
#1

Hello everyone,

My SA:MP server has some extreme memory leakage problems. I've been trying to fix it for months, but at some point I simply gave up because I couldn't figure out what was causing it. But then I realized that maybe someone on the forums can help me with this.

In this thread I'll try to post as much information as I have, including what I've tried to fix the problem myself.

The problem:
Since mid-January I've moved my SA:MP server from Volt-Host to SmartBytesHosting due to the many problems I've experienced with Volt-Host in the past. Before the migration of the server to the new hosting I don't recall there ever being any memory related issues with my script or server, but I can't confirm this for 100% because an uptime longer than a few days was impossible with Volt-Host.

After the hosting company migration it quickly came to my attention that my server would automatically restart every week or so. A quick discussion with the owner of the new hosting company taught me that this happened due to the high memory usage. SmartBytesHosting runs some sort of script that protects their services from getting disrupted by their customers' scripts eating up too much memory or CPU cycles.

The memory leakage causes unwanted server down time (due to the reboots) and it overloads the servers of SmartBytesHosting, so that's obviously a problem.

The first thing I could come up with was the fact that both hosting companies are using a different server operating system. Volt-Host uses CentOS (a Linux distro) and SmartBytesHosting uses Windows Server. This is actually the only obvious difference between now and before.

This is the graph that shows the RAM usage of the past 30 days:

(The dips are the server restarts)

Information about my server:

The SA:MP server in question uses the following default SA:MP filterscripts:
  • ferriswheel.amx
  • ls_elevator.amx
And these plugins:All of these plugins have been updated frequently up to about 2 months ago. The server itself is SA:MP 0.3z-R2.

The gamemode:
My gamemode is heavily based on the MySQL plugin. It uses the database to store the player accounts, achievements, sessions, businesses, houses, stores, races, settings and a few other things. This is why I’ve looked into all the MySQL related scripts first.

I’ve checked whether I’ve forgotten to clear the query results throughout the entire script, verified that all queries were correct (by checking the log files for errors) and searched the script for any obvious mistakes related to this.

There were indeed a few missing mysql_free_result() calls here and there, and one query failed to execute with some players due to the string being cut, but after fixing that I didn’t notice any difference with memory consumption.

Besides the MySQL parts I didn’t know what else to look for inside the gamemode.

Reproducing the high memory usage:
The biggest problem with attempting to diagnose and fix this issue is the fact that I was unable to reproduce the memory leak locally.

Setting up a local copy of the server, both using Linux and Windows, and letting it run unattended for about a month did nothing whatsoever. Staying connected and messing around in-game, trying to execute as many MySQL queries as I can did nothing whatsoever. Creating a script to make my computer rejoin the server constantly did nothing, trying to do the same thing with a few people did nothing…

The only thing that I couldn’t do locally that is possible on the main server was to get around ~20 players to remain connected and play ‘normally’. Because of this, I don’t know the ‘key’ to solving the problem.

Wrapping up this forum post:
I know that this was a somewhat long post, so I apologize if it’s a bit boring to read. I did my best to keep it as short as possible without leaving out any crucial information.

If anyone knows what might cause this, knows what I can try to solve this or have any other general tips that may help me to fix this issue then I’d love to know them! All help is appreciated a lot!

Yoshi
Reply
#2

Do you have a lot of global variables declared?
Reply
#3

Yes, I do have a fair amount of global variables and large (multidimensional) arrays, but I don't see that using up over 1,5GB of memory. That's the highest value I've ever had on the graph SmartBytesHosting provides me.
Reply
#4

Quote:
Originally Posted by ******
View Post
The PAWN script is not the problem, it has static memory so can't "leak". The problem is probably plugins, and I suspect the issue will be SQL queries not being correctly cleaned up after use.
Exactly.
There must be a problem with your MySQL queries. Or we can say poorly optimized script. Try to clean up useless queries or re-script them in a proper way.

EDIT: You may use BlueG's plugin.
Reply
#5

Is StrickenKid's plugin really the one you are using or is that just a link mixup? Because I don't think there have been any updates for a few years now. Also I remember some issues with that plugin on large production servers.

Try updating to BlueG's plugin (which is maintained by someone else nowadays). If the latest version proves to have a difficult learning curve due to its numerous query variations, different caches and flags you may or may not pass to the query function, use an older version such as R7.

I have another idea regarding valgrind. I'm not sure if it would work and I know that it would most likely require you to recompile the plugins that you think may be the culprit and you would also need terminal access to your server, but using valgrind, you can receive a detailed report of what memory was left unallocated on server shutdown. This has been extremely useful for me for discovering small leaks, but it may not work for you as my script is built a bit differently and written in C++.

All in all, you have an extremely interesting issue to fix and best of luck to you.
Reply
#6

Quote:
Originally Posted by ******
View Post
The PAWN script is not the problem, it has static memory so can't "leak". The problem is probably plugins, and I suspect the issue will be SQL queries not being correctly cleaned up after use.
This is what I've thought since the beginning, but I honestly have no idea what to look for inside my SQL queries and the handling of them. I free the results after all SELECT queries.

Quote:
Originally Posted by FahadKing07
View Post
Exactly.
There must be a problem with your MySQL queries. Or we can say poorly optimized script. Try to clean up useless queries or re-script them in a proper way.

EDIT: You may use BlueG's plugin.
A problem in what way? Can you give an example (besides not cleaning up after a SELECT query).

Blacklite told me that he uses StrickenKid’s plugin without any issues whatsoever, so I don't think the plugin is generally 'broken'.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)