15.09.2013, 07:14
There's nothing provided so far that helps me narrow down the cause. In this case the SA-MP server logs and player counts aren't all that useful. Perhaps the raw log with the actual [part] lines would be useful but I don't think it's a good idea to post raw logs with the IP addresses of your players.
There are loads of diagnostics which could be helpful here. Unfortunately, I'd have to write a full article to explain them all. I'll try to give a short overview.
You can monitor your server from several levels. Much of this information applies to any internet service you're running, not just an SA-MP server.
Bandwidth monitoring
Most dedicated server providers provide access to bandwidth graphs. It's also possible to monitor bandwidth of a server/device with snmp.
Does inbound bandwidth increase or decrease when the problem occurs? An unexpected increase in inbound bandwidth could indicate some form of network attack.
Packet level monitoring
You can analyse a packet log and try to determine if there are certain IP addresses sending an unusually high number of packets or an unusually high size of packets.
On linux there is a tool called 'tcpdump' which can be used create a log of packets. For creating a packet log for a server running a UDP service like SA-MP you'd run a command like:
# tcpdump -i eth0 -n udp > my_packet_log.txt
You would leave this command running for a short time and then press Ctrl+c to terminate the logging. my_packet_log.txt would then contain a log of all packets captured while the command was run.
On Windows you would use a tool such as Microsoft Network Monitor to obtain a list of packets sent during a suspected attack.
CPU/memory monitoring
Are there unusual increases or decreases in CPU/memory when the problem occurs?
On linux, CPU/memory is normally monitored with the 'ps' and 'top' commands. On Windows, this usually done inside the Task Manager.
Monitoring network stats from an SA-MP script
SA-MP's pawn scripting provides two functions for network level monitoring: GetPlayerNetworkStats and GetNetworkStats.
For the moment the stats are provided as a string. Monitoring individual stats would require parsing this string. You could however simply add a server command to log all of the players network stats to a file, then inspect this file to see if any of the players are sending an unusually high number of messages to the server; which may indicate an attack.
The GetNetworkStats string also contains the "Server Ticks:" number, which is a very useful diagnostic since it tells you the number of cycles (or frames) your server is doing per second. Drops in server ticks could indicate lag in a script or plugin, or a function that requires some optimisation. When a server has very low "Server Ticks" the players will see lots of visible network lag in-game.
Debugging infinite loops, hangs
SA-MP does generate crash information for program crashes, which often helps narrow down the cause. Unfortunately, the SA-MP server can't generate such reports for infinite loops or hangs.
Debugging infinite loops and hangs is one of the most difficult things to do when you have a program designed to run 24/7. It requires waiting for the problem to occur, using a technique called 'break and enter' where you pause the running program, enter it with a debugger, then attempt to determine the address of the code which is infinite looping or hanging the process. Since most programs run with multiple threads, sometimes you'll need to inspect what all threads are doing. However, a hung program will usually spend most of its time executing the hung section of code. So simply entering the program with a debugger, pausing and running the program several times you will be able to see the addresses of the code which is hanging the process.
If the information gleaned from the debugger doesn't make any sense to you, you can provide it to the developer of the software, which in this case is the SA-MP developers.
On linux, you can use the GNU debugger to examine a program you suspect of hanging or going in to an infinite loop. There are several tutorials on the web including this one: http://www.unknownroad.com/rtfm/gdbtut/gdbinfloop.html
For Windows, you would need to have Visual Studio installed. (You can use other debuggers like OllyDbg or IDA to "attach to process" but I won't give info on how to use those here.)
With the Visual Studio debugger, you go in to the Windows Task Manager and right click on the program you want to debug and click Debug. The Visual Studio debugger will open and the program will be in a paused state. You click the Play button to unpause the process, then when you think the process is hung again you press Pause. The debugger will display the address of the code in the thread currently executing. You can also examine what other threads are executing in the threads list. You can repeat this process several times to get an idea of where the program is spending most of its execution time. This is most likely the source of any infinite loops or hangs.
-------------------------------
Although narrowing down these types of problems might seem a bit overwhelming, it normally only takes one server owner to report the problem to the SA-MP testers/developers. If the problem is caused by the SA-MP server code, the problem will be fixed for all other server owners. So many problems are found and fixed before most server owners are even aware of them.
There are loads of diagnostics which could be helpful here. Unfortunately, I'd have to write a full article to explain them all. I'll try to give a short overview.
You can monitor your server from several levels. Much of this information applies to any internet service you're running, not just an SA-MP server.
Bandwidth monitoring
Most dedicated server providers provide access to bandwidth graphs. It's also possible to monitor bandwidth of a server/device with snmp.
Does inbound bandwidth increase or decrease when the problem occurs? An unexpected increase in inbound bandwidth could indicate some form of network attack.
Packet level monitoring
You can analyse a packet log and try to determine if there are certain IP addresses sending an unusually high number of packets or an unusually high size of packets.
On linux there is a tool called 'tcpdump' which can be used create a log of packets. For creating a packet log for a server running a UDP service like SA-MP you'd run a command like:
# tcpdump -i eth0 -n udp > my_packet_log.txt
You would leave this command running for a short time and then press Ctrl+c to terminate the logging. my_packet_log.txt would then contain a log of all packets captured while the command was run.
On Windows you would use a tool such as Microsoft Network Monitor to obtain a list of packets sent during a suspected attack.
CPU/memory monitoring
Are there unusual increases or decreases in CPU/memory when the problem occurs?
On linux, CPU/memory is normally monitored with the 'ps' and 'top' commands. On Windows, this usually done inside the Task Manager.
Monitoring network stats from an SA-MP script
SA-MP's pawn scripting provides two functions for network level monitoring: GetPlayerNetworkStats and GetNetworkStats.
For the moment the stats are provided as a string. Monitoring individual stats would require parsing this string. You could however simply add a server command to log all of the players network stats to a file, then inspect this file to see if any of the players are sending an unusually high number of messages to the server; which may indicate an attack.
The GetNetworkStats string also contains the "Server Ticks:" number, which is a very useful diagnostic since it tells you the number of cycles (or frames) your server is doing per second. Drops in server ticks could indicate lag in a script or plugin, or a function that requires some optimisation. When a server has very low "Server Ticks" the players will see lots of visible network lag in-game.
Debugging infinite loops, hangs
SA-MP does generate crash information for program crashes, which often helps narrow down the cause. Unfortunately, the SA-MP server can't generate such reports for infinite loops or hangs.
Debugging infinite loops and hangs is one of the most difficult things to do when you have a program designed to run 24/7. It requires waiting for the problem to occur, using a technique called 'break and enter' where you pause the running program, enter it with a debugger, then attempt to determine the address of the code which is infinite looping or hanging the process. Since most programs run with multiple threads, sometimes you'll need to inspect what all threads are doing. However, a hung program will usually spend most of its time executing the hung section of code. So simply entering the program with a debugger, pausing and running the program several times you will be able to see the addresses of the code which is hanging the process.
If the information gleaned from the debugger doesn't make any sense to you, you can provide it to the developer of the software, which in this case is the SA-MP developers.
On linux, you can use the GNU debugger to examine a program you suspect of hanging or going in to an infinite loop. There are several tutorials on the web including this one: http://www.unknownroad.com/rtfm/gdbtut/gdbinfloop.html
For Windows, you would need to have Visual Studio installed. (You can use other debuggers like OllyDbg or IDA to "attach to process" but I won't give info on how to use those here.)
With the Visual Studio debugger, you go in to the Windows Task Manager and right click on the program you want to debug and click Debug. The Visual Studio debugger will open and the program will be in a paused state. You click the Play button to unpause the process, then when you think the process is hung again you press Pause. The debugger will display the address of the code in the thread currently executing. You can also examine what other threads are executing in the threads list. You can repeat this process several times to get an idea of where the program is spending most of its execution time. This is most likely the source of any infinite loops or hangs.
-------------------------------
Although narrowing down these types of problems might seem a bit overwhelming, it normally only takes one server owner to report the problem to the SA-MP testers/developers. If the problem is caused by the SA-MP server code, the problem will be fixed for all other server owners. So many problems are found and fixed before most server owners are even aware of them.