13.09.2013, 16:32
Quote:
|
Originally Posted by [Hunter]
 Why is this isssue still not fixed? Very disappointing..
|
New exploit causing major disconnects!
13.09.2013, 16:33
Quote:
|
Originally Posted by linuxthefish
|
13.09.2013, 19:34
Quote:
|
Originally Posted by [Hunter]
It hasn't been addressed there.. thus the reason for my post.
|
Also to add to this issue, I no longer own a server yet assist with one and found the same problem happening there starting from Wednesday; I've just upgraded their server to R2. I will report back on this issue tommorow (Attack happen around every 4 hours) Not sure why they don't happen every 5 minutes. I would have though a determined "hacker" would run the expolit every 5 minutes but hey.
14.09.2013, 06:13
There isn't enough proof that this is caused by an exploit. It could easily be a script/plugin issue.
https://sampforum.blast.hk/showthread.php?tid=413146 <- read my second reply in this thread.
If it was an issue with the SA-MP server, I couldn't fix it until someone provided me with a backtrace. I normally get these quickly when there are legitimate issues with the server.
https://sampforum.blast.hk/showthread.php?tid=413146 <- read my second reply in this thread.
If it was an issue with the SA-MP server, I couldn't fix it until someone provided me with a backtrace. I normally get these quickly when there are legitimate issues with the server.
14.09.2013, 06:49
Quote:
|
Originally Posted by Kalcor
There isn't enough proof that this is caused by an exploit. It could easily be a script/plugin issue.
https://sampforum.blast.hk/showthread.php?tid=413146 <- read my second reply in this thread. If it was an issue with the SA-MP server, I couldn't fix it until someone provided me with a backtrace. I normally get these quickly when there are legitimate issues with the server. |
That seems highly unlikely.
14.09.2013, 08:38
Quote:
|
Originally Posted by Sgt.TheDarkness
That seems highly unlikely.
|
If there are only a handful of complaints it usually means it's a configuration issue ie. script or plugin.
I don't discount the possibility of an exploit. SA-MP has been dealing with them since 0.1. There are some really advanced server owners who know how to debug these things and send me the information I need to fix it.
So, for now, what I said still stands.
Quote:
|
Originally Posted by Kalcor
There isn't enough proof that this is caused by an exploit.
|
14.09.2013, 13:31
Quote:
|
Originally Posted by Kalcor
There isn't enough proof that this is caused by an exploit. It could easily be a script/plugin issue.
https://sampforum.blast.hk/showthread.php?tid=413146 <- read my second reply in this thread. If it was an issue with the SA-MP server, I couldn't fix it until someone provided me with a backtrace. I normally get these quickly when there are legitimate issues with the server. |
14.09.2013, 15:42
Quote:
|
Originally Posted by Kalcor
There isn't enough proof that this is caused by an exploit. It could easily be a script/plugin issue.
https://sampforum.blast.hk/showthread.php?tid=413146 <- read my second reply in this thread. If it was an issue with the SA-MP server, I couldn't fix it until someone provided me with a backtrace. I normally get these quickly when there are legitimate issues with the server. |
14.09.2013, 16:57
Quote:
|
Originally Posted by Dropexkiller_[FARV]
There Not is an error in plugins / script, excuse my English, but I speak Spanish, this has been happening since yesterday, have been restarting my 2 servers and multiple servers more, when the server reaches 100 users restart the server, if a user came from russia and it said in chat "server is crashing" and turned off the server, Banie it several times, but still manages to change his ip, to ban gave Range
|
I think you're asking about a seperate issue that was fixed yesterday: 0.3x R2 server. Make sure you're running the 0.3x R2 server.
14.09.2013, 17:14
Quote:
|
Originally Posted by Kalcor
This thread is for the problem where the server log shows "Kicking (x) because they didn't logon to the game." and no player can join the game after that. The server appears to be running but no script is being processed.
I think you're asking about a seperate issue that was fixed yesterday: 0.3x R2 server. Make sure you're running the 0.3x R2 server. |
14.09.2013, 18:21
Quote:
|
Originally Posted by Kalcor
This thread is for the problem where the server log shows "Kicking (x) because they didn't logon to the game." and no player can join the game after that. The server appears to be running but no script is being processed.
I think you're asking about a seperate issue that was fixed yesterday: 0.3x R2 server. Make sure you're running the 0.3x R2 server. |
14.09.2013, 23:28
So, a very advanced SAMP/Script bug, that magically fixes itself after most of the players on the server have left. It works like this!
- Attacker spams multiple connections from what look like "legit SA:MP player connections" to our eyes.
- The server is somehow locked up, unable to perform other functions and callbacks, yet, nothing is displayed in the logs regarding a possible "error" in the script.
- Players begin to leave/crash
- The server MAGICALLY fixes itself and is ready to run again without being restarted/GMXed...
Yeah, I don't believe this is a bug in his script/plugins, but stick your theory, I'll stick with mine, almost every server out there is running sscanf, Whirlpool, and streamer, and if you insist that this is a plugin issue, almost all of the servers running the said plugins might experience the issue, and if it were a script problem, it'd be a bit more obvious and noticeable.
I'm telling you, it's a bug with SA:MP's network layer, not the script.
- Attacker spams multiple connections from what look like "legit SA:MP player connections" to our eyes.
- The server is somehow locked up, unable to perform other functions and callbacks, yet, nothing is displayed in the logs regarding a possible "error" in the script.
- Players begin to leave/crash
- The server MAGICALLY fixes itself and is ready to run again without being restarted/GMXed...
Yeah, I don't believe this is a bug in his script/plugins, but stick your theory, I'll stick with mine, almost every server out there is running sscanf, Whirlpool, and streamer, and if you insist that this is a plugin issue, almost all of the servers running the said plugins might experience the issue, and if it were a script problem, it'd be a bit more obvious and noticeable.
I'm telling you, it's a bug with SA:MP's network layer, not the script.
14.09.2013, 23:38
Plugins I'm running: mysql (BlueG's) streamer (Incognito) irc (Incognito) ssscanf (Y_Less) CTime (RyDer)
All these plugins are fairly common amongst most samp servers.
Regarding the script, I've literally spent the past couple of months browsing every line in the script to fix this issue, I literally have every callback printing to the server_log and nothing specific is causing it.
Regarding the infinite loop, I've dealt with those before, this has the same symptoms but it's not it as the server fixes itself within a few minutes..
Also the times aren't consistent, so I'm 100% sure its being triggered by someone. Sometimes we can run throughout the day with no issues, another day this happens 10-20 times every few hours.
Just look at my playerchart:
Just to avoid further confusion, the server DOES NOT crash, the players disconnect displaying as they quit, not timeout, I can't possibly think of anything in the script that would trigger this..
Any tips you can give me to track this issue, or do you have any idea of what can trigger this?
All these plugins are fairly common amongst most samp servers.
Regarding the script, I've literally spent the past couple of months browsing every line in the script to fix this issue, I literally have every callback printing to the server_log and nothing specific is causing it.
Regarding the infinite loop, I've dealt with those before, this has the same symptoms but it's not it as the server fixes itself within a few minutes..
Also the times aren't consistent, so I'm 100% sure its being triggered by someone. Sometimes we can run throughout the day with no issues, another day this happens 10-20 times every few hours.
Just look at my playerchart:
Just to avoid further confusion, the server DOES NOT crash, the players disconnect displaying as they quit, not timeout, I can't possibly think of anything in the script that would trigger this..
Quote:
|
Originally Posted by Kalcor
There isn't enough proof that this is caused by an exploit. It could easily be a script/plugin issue.
https://sampforum.blast.hk/showthread.php?tid=413146 <- read my second reply in this thread. If it was an issue with the SA-MP server, I couldn't fix it until someone provided me with a backtrace. I normally get these quickly when there are legitimate issues with the server. |
15.09.2013, 07:14
There's nothing provided so far that helps me narrow down the cause. In this case the SA-MP server logs and player counts aren't all that useful. Perhaps the raw log with the actual [part] lines would be useful but I don't think it's a good idea to post raw logs with the IP addresses of your players.
There are loads of diagnostics which could be helpful here. Unfortunately, I'd have to write a full article to explain them all. I'll try to give a short overview.
You can monitor your server from several levels. Much of this information applies to any internet service you're running, not just an SA-MP server.
Bandwidth monitoring
Most dedicated server providers provide access to bandwidth graphs. It's also possible to monitor bandwidth of a server/device with snmp.
Does inbound bandwidth increase or decrease when the problem occurs? An unexpected increase in inbound bandwidth could indicate some form of network attack.
Packet level monitoring
You can analyse a packet log and try to determine if there are certain IP addresses sending an unusually high number of packets or an unusually high size of packets.
On linux there is a tool called 'tcpdump' which can be used create a log of packets. For creating a packet log for a server running a UDP service like SA-MP you'd run a command like:
# tcpdump -i eth0 -n udp > my_packet_log.txt
You would leave this command running for a short time and then press Ctrl+c to terminate the logging. my_packet_log.txt would then contain a log of all packets captured while the command was run.
On Windows you would use a tool such as Microsoft Network Monitor to obtain a list of packets sent during a suspected attack.
CPU/memory monitoring
Are there unusual increases or decreases in CPU/memory when the problem occurs?
On linux, CPU/memory is normally monitored with the 'ps' and 'top' commands. On Windows, this usually done inside the Task Manager.
Monitoring network stats from an SA-MP script
SA-MP's pawn scripting provides two functions for network level monitoring: GetPlayerNetworkStats and GetNetworkStats.
For the moment the stats are provided as a string. Monitoring individual stats would require parsing this string. You could however simply add a server command to log all of the players network stats to a file, then inspect this file to see if any of the players are sending an unusually high number of messages to the server; which may indicate an attack.
The GetNetworkStats string also contains the "Server Ticks:" number, which is a very useful diagnostic since it tells you the number of cycles (or frames) your server is doing per second. Drops in server ticks could indicate lag in a script or plugin, or a function that requires some optimisation. When a server has very low "Server Ticks" the players will see lots of visible network lag in-game.
Debugging infinite loops, hangs
SA-MP does generate crash information for program crashes, which often helps narrow down the cause. Unfortunately, the SA-MP server can't generate such reports for infinite loops or hangs.
Debugging infinite loops and hangs is one of the most difficult things to do when you have a program designed to run 24/7. It requires waiting for the problem to occur, using a technique called 'break and enter' where you pause the running program, enter it with a debugger, then attempt to determine the address of the code which is infinite looping or hanging the process. Since most programs run with multiple threads, sometimes you'll need to inspect what all threads are doing. However, a hung program will usually spend most of its time executing the hung section of code. So simply entering the program with a debugger, pausing and running the program several times you will be able to see the addresses of the code which is hanging the process.
If the information gleaned from the debugger doesn't make any sense to you, you can provide it to the developer of the software, which in this case is the SA-MP developers.
On linux, you can use the GNU debugger to examine a program you suspect of hanging or going in to an infinite loop. There are several tutorials on the web including this one: http://www.unknownroad.com/rtfm/gdbtut/gdbinfloop.html
For Windows, you would need to have Visual Studio installed. (You can use other debuggers like OllyDbg or IDA to "attach to process" but I won't give info on how to use those here.)
With the Visual Studio debugger, you go in to the Windows Task Manager and right click on the program you want to debug and click Debug. The Visual Studio debugger will open and the program will be in a paused state. You click the Play button to unpause the process, then when you think the process is hung again you press Pause. The debugger will display the address of the code in the thread currently executing. You can also examine what other threads are executing in the threads list. You can repeat this process several times to get an idea of where the program is spending most of its execution time. This is most likely the source of any infinite loops or hangs.
-------------------------------
Although narrowing down these types of problems might seem a bit overwhelming, it normally only takes one server owner to report the problem to the SA-MP testers/developers. If the problem is caused by the SA-MP server code, the problem will be fixed for all other server owners. So many problems are found and fixed before most server owners are even aware of them.
There are loads of diagnostics which could be helpful here. Unfortunately, I'd have to write a full article to explain them all. I'll try to give a short overview.
You can monitor your server from several levels. Much of this information applies to any internet service you're running, not just an SA-MP server.
Bandwidth monitoring
Most dedicated server providers provide access to bandwidth graphs. It's also possible to monitor bandwidth of a server/device with snmp.
Does inbound bandwidth increase or decrease when the problem occurs? An unexpected increase in inbound bandwidth could indicate some form of network attack.
Packet level monitoring
You can analyse a packet log and try to determine if there are certain IP addresses sending an unusually high number of packets or an unusually high size of packets.
On linux there is a tool called 'tcpdump' which can be used create a log of packets. For creating a packet log for a server running a UDP service like SA-MP you'd run a command like:
# tcpdump -i eth0 -n udp > my_packet_log.txt
You would leave this command running for a short time and then press Ctrl+c to terminate the logging. my_packet_log.txt would then contain a log of all packets captured while the command was run.
On Windows you would use a tool such as Microsoft Network Monitor to obtain a list of packets sent during a suspected attack.
CPU/memory monitoring
Are there unusual increases or decreases in CPU/memory when the problem occurs?
On linux, CPU/memory is normally monitored with the 'ps' and 'top' commands. On Windows, this usually done inside the Task Manager.
Monitoring network stats from an SA-MP script
SA-MP's pawn scripting provides two functions for network level monitoring: GetPlayerNetworkStats and GetNetworkStats.
For the moment the stats are provided as a string. Monitoring individual stats would require parsing this string. You could however simply add a server command to log all of the players network stats to a file, then inspect this file to see if any of the players are sending an unusually high number of messages to the server; which may indicate an attack.
The GetNetworkStats string also contains the "Server Ticks:" number, which is a very useful diagnostic since it tells you the number of cycles (or frames) your server is doing per second. Drops in server ticks could indicate lag in a script or plugin, or a function that requires some optimisation. When a server has very low "Server Ticks" the players will see lots of visible network lag in-game.
Debugging infinite loops, hangs
SA-MP does generate crash information for program crashes, which often helps narrow down the cause. Unfortunately, the SA-MP server can't generate such reports for infinite loops or hangs.
Debugging infinite loops and hangs is one of the most difficult things to do when you have a program designed to run 24/7. It requires waiting for the problem to occur, using a technique called 'break and enter' where you pause the running program, enter it with a debugger, then attempt to determine the address of the code which is infinite looping or hanging the process. Since most programs run with multiple threads, sometimes you'll need to inspect what all threads are doing. However, a hung program will usually spend most of its time executing the hung section of code. So simply entering the program with a debugger, pausing and running the program several times you will be able to see the addresses of the code which is hanging the process.
If the information gleaned from the debugger doesn't make any sense to you, you can provide it to the developer of the software, which in this case is the SA-MP developers.
On linux, you can use the GNU debugger to examine a program you suspect of hanging or going in to an infinite loop. There are several tutorials on the web including this one: http://www.unknownroad.com/rtfm/gdbtut/gdbinfloop.html
For Windows, you would need to have Visual Studio installed. (You can use other debuggers like OllyDbg or IDA to "attach to process" but I won't give info on how to use those here.)
With the Visual Studio debugger, you go in to the Windows Task Manager and right click on the program you want to debug and click Debug. The Visual Studio debugger will open and the program will be in a paused state. You click the Play button to unpause the process, then when you think the process is hung again you press Pause. The debugger will display the address of the code in the thread currently executing. You can also examine what other threads are executing in the threads list. You can repeat this process several times to get an idea of where the program is spending most of its execution time. This is most likely the source of any infinite loops or hangs.
-------------------------------
Although narrowing down these types of problems might seem a bit overwhelming, it normally only takes one server owner to report the problem to the SA-MP testers/developers. If the problem is caused by the SA-MP server code, the problem will be fixed for all other server owners. So many problems are found and fixed before most server owners are even aware of them.
15.09.2013, 22:56
Quote:
|
Originally Posted by [Hunter]
Is no other server experiencing this??
|
16.09.2013, 03:46
Quote:
|
Originally Posted by khakain
It's happening to a lot of servers. I'm admin at a popular server, and we have these problems.
|
27.09.2013, 08:58
[Hunter]: instructions.
1. install process explorer
2. select samp-server.exe process
3. click properties
4. go to threads tab
the moment it happens create a snapshot of that view, also check thread 1's stack via button at the moment it'll be even more helpful.
1. install process explorer
2. select samp-server.exe process
3. click properties
4. go to threads tab
the moment it happens create a snapshot of that view, also check thread 1's stack via button at the moment it'll be even more helpful.
« Next Oldest | Next Newest »
Users browsing this thread: 4 Guest(s)