[Adi007]

Hello everybody!

I write this message on behalf of a large online gaming community in Romania, which currently has 5 SA:MP servers (Roleplay Gamemode).

About 2 weeks ago we were facing a serious problem. The samp servers receive certain attacks (I think attacks) and they get some freeze time. By "freeze" I refer to the fact that there is no synchronization between the server and the players... so the players are receiving the message "the server is not responding".
The servers remain freezing for various durations of time, for a few minutes, after which they get back and all players can connect. All this happens while the samp03svr process is not using abusive resources. At the time of "freezing" the servers do not shut down, the processes remain open, they do not get a crash. The problem happens randomly at various times of the day.

What i have already tried:

Server_Log.txt
There is nothing to see there, only:
[part] X left the server...

1. The first time I thought it could be a network attack, so I took a few tools to see if the problem was there. The dedicated server does not lose pings or SSH connections.
2. I checked server_log.txt and I did not see errors there (The server runs with the crashdetect plugin).
3. I checked if there were enough resources dedicated on the attack moment. RAM ok, CPU ok, space on SSD ok.
4. I checked mysql_error.log, everything clean.
5. We put the firewall solution on https://gta-mp.ro/forum/topic/420-vu...-sa-mp-037-r2/, no results...
6. We have disabled server query from server.cfg, the problem was still there...
7. I took server_logs from all the SA:MP servers and I did cross checks to see if there is any common IP connected before the server dropout, nothing suspicious.
8. I've done a few things on the net without a result.
9. I tried a previous version of GM, the problem still existed.
10. We made a log in which we record all commands executed in the idea we can catch something before the drop. The log looks ok and I even tried to type command by command, nothing worked.
11. I sat with the SSH console in front of the drops and did not catch anything suspicious. In the case of an infinite loop I would see the CPU going up (but it was ok).
12. I talked to many people who have DDoS codes, exploits and attacks and no one has any explanation.
13. There are probably other things I've tried and that I do not remember.


Mentions:
- The server runs with crashdetect plugin. There is no error/warning when the server freeze.
- The host has sufficient resources.
- No updates have been made recently to the gamemode, so it can't be considered a programming error.
- The network is always ok.
- No recent plugins have been added and no recent updates have been made to the old ones.


So ... has anyone ever faced a problem of this type recently? Is this a SA:MP exploit? Does anyone have any ideas other than those I've already tried? Thank You!

Don't have a forum account? If you have any answer/suggestion, contact me at: voxfilm13@gmail.com

[Battlezone]

According to your description of the issue, I have faced a similar thing 3 times on my server.
The CPU usage would drop to 0% and all players would just lose connection, none would be able to rejoin until I would restart the server.
It's probably the exploit ****** was talking and warning about.

[Adi007]

Quote:

Originally Posted by Battlezone According to your description of the issue, I have faced a similar thing 3 times on my server. The CPU usage would drop to 0% and all players would just lose connection, none would be able to rejoin until I would restart the server. It's probably the exploit ****** was talking and warning about.

Could you give me a link please?

[narwn]

server version?

[Adi007]

Quote:

Originally Posted by narwn server version?

0.3.7 R2-2


Note: The server does not crash and the samp03svr task is not closed during this exploit, it juse freeze and after a while it come back.

Some people told me that the "OnPlayerSpawn" callback could be the problem if some players come on the server and call this function for about 1000-2000 times / second. I tried to simulate this and called the function for 1000 times "OnPlayerSpawn(0);" (using CallLocalFunction) where 0 is my player ID. The result was that the server has kicked me (ID 0) for ackslimit exceeded, but only I was kicked, the rest of the players remained online.


The only thing I found about exploit and ****** was this (about dialogs): https://sampforum.blast.hk/showthread.php?tid=330118
Checked, this doesn't seem to be my problem.

[Battlezone]

Quote:

Originally Posted by Adi007 0.3.7 R2-2 Note: The server does not crash and the samp03svr task is not closed during this exploit, it juse freeze and after a while it come back. Some people told me that the "OnPlayerSpawn" callback could be the problem if some players come on the server and call this function for about 1000-2000 times / second. I tried to simulate this and called the function for 1000 times "OnPlayerSpawn(0);" (using CallLocalFunction) where 0 is my player ID. The result was that the server has kicked me (ID 0) for ackslimit exceeded, but only I was kicked, the rest of the players remained online. The only thing I found about exploit and ****** was this (about dialogs): https://sampforum.blast.hk/showthread.php?tid=330118 Checked, this doesn't seem to be my problem.

It's not that, I don't have any link but I believe I've seen ****** mentioning it on SA-MP discord, it's related to Raknet. From what I understood from a discussion with Jelly, an invalid packet is sent to the server, but they have to hook the client's raknet interface to be able to send out the packet. There's no possible fixes using plugins because they process packets after they're already received by the server.

[Battlezone]

It's striking again, still no answer from sa-mp team...

[Assish]

This happened to my server last week too I hope it won't start agin

[Battlezone]

I also noticed that the VSZ used jumps from 200mb to 800+mb after the server gets attacked with the exploit, and from 180mb to 780mb for the RSS as well (comparing to the server usage in its normal state), maybe this could help..

[Battlezone]

Yeah, look at my server status in the signature, from 100 to 0, thanks for caring guys
@KALCOR@KALCOR@KALCOR@HELP@HELP@HELP

[Logic_]

I guess we can't protect our servers by locking and giving password on the forums.

Fuck it.

[Variable™]

Quote:

Originally Posted by Logic_ I guess we can't protect our servers by locking and giving password on the forums. Fuck it.

The attacker may just register on the forums and do the same... Actually it would get his attention. You won't find out who is he on the forums in that situation. There should be an official fix for this.

[Logic_]

We can at least identify who it can be possibly and trace him. As for the VPNs, block 'em temporary.

[xMoBi]

we need security update now

[Kaperstone]

Quote:

Originally Posted by [HLF]Southclaw Now all we can do is wait, if the big servers get hit hard then I'm sure there will be action, so just stay calm and be patient.

Maybe if Horizon, RCMP or BARP will experience this as well, maybe something will finally move

[Spmn]

Quote:

Originally Posted by [HLF]Southclaw At least there's a fix in this RakNet branch: https://git.io/vp4Zb Now all we can do is wait, if the big servers get hit hard then I'm sure there will be action, so just stay calm and be patient.

That bug was most likely addressed back in 0.3z in the sa-mp raknet code. (see messageholelimit and ackslimit cvars; iirc there's a post by kalcor explaining them)
However, the said exploit makes "the loop at line 746 and 818 going to be executed 200 * 4294967294 = 858993458800 times", meaning that CPU usage should raise considerably, which doesn't happen for OP.

So most likely OP's problem is related to some faulty scripts/plugins, otherwise more (big) servers would be affected.

[Battlezone]

Quote:

Originally Posted by [HLF]Southclaw Yeah that's one thing I did consider, if this is an actual exploit, surely the largest servers would be hit first.

My guess is they wouldn't make the same mistake again and attack those servers.

[Battlezone]

Quote:

Originally Posted by MustangV10 Any news here? Anyone still seeing problems or have a solution?

The exploit is still there, working and even for sale.