[kaisersouse]

So my server is 'closing' (aka crashing) on its own. This started happening suddenly out-of-the-blue when average player pop started to increase. For the record, its had this many people before (it aint many) without any problems.

I've added debug printf to all callbacks and most functions, and there is no 'pattern' that seems to be causing this. No common last-command that stops everything, no common connecting IPs, and I've checked every single foreach loop I have to make sure I wasn't being dumb.

I've disabled any scripts that would allow object spawning by ID...can only choose from menu now. Car mods aren't the cause as they've been as they are for years without any issues. I also have bad car mod logic in place, as well as stuff to protect against bullet vector exploits, text formatting exploits, and I DID find a random CreateTextDraw that used an empty string value (" ") and I've changed that to "~w~" per a thread I dug up here (I dont think it matters really, but covering all bases).

I'm trying to figure out the cause of this before I start (finally) converting to 0.3.7. System is Windows 2008 Standard R2 fully patched with VC++ redists 2008, 2010, 2013 both x86 and x64 versions installed. Also have all the .Net updates except 4.6.

Pasting the crashinfo.txt file that is created when this thing tanks. As I mentioned...I have crashdetect running and HAVE confirmed it works as of a couple of months ago. I don't think the script is folding in on itself, otherwise I would expect to see some crashdetect stuff somewhere.

Any sane/logical troubleshooting tips would be massively appreciated, as I'm about to shave my head with broken glass and pour gasoline on it at this point. Thanks!


Midline Edit: I've been watching resources via processexplorer, and there are no spikes in CPU use, memory use, or disk I/O when this happens. Its humming along fine one minute, and just gone the next.


Heres the Windows error:

Quote:

-------------------------- SA-MP Server: 0.3z-R4 Exception At Address: 0x00488746 Module: (samp-server.exe) Registers: EAX: 0x0018F984 EBX: 0x03C32994 ECX: 0x0018F995 EDX: 0x00470025 ESI: 0x00000000 EDI: 0x00470025 EBP: 0x03CD4304 ESP: 0x0018F978 EFLAGS: 0x00010283 Stack: +0000: 0x02A3AB78 0x03F3EB98 0x00000000 0x00000000 +0010: 0x00000800 0x00000000 0x0018F995 0x1C5AD101 +0020: 0x00302450 0x00000000 0x0000FFFF 0x74270000 +0030: 0x000001E3 0x0018FA74 0x00000005 0x00000000 +0040: 0x002F5018 0x0018F9C8 0xFFBADD11 0xC0000139 +0050: 0x6E4F0000 0x79616C50 0x78457265 0x64657469 +0060: 0x756E654D 0x004AE600 0x0018FA30 0x742A5650 +0070: 0x01DC7EE0 0x004AE65C 0x0018FA30 0x02D6D450 +0080: 0x00000007 0x0000FFFF 0x00000000 0x00302450 +0090: 0x0018FA3C 0x772B63CF 0x772B63D4 0x1C5AD10C +00A0: 0x00302450 0x00000000 0x0000FFFF 0x0018FA14 +00B0: 0x772B01F2 0x0018FCB0 0x772F3145 0x6B68E5F8 +00C0: 0xFFFFFFFE 0x0018FA4C 0x76B0779A 0x0000007F +00D0: 0x00000000 0x0018FA68 0x76AE1E63 0xC0000139 +00E0: 0x03BB5110 0x02A3AB78 0x00130012 0x00302450 +00F0: 0x0018FB14 0x7427440E 0x74270000 0x00302450 +0100: 0x02A3AB78 0x0000005B 0x00000001 0x00000004 +0110: 0x00000025 0x00000000 0x00000000 0x00000000 +0120: 0x0018FCB0 0x004A683B 0xFFFFFFFF 0x004743D6 +0130: 0x00470025 0x02A3AB78 0x004010B6 0x02A3AB78 -------------------------- Loaded Modules: samp-server.exe A: 0x00400000 - 0x004F6000 (D:\Servers\apocalypsecow\GameServers\Everystuff\s amp-server.exe) ntdll.dll A: 0x77280000 - 0x77400000 (C:\Windows\SysWOW64\ntdll.dll) kernel32.dll A: 0x74F30000 - 0x75040000 (C:\Windows\syswow64\kernel32.dll) KERNELBASE.dll A: 0x76AD0000 - 0x76B17000 (C:\Windows\syswow64\KERNELBASE.dll) SHELL32.dll A: 0x75300000 - 0x75F4B000 (C:\Windows\syswow64\SHELL32.dll) msvcrt.dll A: 0x75250000 - 0x752FC000 (C:\Windows\syswow64\msvcrt.dll) SHLWAPI.dll A: 0x74CA0000 - 0x74CF7000 (C:\Windows\syswow64\SHLWAPI.dll) GDI32.dll A: 0x76B50000 - 0x76BE0000 (C:\Windows\syswow64\GDI32.dll) USER32.dll A: 0x76580000 - 0x76680000 (C:\Windows\syswow64\USER32.dll) ADVAPI32.dll A: 0x74E80000 - 0x74F21000 (C:\Windows\syswow64\ADVAPI32.dll) sechost.dll A: 0x76290000 - 0x762A9000 (C:\Windows\SysWOW64\sechost.dll) RPCRT4.dll A: 0x76900000 - 0x769F0000 (C:\Windows\syswow64\RPCRT4.dll) SspiCli.dll A: 0x74BE0000 - 0x74C40000 (C:\Windows\syswow64\SspiCli.dll) CRYPTBASE.dll A: 0x74BD0000 - 0x74BDC000 (C:\Windows\syswow64\CRYPTBASE.dll) LPK.dll A: 0x76100000 - 0x7610A000 (C:\Windows\syswow64\LPK.dll) USP10.dll A: 0x74D00000 - 0x74D9D000 (C:\Windows\syswow64\USP10.dll) WSOCK32.dll A: 0x733E0000 - 0x733E7000 (C:\Windows\system32\WSOCK32.dll) WS2_32.dll A: 0x75F70000 - 0x75FA5000 (C:\Windows\syswow64\WS2_32.dll) NSI.dll A: 0x74C90000 - 0x74C96000 (C:\Windows\syswow64\NSI.dll) WINMM.dll A: 0x737A0000 - 0x737D2000 (C:\Windows\system32\WINMM.dll) IMM32.DLL A: 0x764F0000 - 0x76550000 (C:\Windows\system32\IMM32.DLL) MSCTF.dll A: 0x76DB0000 - 0x76E7C000 (C:\Windows\syswow64\MSCTF.dll) sscanf.DLL A: 0x10000000 - 0x1000D000 (D:\Servers\apocalypsecow\GameServers\Everystuff\p lugins\sscanf.DLL) MSVCR100.dll A: 0x742C0000 - 0x7437E000 (C:\Windows\system32\MSVCR100.dll) streamer.DLL A: 0x74270000 - 0x742B4000 (D:\Servers\apocalypsecow\GameServers\Everystuff\p lugins\streamer.DLL) MSVCP120.dll A: 0x741F0000 - 0x74261000 (C:\Windows\system32\MSVCP120.dll) MSVCR120.dll A: 0x74100000 - 0x741EE000 (C:\Windows\system32\MSVCR120.dll) irc.DLL A: 0x740B0000 - 0x740F8000 (D:\Servers\apocalypsecow\GameServers\Everystuff\p lugins\irc.DLL) mswsock.dll A: 0x74500000 - 0x7453C000 (C:\Windows\system32\mswsock.dll) wshtcpip.dll A: 0x744E0000 - 0x744E5000 (C:\Windows\System32\wshtcpip.dll) NLAapi.dll A: 0x6FA80000 - 0x6FA90000 (C:\Windows\system32\NLAapi.dll) napinsp.dll A: 0x6FA70000 - 0x6FA80000 (C:\Windows\system32\napinsp.dll) DNSAPI.dll A: 0x73FF0000 - 0x74034000 (C:\Windows\system32\DNSAPI.dll) winrnr.dll A: 0x6FA60000 - 0x6FA68000 (C:\Windows\System32\winrnr.dll) IPHLPAPI.DLL A: 0x74050000 - 0x7406C000 (C:\Windows\system32\IPHLPAPI.DLL) WINNSI.DLL A: 0x74040000 - 0x74047000 (C:\Windows\system32\WINNSI.DLL) rasadhlp.dll A: 0x6FA10000 - 0x6FA16000 (C:\Windows\system32\rasadhlp.dll) PROPSYS.dll A: 0x6FB40000 - 0x6FC35000 (C:\Windows\system32\PROPSYS.dll) OLEAUT32.dll A: 0x76870000 - 0x768FF000 (C:\Windows\syswow64\OLEAUT32.dll) comctl32.dll A: 0x6C980000 - 0x6CB1E000 (C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.18837_none_41e8 55142bd5705d\comctl32.dll) apphelp.dll A: 0x683E0000 - 0x6842C000 (C:\Windows\system32\apphelp.dll) CLBCatQ.DLL A: 0x76680000 - 0x76703000 (C:\Windows\syswow64\CLBCatQ.DLL) api-ms-win-downlevel-advapi32-l1-1-0.dll A: 0x76550000 - 0x76555000 (C:\Windows\syswow64\api-ms-win-downlevel-advapi32-l1-1-0.dll) api-ms-win-downlevel-shlwapi-l1-1-0.dll A: 0x76120000 - 0x76124000 (C:\Windows\syswow64\api-ms-win-downlevel-shlwapi-l1-1-0.dll) SETUPAPI.dll A: 0x76C10000 - 0x76DAD000 (C:\Windows\syswow64\SETUPAPI.dll) CFGMGR32.dll A: 0x76B20000 - 0x76B47000 (C:\Windows\syswow64\CFGMGR32.dll) DEVOBJ.dll A: 0x74E30000 - 0x74E42000 (C:\Windows\syswow64\DEVOBJ.dll) ntmarta.dll A: 0x73210000 - 0x73231000 (C:\Windows\system32\ntmarta.dll) WLDAP32.dll A: 0x74C40000 - 0x74C85000 (C:\Windows\syswow64\WLDAP32.dll) profapi.dll A: 0x76110000 - 0x7611B000 (C:\Windows\syswow64\profapi.dll) Secur32.dll A: 0x72890000 - 0x72898000 (C:\Windows\system32\Secur32.dll) api-ms-win-downlevel-advapi32-l2-1-0.dll A: 0x6CD40000 - 0x6CD44000 (C:\Windows\system32\api-ms-win-downlevel-advapi32-l2-1-0.dll)

[Kyle]

Not meaning to be rude but have you contacted your inside source?

Surely you don't need to ask here when you've a direct contact to Kalcor?

If not I would advice querying Jernejl_Work on the IRC tomorrow at around 12:00 GMT.

[kaisersouse]

Quote:

Originally Posted by KyleSmith Not meaning to be rude but have you contacted your inside source? Surely you don't need to ask here when you've a direct contact to Kalcor? If not I would advice querying Jernejl_Work on the IRC tomorrow at around 12:00 GMT.

No. I haven't had time to participate in anything official SA-MP in about two years. Last I checked, yes I have/had access to the beta channel, yes I can probably contact current and past devs etc etc. Since I haven't been actively participating, I'm certainly not going to just pop in for my own self-centered needs.

The second half of that is called "the point of using the forum". By posting my issues here, it may hopefully help someone else who runs into similar troubles. I like to think that I've never used my 'position' for technical help, as evidenced by having a charmingly buggy server for the past decade.

While you stated you didn't mean to be rude, your reply came off as a mildly passive-aggressive attempt to put some sort of my shit on blast It was odd, and unhelpful.

[Virtual1ty]

What crashdetect version are you running? Also, are the other plugins (namely sscanf) the latest version?

Nothing else pops to mind, except for one thing which I read somewhere on this forum and that are timers.
Are you calling many timers, or starting many per-player (SetTimerEx) timers? I read somewhere that timer IDs are never reused and when you hit the 32-bit integer mark the server could(?) crash.

[kaisersouse]

Quote:

Originally Posted by Virtual1ty What crashdetect version are you running? Also, are the other plugins (namely sscanf) the latest version? Nothing else pops to mind, except for one thing which I read somewhere on this forum and that are timers. Are you calling many timers, or starting many per-player (SetTimerEx) timers? I read somewhere that timer IDs are never reused and when you hit the 32-bit integer mark the server could(?) crash.

sscanf is relatively new, unsure if the newest. Ill make sure and fix it definitely. I have a central core of four global timers (onesecond,fiveseconds,oneminute,tenminutes) that are all offset by a few (prime number) milliseconds. Theyve been there for years, with minimal maintenance. I hate Ex timers with a passion, but have a few to handle intentional delays whilst kick/banning someone. I make sure to clean them up on OnPlayerDisconnect. I also avoid using all things TickCount, and only do so twice in portions of the script that are currently #defined out (and have been for days).

Thank you for your advice, I'll go check my plugin(s) and have another look at timers and their handling

EDIT: crashdetect downloaded within the last month or so to troubleshoot an exploding chicken issue.

[Evocator]

I'm not sure if you knew, but crashdetect can pretty much trace all functions/callbacks and print them for you. This 'might' help you catch a plugin that may be causing this issue rather than debugging SAMP functions yourself only. However, it consumes hell of resources when enabling.
https://github.com/Zeex/samp-plugin-...#configuration

[Kyle]

Quote:

Originally Posted by kaisersouse No. I haven't had time to participate in anything official SA-MP in about two years. Last I checked, yes I have/had access to the beta channel, yes I can probably contact current and past devs etc etc. Since I haven't been actively participating, I'm certainly not going to just pop in for my own self-centered needs. The second half of that is called "the point of using the forum". By posting my issues here, it may hopefully help someone else who runs into similar troubles. I like to think that I've never used my 'position' for technical help, as evidenced by having a charmingly buggy server for the past decade. While you stated you didn't mean to be rude, your reply came off as a mildly passive-aggressive attempt to put some sort of my shit on blast It was odd, and unhelpful.

Sorry that's what happens when you're in bed late replying.

[kaisersouse]

Quote:

Originally Posted by Ralfie I'm not sure if you knew, but crashdetect can pretty much trace all functions/callbacks and print them for you. This 'might' help you catch a plugin that may be causing this issue rather than debugging SAMP functions yourself only. However, it consumes hell of resources when enabling. https://github.com/Zeex/samp-plugin-...#configuration

As stated in initial post, crashdetect wasn't detecting anything at the time. Since then I moved the crashdetect plugin to the first in the list (in server.cfg) and apparently it started working again:

Crashdetect is mentioning a fault with HideMenuForPlayer when the server crashes now. I only have old-school menus in my equally old-school race system...but only RCON admin uses those menus (to build races) and I'm the only RCON admin (and haven't used the race menus in well over 4 years). I checked all of my FS and they aren't using HideMenuForPlayer anywhere in them.

After some research*, I loaded 'fixes2' and the crashing has stopped. That said, now the timers in my ancient race system (Yagu's) are all screwed up and the countdowns are all broken haha. Better than a server outage...I'm off to figure out how to correct the race timers.



*Research = looking up HideMenuForPlayer in the wiki:
https://sampwiki.blast.hk/wiki/HideMenuForPlayer

...and seeing this note:
"Crashes the both server and player if an invalid menu ID Given."

...and remembering/re-seeing on the forums that fixes2 helps with that.

[Kyle]

Quote:

Originally Posted by kaisersouse As stated in initial post, crashdetect wasn't detecting anything at the time. Since then I moved the crashdetect plugin to the first in the list (in server.cfg) and apparently it started working again: Crashdetect is mentioning a fault with HideMenuForPlayer when the server crashes now. I only have old-school menus in my equally old-school race system...but only RCON admin uses those menus (to build races) and I'm the only RCON admin (and haven't used the race menus in well over 4 years). I checked all of my FS and they aren't using HideMenuForPlayer anywhere in them. After some research*, I loaded 'fixes2' and the crashing has stopped. That said, now the timers in my ancient race system (Yagu's) are all screwed up and the countdowns are all broken haha. Better than a server outage...I'm off to figure out how to correct the race timers. *Research = looking up HideMenuForPlayer in the wiki: https://sampwiki.blast.hk/wiki/HideMenuForPlayer ...and seeing this note: "Crashes the both server and player if an invalid menu ID Given." ...and remembering/re-seeing on the forums that fixes2 helps with that.

I wonder why it's only suddenly started crashing though?

[kaisersouse]

Quote:

Originally Posted by KyleSmith I wonder why it's only suddenly started crashing though?

The wheat-side of me thinks it was an 'error in hiding' that only came about once more people were playing at once.

The frosted-side of me suspects that the HideMenuForPlayer crashing isn't a coincidence (since players dont get to use any of the old-fashioned menus), and is deliberate on someone elses part.


Either way, still no crashes since running fixes2 (which I had removed long ago and never put back)