[[HLF]Southclaw]

Ah it's good to be back making my first new topic!


Intro
Anyway, for the past year I've been working on a gamemode script and from the start I wanted a robust, speedy and flexible method of saving. INI files are flexible but not speedy, SQL is sort of flexible but not amazingly fast and Pawn line limit is a bastard for queries.

So I decided on using the fblock functions, I've loved these since I started working with files but never really found a use for them until around 2 years ago. There's a lack of love for them on the forums! I will write a nice tutorial/guide at some point. But until then, there's a decent amount of information in this topic. (My guide will focus more on how to use the functions in real situations such as saving player data and other things)


Testing
I've been testing the encoded vs binary methods recently and I am interested in sharing these tests and getting some feedback from the more experienced coders who I see around the scripting discussion area.

So far, I have two tests. The first is a basic write and read of 4mb of data (1024 32b cells or 4096 bytes) and the second is a more practical test which writes the encoded data with delimiters then uses sscanf to extract the data afterwards (a common practice)


Code
Here is the test code and results from my windows machine:

https://gist.github.com/Southclaw/8482426

Run it as a filterscript, preferably on a completely blank gamemode for obvious reasons.

"What the hell is y_speedtest?" I hear you ask. Sorry that's my own include name because Y_Less only released the code as a snippet! You can grab that from over here. It's not any part of YSI, just a standalone benchmarking include that really should have it's own topic because it's so useful *stares at Y_Less*.


Notes
Q: Why are the fopen and fclose functions included in the benchmark?
A: Because I didn't want to let the file size grow, I am unsure whether or not that affects performance. And generally, fopen/fwrite/fclose is all one process (fread is a different story, but the tests don't deal with multi-line files at the moment)

Q: What about SQL?
A: I do plan to add an SQL test that deals with the same method of data. I will do this on the third test I have planned, which deals with a much more practical implementation (saving a player's password hash, IP, and some stats such as cash).

[Y_Less]

The thing I want to point out is the "Range" values for your write functions - they are huge. A mean of 283us with a range of 157us means that that op takes anywhere from about 205us to 362us, not vast numbers but not insignificant either. The reason is that any file operations are IO-bound, the processing done in PAWN is nothing compared to the time taken to actually write the data to disk. If you really are concerned about performance there are two options:

1) JernejL wrote a threaded IO plugin, so the slow disk accesses don't interfere with continuing to run PAWN code.

2) SQL could be an option - it isn't that slow and is again threaded so that any slowness it does have doesn't interfere with running PAWN. You start a query running then leave it to it in the background while doing other things.

Also: That snippet is in y_utils in 4.0 .

[[HLF]Southclaw]

Oh geez after skim reading it quickly when you replied it was marked as read and I forgot about it, sorry!

Anyway, the reason for all this file testing is that I'm working on a binary library designed specifically for modular gamemodes that doesn't depend on a specific order of the data in the file (a problem I ran into with binary files, the write order has to match the read order). I've been studying how relational database files are structured.

It's mostly just coding practice but I haven't seen much use of fblock around here. It's mostly an alternative to storing things as plaintext INI/JSON/etc.

Anyway, I'll likely post some progress and more info in this thread soon, with some SQL comparisons too (however SQL is a little too different to compare technically, it's mostly just practical tests and "real" situations).

[Misiur]

SQL can be really fast as well (even without threading), but you have to use indexes, and some additional settings. For example for SQLite:

pawn Код:

db_query(dbhandle, "PRAGMA synchronous = OFF");
db_query(dbhandle, "PRAGMA journal_mode = MEMORY");

It makes the whole app a little more prone to errors in case of crashes, but gets job done.

[[HLF]Southclaw]

I really need to re-write all my SQL as I know it's pretty bad and has been since the start (no indexes or anything, just basics)

But anyway, aside from that I managed to finish the first version of the binary file library: https://github.com/Southclaw/modio (modio = modular IO). It's a bit awkward to speedtest but shouldn't be too slow; aside from the single calls to fblockread/write the only heavy stuff is a few loops and array stuff.

I may have just wasted 3 nights coding and this is utterly useless, but it was interesting nonetheless with doing all the planning etc. Let me know what you guys think if you take a look at the code; it's mostly commented.