[BigETI]

Intro

I haven't seen any sort of tutorial, which basicly describes how to read and write binary files.
Before we handle with binary and binary operators, you should read this thread: https://sampforum.blast.hk/showthread.php?tid=177523


What is a "Binary file"?

This article shows you useful information about binary files: http://en.wikipedia.org/wiki/Binary_file
Briefly binary files are used for computer storage, and for processing purpose. If you need from or store data to a plain text file, you have to convert it first to binary. Binary files usually don't need this step at all, which will save us storage capacity.


Why should we use binary files systems for our SA:MP servers?

People should start making systems, where accounts can be stored and loaded much more efficient by using binaries and having much better loading/saving storage than for examples INI systems do. Even though creating tools in game to load the data and manipulate them however you like, instead connecting to your root and edit some INI file. With this knowledge people can design PAWN side "some kind of" database systems, where data can be loaded and saved very efficient.
Even though, if you are going to make some kind of a map loading system in PAWN, this is one of the best solutions, since you can convert map files with selfmade tools into binary and store the map more efficient on your server directories.
This rule applies to plugin developers aswell, but this tutorial will ONLY show how to do this in PAWN and not in C/C++. If you are interested to learn about how to read and write binary files using C/C++, you'll find very good tutorials in the internet.
And believe me, it's much more effortless to do it in C/C++ than in PAWN.
Anyway...


How to get started?

First of all we need to know how to create and open files, to continue the further steps.

fopen in PAWN gives you 4 different kind of modes we can use to create a file stream

• io_read
  • Read only mode
  • Can't write into file
  • The specified file have to exist
• io_write
  • Write only mode
  • Can't read file
  • Creates the file, if not exist, otherwise it will clear the file
• io_readwrite
  • Some kind of hybrid mode
  • During the test it seemed act odd for me
  • We'll not going to use this mode in this tutorial
• io_append
  • Write only mode
  • Can't read file
  • Creates a new file, if not exist, oherwise it will APPEND on the file, means there is no file clearing step
  • Usually log files are used in append mode
  • Not used in this tutorial

Now we need to know, how to use or create files in PAWN

Read only file stream

pawn Code:

fopen("my_test_file.txt", io_read);

Write only file stream

pawn Code:

fopen("my_test_file.txt", io_write);

And then we have to check, if our file stream has been successfully created.
Note: ALWAYS use fclose after you have opened a file stream, it's like ying and yang, good and bad or forwards and backwards.
Hint: fopen returns zero, if the file stream has failed.

pawn Code:

// Works on all modes
new File my_file = fopen("my_test_file.txt", io_read);

// If my_file doesn't return zero
if(my_file)
{
    // Our file stream is ready to be processed

    // Further process...

    //DON'T FORGET fclose after fopen and the process after fopen
    fclose(my_file);    
}
// Else case
else
{
    // Our file stream has failed
    // No need for fclose here, because our file stream was not created at all.
    print("Can't open file");
}

Now we're almost done with the basics of fopen and fclose, to create and destroy our file stream
After that step I want to show you some useful macros for writing binary data.

These macros are from my pastebin version of this tutorial: http://pastebin.com/LJYvzgiG

pawn Code:

// Usage: fwritechar(File:file_handle, value);
// Writes 1 byte data into a file
#define fwritechar(%0)  fputchar(%0, false)

// Usage: freadchar(File:file_handle);
// Reads 1 byte data from a file
#define freadchar(%0)   fgetchar(%0, 0, false)

Safer version of fwritechar; It will basicly filter the value to prevent writing a value, which is higher than a 8 Bit number (8 Bit goes from 0 to 255, means 256 numbers; 2^8 )

pawn Code:

#define fwritechar(%0,%1)  fputchar(%0,(%1)&0xFF, false)

The reasons why I use macros are, because to prevent enabling UTF-8 encoding and decoding ( http://en.wikipedia.org/wiki/UTF-8 ) and of course the implementation of fgetchar is odd ( The second argument in fgetchar is useless, so I keep it zero ).

Now the question is what fwritechar and freadchar exactly do?

• fwritechar(File:file_handle, value)
  • This macro writes basicly 1 Byte (8 Bit) into a file stream.
• freadchar(File:file_handle)
  • This macro reads basicly 1 Byte (8 Bit) from a file stream.

Example:

pawn Code:

// Defines
#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

#define TEST_FILE   "test_file.bin"

//...
public OnGameModeInit()
{
    // Attempts to create our file stream to write into the file
    new File:my_file = fopen(TEST_FILE, io_write);

    // If successful
    if(my_file) 
    {
        // Let us write the number 90
        fwritechar(my_file, 90);

        // Closes our file stream
        fclose(my_file);

        // Resets our variable to zero
        my_file = File:0;

        // Attempts to create our file stream to read the file and checks, if it was successful
        if((my_file = fopen(TEST_FILE, io_read)))
        {
            // Reads the file and returns the 90, we wrote before
            printf("Our stored number was: %d", freadchar(my_file));

            // Closes our file stream
            fclose(my_file);
        }

        // If not successful
        else printf("Failed to load \"%s\"", TEST_FILE);
    }
    // If not successful
    else printf("Failed to load \"%s\"", TEST_FILE);
    
}
//...

This example above writes the number 90 (1 Byte) into a file as binary and will be read from the file, which is seen in the console.
If you try to open this file with an ordinary text editor as plain text, in ASCII the number 90 is seen as the character "Z"
You can imagine that storing in binary can mostly take much less space than as plain text.

Number 65 as plain text

Code:

65

Takes 2 Bytes space

Number 65 as binary

Code:

A

Takes only 1 Byte space

And even though, if you build up on a system which reads from a plain text file, you even have to convert this into binary aswell!

Anyway mostly people do not work with numbers from 0 to 255. Instead they want to store whole integers (32 Bit) and even if posible read and write 16 and 24 Bit numbers.
The common way to do is to do something like this:

pawn Code:

new my_number = 1000000; // Our one million in a variable

// Writes every each 8 Bit into the file stream (4 Byte)
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
fwritechar(my_file, (my_number>>>16)&0xFF);
fwritechar(my_file, (my_number>>>24)&0xFF);
// or
for(new i = 0; i < 4; i++) fwritechar(my_file, (my_number>>>(i*8))&0xFF)

What we do is basicly we use my_number BINARY AND 0xFF, which means that this number will only keep its first 8 Bits.
Then we start to BITSHIFT RIGHT LOGICAL ( >>> ) 8 times my_number and later we use the method above to disable all bits going above 8 Bit, making this the second chunk of our number
Applies to other steps aswell, only we gain the 3rd and last chunk of our number.

Now how should we read this number later? We just do something like this:

pawn Code:

// 32 Bit (int) - Read
my_number |= freadchar(my_file);
my_number |= (freadchar(my_file)<<8);
my_number |= (freadchar(my_file)<<16);
my_number |= (freadchar(my_file)<<24);
// or
for(new i = 0; i < 4; i++) my_number |= (freadchar(my_file)<<(i*8));

// 32 Bit (int) - Write
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
fwritechar(my_file, (my_number>>>16)&0xFF);
fwritechar(my_file, (my_number>>>24)&0xFF);
// or
for(new i = 0; i < 4; i++) fwritechar(my_file, (my_number>>>(i*8))&0xFF);

Magically this thing will return our desired number!

Basicly to read and write 16 and 24 Bit numbers, we have to do something like this:

pawn Code:

// 16 Bit - Read
my_number |= freadchar(my_file);
my_number |= (freadchar(my_file)<<8);
// or
for(new i = 0; i < 2; i++) my_number |= (freadchar(my_file)<<(i*8));

// 16 Bit - Write
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
// or
for(new i = 0; i < 2; i++) freadchar(my_file, (my_number>>>(i*8))&0xFF);

pawn Code:

// 24 Bit - Read
my_number |= freadchar(my_file);
my_number |= (freadchar(my_file)<<8);
my_number |= (freadchar(my_file)<<16);
// or
for(new i = 0; i < 3; i++) my_number |= (freadchar(my_file)<<(i*8));

// 24 Bit - Write
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
fwritechar(my_file, (my_number>>>16)&0xFF);
// or
for(new i = 0; i < 3; i++) fwritechar(my_file, (my_number>>>(i*8))&0xFF);

If we try to read and write a value using a tag for example Float, we can simply do this step:

pawn Code:

// Float - Read
my_floating_number |= Float:(freadchar(my_file));
my_floating_number |= Float:(freadchar(my_file)<<8);
my_floating_number |= Float:(freadchar(my_file)<<16);
my_floating_number |= Float:(freadchar(my_file)<<24);
// or
for(new i = 0; i < 4; i++) my_floating_number |= Float:(freadchar(my_file)<<(i*8));

// Float - Write
fwritechar(my_file, (_:my_floating_number)&0xFF);
fwritechar(my_file, ((_:my_floating_number)>>>8)&0xFF);
fwritechar(my_file, ((_:my_floating_number)>>>16)&0xFF);
fwritechar(my_file, ((_:my_floating_number)>>>24)&0xFF);
// or
for(new i = 0; i < 4; i++) fwritechar(my_file, (my_floating_number>>>(i*8))&0xFF);

Summary:

pawn Code:

// Defines
#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

// Our test file name
#define TEST_FILE   "test_file.bin"

// Our array, which will be saved and loaded afterwards from a binary file.
new const my_array[5] = {1234, 1000, 1337, 2837645, -1};

// Some callback or function you want to use in, example OnGameModeInit
public OnGameModeInit()
{
    // Opens a file stream in write only mode
    new File:my_file = fopen(TEST_FILE, io_write);

    // If successful
    if(my_file)
    {
        // Some variables
        new i, j;

        // Iterates through the whole array and writes all data into a file stream
        for(i = 0; i < sizeof my_array; i++) for(j = 0; j < 4; j++) fwritechar(my_file, (my_array[i]<<(j*8))&0xFF);

        // Closes and saves the file
        fclose(my_file);

        // Sets our file handle variable to zero
        my_file = File:0;

        // Opens a file stream in read only mode and checks if successful
        if((my_file = fopen(TEST_FILE, io_read)))
        {
            // A variable to store our temporary result
            new buffer = 0;

            // Iterates through the whole array
            for(i = 0; i < sizeof my_array; i++)
            {
                // Reads the values from the file stream
                for(j = 0; j < 4; j++) buffer |= (freadchar(my_file)<<(j*8));

                // Prints the results for us
                printf("Returns %d", buffer);

                // Sets buffer back to zero
                buffer = 0;
            }

            // Closes the file
            fclose(my_file);
        }

        // If not successful
        else printf("Failed to open \"%s\"", TEST_FILE);    
    }

    // If not successful
    else printf("Failed to open or create \"%s\"", TEST_FILE);
}

Prints

Code:

Returns 1234
Returns 1000
Returns 1337
Returns 2837645
Returns -1

But files don't have static sizes. What would happen, if we try to read a file like this above and we already are at its EOF (End Of File)?
The example below shows you how to build up an easy binary file reader, which I guaranty it will read every existing file you can have on your computer!

pawn Code:

#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

// Let us make some kind of a stock, which will read every existing file you want
stock readFile_GodMode(file_name[])
{
    // Opens a file stream in read only mode
    new File:my_file = fopen(file_name, io_read);

    // If successful
    if(my_file)
    {
        // Some useful variables
        new buffer = EOF, pos = 0;

        // Iterates through a whole file
        while((buffer = freadchar(my_file)) != EOF) printf("%x\t%x", pos++, buffer);

        // Closes our file stream
        fclose(my_file);

        // Stock returns 1
        return 1;
    }

    // Only, if it was not successful at all
    printf("Failed to open \"%s\"", file_name);

    // Stock returns 0
    return 0;
}

In a case when we would like to read and write without creating 2 file streams, we can use the native ftemp.
This native allows you to create file streams without being load from a real existing file, instead it's useful for storing temporary data without using expensive process for example file check systems, database systems, runtime converter, etc.
Also most useful native to detect the current position inside a file stream and jump around data, the native fseek will do its job very good.
An example below shows, how to use ftemp and fseek to "mess around" file streams:

pawn Code:

// Defines
#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

#define TEST_FILE   "test_file.txt"

public OnGameModeInit()
{
    // Creates a temporary file stream which supports reading and writing
    new File:my_temp_file = ftemp();

    // If successful
    if(my_temp_file)
    {
        // Prints the current file stream position
        printf("We are now at position %d", fseek(my_temp_file, _, seek_current));

        // Write some random data
        for(new i = 0; i < 100; i++) fwritechar(my_temp_file, random(0x100));

        // Let us know where the last point of this file stream is, and jump to it
        printf("The last point of this file stream is at %d", fseek(my_temp_file, _, seek_end));

        // Now let us jump to the 11th Byte of this file stream
        fseek(my_temp_file, 10);

        // And Print the value at the 11th Byte, with this step we get automaticly to the 12th Byte
        printf("Value 1: %d", freadchar(my_temp_file));

        // Let us move from our current position (12th Byte) to the 2nd Byte of our file stream
        fseek(my_temp_file, -10, seek_current);

        // And print its value
        printf("Value 2: %d", freadchar(my_temp_file));

        // Let us export this temporary created file stream into our scriptfiles directory
        new File:my_file = fopen(TEST_FILE, io_write);

        if(my_file)
        {
            // Some useful variable
            new buffer = EOF;

            // Goes to the 1st Byte of our first file stream
            fseek(my_temp_file);

            // Iterate through our first file stream
            while((buffer = freadchar(my_temp_file)) != EOF)
            {
                // Writes each byte into our final test file
                fwritechar(my_file, buffer);
            }

            // Close all of our file streams
            fclose(my_temp_file);
            fclose(my_file);
        }

        // If not successful
        else
        {
            printf("Failed to open or create \"%s\"", TEST_FILE);

            // Destroys our temporary file stream
            fclose(my_temp_file);
        }
    }
    else print("Failed to create a temporary file stream.");
}

This step can work vice versa aswell!

MOTD
I hope this tutorial have helped you to give you some ideas on what we could develop for our SA:MP servers to improve its data storage.


Best Regards:

~ BigETI

[IPrototypeI]

Nice Tutorial BigETI i only had the information to save the data in 4 bytes via array

example:

PHP Code:

public OnGameModeInit()
{
    new File:ExFile= fopen("test.txt", io_read),
        array[6];
    if(!ExFile)return 1;
    fread(ExFile,array);
    printf("| %d | %d | %d | %d | %d",array[0],array[1],array[2],array[3],array[4]);
    fclose(ExFile);
    return 1;
}
public OnGameModeExit()
{
    new File:ExFile= fopen("test.txt", io_write),
        array[5];
    array[0] = 1;
    array[1] = 2;
    array[2] = 3;
    array[3] = 4;
    array[4] = 5;
    fwrite(ExFile,array);
    fclose(ExFile);
    return 1;
} 


[Vince]

Quote:

Originally Posted by BigETI Hint: fopen returns zero (NULL), if the file stream has failed.

Zero is zero (0). Not null. Pawn doesn't recognize null.
Haven't read the rest yet. I'll do that when I get home from work. Looks promising, though.

[BigETI]

The definition of null is basicly void or invalid, even in the german language people use to say "Null" to zero.
I just implemented this, because in another scripting and programming languages the definition NULL exists.
But I've removed it from this tutorial, if you're getting confused with it.

And thanks.

[thefatshizms]

This makes my brain hurt. Anyway, looks like a good tutorial even though I don't really understand it

[iggy1]

Very nice, i never even thought you could make binary files in PAWN.

An SQLite database is just a binary file, so that proves how fast you can get data from large binary files. I attempted creating my own db system with bin files and failed miserably. Only way i could get it to work is by using intermediary files for each user, which was kind of pointless as i wanted all data in one file.

In C/C++ 'NULL' is equal to zero, if you mouse over 'NULL' in visual studio you will see it is simply:

Code:

#define NULL 0

Not sure if it's the same for Linux. Obviously that's not very relevant to PAWN but thought i'd just put that out there.

Will be interesting to see if someone releases a .bin userfile system and how it works.

[BigETI]

Thank you all

[michaelcosyns]

Nice tut.

[thefatshizms]

I have a question, why do you need to read/write each 8 bit? Why not 5 etc?

I'm trying to figure this out slowly.. it's kinda hard for me lol

[BigETI]

This is the way how your Harddrive/Memory reads/writes data from/to.

[thefatshizms]

So it only reads it in 8 bits then?

[ReneG]

Quote:

Originally Posted by thefatshizms So it only reads it in 8 bits then?

Us humans count in powers of 10 whereas computers count in powers of 2. Hence 2^3 = 8. I guess it's 8 bits, because that's all you really need for characters. An unsigned 8 bit number has a max value of 2^8 - 1 (255 because we have to include 0 as a number). The extended ascii chart conveniently has 256 characters!

If I'm wrong , then it's just 2^7 - 1 so the cpu can process negative numbers as well.

@OP - This tutorial was a great read.

[xeeZ]

What's that __@private@__ for? Why not static? or just local variables?

[BigETI]

I've written a not released yet include to handle with enums, also there is a not yet released plugin made by me, which is faster with reading and writing arrays than fblockread() and fblockwrite(). Also I wrote a working account system using binary type of files.

Ini systems usually have to search, read, and format to store the value. You can loose accurate data, if you store floating values as text. As binary you can store and get a real copy of a floating value.

[BigETI]

Quote:

Originally Posted by Y_Less There are already released versions of what you just said!

I've mentioned like 3 things, which of them for example has been released so far?

[BigETI]

I have to say, that the plugin I've made and compared with the natives known from file.inc some time ago, and you saw it on IRC, hasn't been released yet here on these forums. The lastest SC is still on pastebin.

[Hitman-97-]

Can we bind the binary with MySQL?

[Nero_3D]

Isn't that what fblockwrite and fblockread do ?, just that there won't be any null characters ?

[BigETI]

Quote:

Originally Posted by Hitman-97- Can we bind the binary with MySQL?

You can convert each byte easily into 2 readable characters, so the raw data can be stored as a string into columns.

Quote:

Originally Posted by Nero_3D Isn't that what fblockwrite and fblockread do ?, just that there won't be any null characters ?

fblockread() and fblockwrite() only handle 4 bytes at once, hence they won't allow bytewise operations for file streams.

[Bombaci]

Thanks.