[BigETI]

Intro

I haven't seen any sort of tutorial, which basicly describes how to read and write binary files.
Before we handle with binary and binary operators, you should read this thread: https://sampforum.blast.hk/showthread.php?tid=177523


What is a "Binary file"?

This article shows you useful information about binary files: http://en.wikipedia.org/wiki/Binary_file
Briefly binary files are used for computer storage, and for processing purpose. If you need from or store data to a plain text file, you have to convert it first to binary. Binary files usually don't need this step at all, which will save us storage capacity.


Why should we use binary files systems for our SA:MP servers?

People should start making systems, where accounts can be stored and loaded much more efficient by using binaries and having much better loading/saving storage than for examples INI systems do. Even though creating tools in game to load the data and manipulate them however you like, instead connecting to your root and edit some INI file. With this knowledge people can design PAWN side "some kind of" database systems, where data can be loaded and saved very efficient.
Even though, if you are going to make some kind of a map loading system in PAWN, this is one of the best solutions, since you can convert map files with selfmade tools into binary and store the map more efficient on your server directories.
This rule applies to plugin developers aswell, but this tutorial will ONLY show how to do this in PAWN and not in C/C++. If you are interested to learn about how to read and write binary files using C/C++, you'll find very good tutorials in the internet.
And believe me, it's much more effortless to do it in C/C++ than in PAWN.
Anyway...


How to get started?

First of all we need to know how to create and open files, to continue the further steps.

fopen in PAWN gives you 4 different kind of modes we can use to create a file stream

• io_read
  • Read only mode
  • Can't write into file
  • The specified file have to exist
• io_write
  • Write only mode
  • Can't read file
  • Creates the file, if not exist, otherwise it will clear the file
• io_readwrite
  • Some kind of hybrid mode
  • During the test it seemed act odd for me
  • We'll not going to use this mode in this tutorial
• io_append
  • Write only mode
  • Can't read file
  • Creates a new file, if not exist, oherwise it will APPEND on the file, means there is no file clearing step
  • Usually log files are used in append mode
  • Not used in this tutorial

Now we need to know, how to use or create files in PAWN

Read only file stream

pawn Code:

fopen("my_test_file.txt", io_read);

Write only file stream

pawn Code:

fopen("my_test_file.txt", io_write);

And then we have to check, if our file stream has been successfully created.
Note: ALWAYS use fclose after you have opened a file stream, it's like ying and yang, good and bad or forwards and backwards.
Hint: fopen returns zero, if the file stream has failed.

pawn Code:

// Works on all modes
new File my_file = fopen("my_test_file.txt", io_read);

// If my_file doesn't return zero
if(my_file)
{
    // Our file stream is ready to be processed

    // Further process...

    //DON'T FORGET fclose after fopen and the process after fopen
    fclose(my_file);    
}
// Else case
else
{
    // Our file stream has failed
    // No need for fclose here, because our file stream was not created at all.
    print("Can't open file");
}

Now we're almost done with the basics of fopen and fclose, to create and destroy our file stream
After that step I want to show you some useful macros for writing binary data.

These macros are from my pastebin version of this tutorial: http://pastebin.com/LJYvzgiG

pawn Code:

// Usage: fwritechar(File:file_handle, value);
// Writes 1 byte data into a file
#define fwritechar(%0)  fputchar(%0, false)

// Usage: freadchar(File:file_handle);
// Reads 1 byte data from a file
#define freadchar(%0)   fgetchar(%0, 0, false)

Safer version of fwritechar; It will basicly filter the value to prevent writing a value, which is higher than a 8 Bit number (8 Bit goes from 0 to 255, means 256 numbers; 2^8 )

pawn Code:

#define fwritechar(%0,%1)  fputchar(%0,(%1)&0xFF, false)

The reasons why I use macros are, because to prevent enabling UTF-8 encoding and decoding ( http://en.wikipedia.org/wiki/UTF-8 ) and of course the implementation of fgetchar is odd ( The second argument in fgetchar is useless, so I keep it zero ).

Now the question is what fwritechar and freadchar exactly do?

• fwritechar(File:file_handle, value)
  • This macro writes basicly 1 Byte (8 Bit) into a file stream.
• freadchar(File:file_handle)
  • This macro reads basicly 1 Byte (8 Bit) from a file stream.

Example:

pawn Code:

// Defines
#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

#define TEST_FILE   "test_file.bin"

//...
public OnGameModeInit()
{
    // Attempts to create our file stream to write into the file
    new File:my_file = fopen(TEST_FILE, io_write);

    // If successful
    if(my_file) 
    {
        // Let us write the number 90
        fwritechar(my_file, 90);

        // Closes our file stream
        fclose(my_file);

        // Resets our variable to zero
        my_file = File:0;

        // Attempts to create our file stream to read the file and checks, if it was successful
        if((my_file = fopen(TEST_FILE, io_read)))
        {
            // Reads the file and returns the 90, we wrote before
            printf("Our stored number was: %d", freadchar(my_file));

            // Closes our file stream
            fclose(my_file);
        }

        // If not successful
        else printf("Failed to load \"%s\"", TEST_FILE);
    }
    // If not successful
    else printf("Failed to load \"%s\"", TEST_FILE);
    
}
//...

This example above writes the number 90 (1 Byte) into a file as binary and will be read from the file, which is seen in the console.
If you try to open this file with an ordinary text editor as plain text, in ASCII the number 90 is seen as the character "Z"
You can imagine that storing in binary can mostly take much less space than as plain text.

Number 65 as plain text

Code:

65

Takes 2 Bytes space

Number 65 as binary

Code:

A

Takes only 1 Byte space

And even though, if you build up on a system which reads from a plain text file, you even have to convert this into binary aswell!

Anyway mostly people do not work with numbers from 0 to 255. Instead they want to store whole integers (32 Bit) and even if posible read and write 16 and 24 Bit numbers.
The common way to do is to do something like this:

pawn Code:

new my_number = 1000000; // Our one million in a variable

// Writes every each 8 Bit into the file stream (4 Byte)
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
fwritechar(my_file, (my_number>>>16)&0xFF);
fwritechar(my_file, (my_number>>>24)&0xFF);
// or
for(new i = 0; i < 4; i++) fwritechar(my_file, (my_number>>>(i*8))&0xFF)

What we do is basicly we use my_number BINARY AND 0xFF, which means that this number will only keep its first 8 Bits.
Then we start to BITSHIFT RIGHT LOGICAL ( >>> ) 8 times my_number and later we use the method above to disable all bits going above 8 Bit, making this the second chunk of our number
Applies to other steps aswell, only we gain the 3rd and last chunk of our number.

Now how should we read this number later? We just do something like this:

pawn Code:

// 32 Bit (int) - Read
my_number |= freadchar(my_file);
my_number |= (freadchar(my_file)<<8);
my_number |= (freadchar(my_file)<<16);
my_number |= (freadchar(my_file)<<24);
// or
for(new i = 0; i < 4; i++) my_number |= (freadchar(my_file)<<(i*8));

// 32 Bit (int) - Write
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
fwritechar(my_file, (my_number>>>16)&0xFF);
fwritechar(my_file, (my_number>>>24)&0xFF);
// or
for(new i = 0; i < 4; i++) fwritechar(my_file, (my_number>>>(i*8))&0xFF);

Magically this thing will return our desired number!

Basicly to read and write 16 and 24 Bit numbers, we have to do something like this:

pawn Code:

// 16 Bit - Read
my_number |= freadchar(my_file);
my_number |= (freadchar(my_file)<<8);
// or
for(new i = 0; i < 2; i++) my_number |= (freadchar(my_file)<<(i*8));

// 16 Bit - Write
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
// or
for(new i = 0; i < 2; i++) freadchar(my_file, (my_number>>>(i*8))&0xFF);

pawn Code:

// 24 Bit - Read
my_number |= freadchar(my_file);
my_number |= (freadchar(my_file)<<8);
my_number |= (freadchar(my_file)<<16);
// or
for(new i = 0; i < 3; i++) my_number |= (freadchar(my_file)<<(i*8));

// 24 Bit - Write
fwritechar(my_file, my_number&0xFF);
fwritechar(my_file, (my_number>>>8)&0xFF);
fwritechar(my_file, (my_number>>>16)&0xFF);
// or
for(new i = 0; i < 3; i++) fwritechar(my_file, (my_number>>>(i*8))&0xFF);

If we try to read and write a value using a tag for example Float, we can simply do this step:

pawn Code:

// Float - Read
my_floating_number |= Float:(freadchar(my_file));
my_floating_number |= Float:(freadchar(my_file)<<8);
my_floating_number |= Float:(freadchar(my_file)<<16);
my_floating_number |= Float:(freadchar(my_file)<<24);
// or
for(new i = 0; i < 4; i++) my_floating_number |= Float:(freadchar(my_file)<<(i*8));

// Float - Write
fwritechar(my_file, (_:my_floating_number)&0xFF);
fwritechar(my_file, ((_:my_floating_number)>>>8)&0xFF);
fwritechar(my_file, ((_:my_floating_number)>>>16)&0xFF);
fwritechar(my_file, ((_:my_floating_number)>>>24)&0xFF);
// or
for(new i = 0; i < 4; i++) fwritechar(my_file, (my_floating_number>>>(i*8))&0xFF);

Summary:

pawn Code:

// Defines
#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

// Our test file name
#define TEST_FILE   "test_file.bin"

// Our array, which will be saved and loaded afterwards from a binary file.
new const my_array[5] = {1234, 1000, 1337, 2837645, -1};

// Some callback or function you want to use in, example OnGameModeInit
public OnGameModeInit()
{
    // Opens a file stream in write only mode
    new File:my_file = fopen(TEST_FILE, io_write);

    // If successful
    if(my_file)
    {
        // Some variables
        new i, j;

        // Iterates through the whole array and writes all data into a file stream
        for(i = 0; i < sizeof my_array; i++) for(j = 0; j < 4; j++) fwritechar(my_file, (my_array[i]<<(j*8))&0xFF);

        // Closes and saves the file
        fclose(my_file);

        // Sets our file handle variable to zero
        my_file = File:0;

        // Opens a file stream in read only mode and checks if successful
        if((my_file = fopen(TEST_FILE, io_read)))
        {
            // A variable to store our temporary result
            new buffer = 0;

            // Iterates through the whole array
            for(i = 0; i < sizeof my_array; i++)
            {
                // Reads the values from the file stream
                for(j = 0; j < 4; j++) buffer |= (freadchar(my_file)<<(j*8));

                // Prints the results for us
                printf("Returns %d", buffer);

                // Sets buffer back to zero
                buffer = 0;
            }

            // Closes the file
            fclose(my_file);
        }

        // If not successful
        else printf("Failed to open \"%s\"", TEST_FILE);    
    }

    // If not successful
    else printf("Failed to open or create \"%s\"", TEST_FILE);
}

Prints

Code:

Returns 1234
Returns 1000
Returns 1337
Returns 2837645
Returns -1

But files don't have static sizes. What would happen, if we try to read a file like this above and we already are at its EOF (End Of File)?
The example below shows you how to build up an easy binary file reader, which I guaranty it will read every existing file you can have on your computer!

pawn Code:

#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

// Let us make some kind of a stock, which will read every existing file you want
stock readFile_GodMode(file_name[])
{
    // Opens a file stream in read only mode
    new File:my_file = fopen(file_name, io_read);

    // If successful
    if(my_file)
    {
        // Some useful variables
        new buffer = EOF, pos = 0;

        // Iterates through a whole file
        while((buffer = freadchar(my_file)) != EOF) printf("%x\t%x", pos++, buffer);

        // Closes our file stream
        fclose(my_file);

        // Stock returns 1
        return 1;
    }

    // Only, if it was not successful at all
    printf("Failed to open \"%s\"", file_name);

    // Stock returns 0
    return 0;
}

In a case when we would like to read and write without creating 2 file streams, we can use the native ftemp.
This native allows you to create file streams without being load from a real existing file, instead it's useful for storing temporary data without using expensive process for example file check systems, database systems, runtime converter, etc.
Also most useful native to detect the current position inside a file stream and jump around data, the native fseek will do its job very good.
An example below shows, how to use ftemp and fseek to "mess around" file streams:

pawn Code:

// Defines
#define fwritechar(%0)  fputchar(%0, false)
#define freadchar(%0)   fgetchar(%0, 0, false)

#define TEST_FILE   "test_file.txt"

public OnGameModeInit()
{
    // Creates a temporary file stream which supports reading and writing
    new File:my_temp_file = ftemp();

    // If successful
    if(my_temp_file)
    {
        // Prints the current file stream position
        printf("We are now at position %d", fseek(my_temp_file, _, seek_current));

        // Write some random data
        for(new i = 0; i < 100; i++) fwritechar(my_temp_file, random(0x100));

        // Let us know where the last point of this file stream is, and jump to it
        printf("The last point of this file stream is at %d", fseek(my_temp_file, _, seek_end));

        // Now let us jump to the 11th Byte of this file stream
        fseek(my_temp_file, 10);

        // And Print the value at the 11th Byte, with this step we get automaticly to the 12th Byte
        printf("Value 1: %d", freadchar(my_temp_file));

        // Let us move from our current position (12th Byte) to the 2nd Byte of our file stream
        fseek(my_temp_file, -10, seek_current);

        // And print its value
        printf("Value 2: %d", freadchar(my_temp_file));

        // Let us export this temporary created file stream into our scriptfiles directory
        new File:my_file = fopen(TEST_FILE, io_write);

        if(my_file)
        {
            // Some useful variable
            new buffer = EOF;

            // Goes to the 1st Byte of our first file stream
            fseek(my_temp_file);

            // Iterate through our first file stream
            while((buffer = freadchar(my_temp_file)) != EOF)
            {
                // Writes each byte into our final test file
                fwritechar(my_file, buffer);
            }

            // Close all of our file streams
            fclose(my_temp_file);
            fclose(my_file);
        }

        // If not successful
        else
        {
            printf("Failed to open or create \"%s\"", TEST_FILE);

            // Destroys our temporary file stream
            fclose(my_temp_file);
        }
    }
    else print("Failed to create a temporary file stream.");
}

This step can work vice versa aswell!

MOTD
I hope this tutorial have helped you to give you some ideas on what we could develop for our SA:MP servers to improve its data storage.


Best Regards:

~ BigETI