Best way to store large amounts of data?.
#1

Hello pepel!

I'm making an object editor for a server, and i want it so every player can have their own map in their own virtual world. I have finished most of it now I'm figuring out the best way to save the maps. Ideally i want to be able to save each players map. Now if i allow them to create 1000 objects that's quite a lot of data that needs to be saved. I figure flat files would be no good for this much data, i could be very wrong.

So at the moment I'm thinking of 2 options.

1) Create a single database and each player has his/her own table with the map inside.
2) Each player has their own database with just their map inside.

I'm kind of thinking option 2 might be the best option, because option 1 will eventually become a very large database and queries might become slow. (i'm not even sure if they do get slower with larger databases)

Which option would you choose? If none how would you do it?

I would really appreciate it if someone could provide a better method than the 2 i posted.

Thanks in advance for any tips.
Reply
#2

One table structured like so:

objectid, objectmapid, objectownerid, objectx, objecty, objectz, objectrx, objectry, objectrz, objectmodel, objectvw (and other fields).

With this structure, you can pull data from the table by the map ID (so each player can have multiple maps AND objects) and it's a neat approach.

There's no need for an entire database or an entire table just for 1000 rows. MySQL was specifically designed to deal with heavy loads of data, 1000 rows per 1 player isn't really much in comparison to what MySQL really is capable of.
Reply
#3

Quote:
Originally Posted by Calgon
Посмотреть сообщение
One table structured like so:

objectid, objectmapid, objectownerid, objectx, objecty, objectz, objectrx, objectry, objectrz, objectmodel, objectvw (and other fields).

With this structure, you can pull data from the table by the map ID (so each player can have multiple maps AND objects) and it's a neat approach.

There's no need for an entire database or an entire table just for 1000 rows. MySQL was specifically designed to deal with heavy loads of data, 1000 rows per 1 player isn't really much in comparison to what MySQL really is capable of.
Thanks, it is indeed a neat approach, players can have more than one map. Very nice. I was a little worried about the size the DB will get, but i guess that doesn't matter. At the moment I'm going with your suggestion. Thanks again.

I'm still open to other suggestions I'm not touching the script again until tomorrow, so if anyone else has a suggestion let me know please.
Reply
#4

IMO saving objects to a file is more suitable for this kind of use. In my object editor, I save objects to a file in this format:
pawn Код:
//model, pos x, pos y, pos z, rot x, rot y, rot z, ...
496C|391E27F8|42E5EBE1|4439C3D1|368483F1|43A8C000|42B3FFEE
496C|397FB81A|43197BA4|443C586D|36C3EF17|43B04000|42B3FFED
4964|39B4E559|434169F9|443CAC16|3705A89D|37000000|42B3FFE8
Integers and floats are stored as hex values, which can then be easily exploded with sscanf:
pawn Код:
new
    val[7];
sscanf(str, "p<|>a<x>[7]", val);
CreateObject(val[0], Float:val[1], Float:val[2], Float:val[3], Float:val[4], Float:val[5], Float:val[6]);
And that took me less than one second to load 900 objects (1.8Ghz single core processor).
Reply
#5

Quote:
Originally Posted by wikianswerer
Having a couple of hundred thousands rows is perfectly fine, as long as :

they are indexed properly
and your queries are done properly (i.e. using the right indexes, for instance)
and
Quote:
Originally Posted by wikianswerer
I maintain a 90GB reporting database (for web server logs) which has several tables with 100s of millions of rows the biggest is 318m. I can get results from a standard select query with a join here and there (under moderate load) in 10 - 50ms.
Not sure about the comparison of BlueG's MySQL plugin to what webservers use, but it was made to handle tons of data.



Quote:
Originally Posted by YJIET
Посмотреть сообщение
IMO saving objects to a file is more suitable for this kind of use. In my object editor, I save objects to a file in this format:
pawn Код:
//model, pos x, pos y, pos z, rot x, rot y, rot z, ...
496C|391E27F8|42E5EBE1|4439C3D1|368483F1|43A8C000|42B3FFEE
496C|397FB81A|43197BA4|443C586D|36C3EF17|43B04000|42B3FFED
4964|39B4E559|434169F9|443CAC16|3705A89D|37000000|42B3FFE8
Integers and floats are stored as hex values, which can then be easily exploded with sscanf:
pawn Код:
new
    val[7];
sscanf(str, "p<|>a<x>[7]", val);
CreateObject(val[0], Float:val[1], Float:val[2], Float:val[3], Float:val[4], Float:val[5], Float:val[6]);
And that took me less than one second to load 900 objects (1.8Ghz single core processor).
Props.
Reply
#6

Quote:
Originally Posted by ******
Посмотреть сообщение
I've said this before - we can't possibly know!
I wasn't asking for solid evidence, i just wanted to get peoples opinions on how they would do it. Somebody may have already been through all of this and be able give me tips.

It's not just the 1000 *lines that would be the large amount of data its the 1000x registered players i was thinking about. For example 1000 registered players = 1,000, 000 objects to be saved. Obviously not at once, and even then they would need to have created 1000 objects, but you get my point.

I have decided to use MYSql because of how easy it is to use, and OFC all of it's features.

Thanks all for your responses.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)