P2P like HTTP-CLIENT/SERVER Node for updating data

[Lucifer]

Hi there....

I'm designing a financial game and it's ofcourse full with "DATA". Financial data that is. This data changes every year (4 day's), Quarter of a year (1 day), every day (15 Min.) or real-time data like the "Market's".

So the public data that only changes 1 time a day.... Like the 'Rating' of a company, is ideal to cache with HTTP 1.1. That way the used bandwith and server usage will be very low.


But: There is also 'real-time' data from markets. Like the local-goods market with always changing prices.

Now the problem is that if i don't use caching that "ALL" the players will refresh there pages to get the new data. So i want to do the refresh for them.... And 'SHARE' the data with other clients that are in the same market. The 'Data' is only for viewing and if changed by cheating it really doesn't mater to mutch.


So i want to make a EXE with delphi. That EXE has a WEB-SERVER and a WEB-CLIENT. The WEB-CLIENT is for updating the data from 'a' server. The real server or a other P2P-Data-Server. The WEB-SERVER is for sharing the latest known data, to other WEB-CLIENTS. And for getting the data into the browser like: <FRAME SRC="http://127.0.0.1:xxxx/yyyyy.php?zzzzzzzz">. So if the player refreshes it will connect to his local p2p web-server.

So every 30+ seconds the client will 'GET' the data from the Game-Server with caching headers. So he knows when the 'LAST-MODIFIED' date is and a 'ETag' the data has. Also in the HTML HEADer wil be a special tag <AUTO_UPDATE refresh="30s">, so a java script knows when to refresh the frame. And there will be a special tag <P2P_NODES IPs="xxx.xxx.xxx.xxx:yyy"> for the servers to connect to for getting extra updates. And also to connect to when received 'NEW' data from the 'game-server'.

The EXE also can be used to share and store image's. So it will reduce bandwith for images that don't change so mutch.


Is this a good idea? or just a stupid one....
And if it works, does some one want to use it?


Greatings: Lucifer.

[Chris]

I vote for "bad idea", there are numerous reasons. There is no way for you to have experience in writing such applications comparable with experience of people who wrote the standard webservers, but assume you have, still there is no way for you to have it tested as well as the standard solutions used worldwide. But assume you have it by some miracle, then you still don't have the binary preoptimised and precompiled to match the machine/linux version, also by using Delphi you are in disadvantage at the very beginning, since this language, while quiet fast, is not designed with speed as priority, also there are much fewer highly optimised compliers compared to C/C++ available on nix platforms (I assume you don't think of using Windows as a server which would be an insanity beyond insanity form my point of view ).
But let's assume you can overcome all the points above, there is still the issue of time. To write such application properly you would need to spend so much time, that you could probably code a whole game within that time budget.

[lolninja]

You could do something like what you mentioned, but without having to write your own server. Using something like Amazons S3 service you could route a sub-domain to your S3 account, which hosts all the pre-generated stock information for client downloading, this will allow your main script to sit there and process tons of data, then farm out the results to an external server which serves them to the client upon request.

So for example you could have static.example.com which points to the S3 service, where you upload your data files, which can be organized how ever you see fit. To drop the files into the S3 service you can implement a SOAP handler, which handles your login stuff, and then uploads a text payload which is your json data or what ever.

[JGadrow]

Personally, this sounds like something that'd be best to handle with application caching. On pages that display ONLY the cachable data, feel free to place a cache header. On pages that aggregate cached and non-cached data, grab the cached data from a cache table and display.

Text is VERY small and won't really contribute much to your bandwidth usage compared to the complexity or time investiture of coding an entirely new application which will need to be downloaded by your users which will limit your audience. Really, you're better off just making sure all of your media files (images, videos, etc.) have a cache header as that is where more of your traffic is going to be used.

Just remember that for anything you cache to include a timestamp in the URL so that when it's changed to a new version, the user will be directed to it because the URL will be different than the previous one and, therefore, the browser will treat is as uncached.

Do this for ALL of your images. If they never need to change, great! What did you lose by including the datestamp in the URL? If they DO need to change, you'll be glad to not have users complaining that it's "not working" in their browser when we all know it's just a cache issue.

Hope this helps.

[Chris]

Text is VERY small and won't really contribute much to your bandwidth usage compared to the complexity or time investiture of coding an entirely new application which will need to be downloaded by your users which will limit your audience. Really, you're better off just making sure all of your media files (images, videos, etc.) have a cache header as that is where more of your traffic is going to be used.

My game uses 96% of bandwidth on pure TXT But I do agree that such investment of time to optimise bandwidth is not justified.

[Lucifer]

I vote for "bad idea", there are numerous reasons. There is no way for you to have experience in writing such applications comparable with experience of people who wrote the standard webservers, but assume you have, still there is no way for you to have it tested as well as the standard solutions used worldwide.

I don't think you realy understand it.

[Lucifer]

Do this for ALL of your images. If they never need to change, great! What did you lose by including the datestamp in the URL? If they DO need to change, you'll be glad to not have users complaining that it's "not working" in their browser when we all know it's just a cache issue.

Because if you have like 10 Mb of image's for the game and let say a 'MASSIVE' online game has 1.000.000 players. That would make 10 Tera Byte of data that get's downloaded with private caching. With public caching i don't know, it depends i think... With p2p it's just 10Mb of course.... the 2the player just download's 10Mb of data from the first player. The 3th player download's 10Mb from the 1st and 2th player and so on...

So i dont need to have 10 Terabyte hosting to run a 'MASSIVE' game. Just 10 Mb for the pictures.... And the server doesn't get bothered for pictures or other 'DATA' that other players already have.



Lucifer

[Chris]

I don't think you realy understand it.

Possibly

Because if you have like 10 Mb of image's for the game and let say a 'MASSIVE' online game has 1.000.000 players. That would make 10 Tera Byte of data that get's downloaded with private caching.

Million active players!? For a browser game!? Is there even one game that have such volume cerrently? Well, if you have a million players you don't care about mere TBs of bandwidth, this would be the last of your problems then, really You could just rake an insignificant $10000 a month to pay for some insanely good connectivity and there is no problem anymore

[JGadrow]

Because if you have like 10 Mb of image's for the game and let say a 'MASSIVE' online game has 1.000.000 players. That would make 10 Tera Byte of data that get's downloaded with private caching. With public caching i don't know, it depends i think... With p2p it's just 10Mb of course.... the 2the player just download's 10Mb of data from the first player. The 3th player download's 10Mb from the 1st and 2th player and so on...

So i dont need to have 10 Terabyte hosting to run a 'MASSIVE' game. Just 10 Mb for the pictures.... And the server doesn't get bothered for pictures or other 'DATA' that other players already have.

Your estimation is slightly off base. The server does get bothered for those requests because the server acts as an additional download resource. After all, the first player must download the data from somewhere. And you'd have to handle cases where no players are currently running their p2p clients. I suppose it would be possible to have the client send a current number of active connections when it makes a connection request which would allow your server to deny requests if they're already connected to another source. And those connection requests do take bandwidth. Not very much, but it does mean that your statement of 'just 10 Mb' is false. Additionally, this is likely to discourage users because your server - being a dedicated resource - would probably allow them to download the files much faster if they could connect to it. I don't know of any games that I'd be willing to wait around on a p2p download to complete before I was even allowed to play it.

However, as an added consideration, moving to this structure takes you squarely out of the realm of a pure Browser-Based Game (which is, to my understanding, the scope of these forums) due to the fact that your players MUST download additional executables in order to run your web-based application. The beauty of this platform is that you do not need to download additional resources to make the application run. If you're defeating that, you may as well just code a standard, distributable application as the audience reception would be larger. I would be willing to go out on a limb and state that at least half of all bbg players do so simply because they do not want to download any additional executables. They want to be able to just sit down at a machine and play a game no matter which machine they're accessing it from.

Now, that having been said, there is some value in distributing some of your static data (images and the like) via p2p programs because it would utilize less of your bandwidth. However, you would maintain a greater audience if you made it optional and just placed a standard .torrent or similar file available for download on your server. Why waste your time coding up an application that would need to gain wide-spread acceptance when there are already several available for use. In this scenario, you simply allow the user to input a configuration setting for their account (probably also based on IP address since they might access from multiple machines) specifying a directory that the downloaded files reside within.

@Chris: True, but your applications are VERY text heavy with few images (at least the last time I played one they were) so that's to be expected.

[codestryke]

A very long time ago I wrote and marketed a program called Magic Link that allowed player's to play Magic The Gathering online. The program itself was written in C++ and used UDP sockets to communicate between the players or watchers of the game in progress. I personally think this is a great exercise for any programmer to undertake, you'll learn a lot of very valuable information from the process.

However as JGadrow stated this is a web development forum so we are going to come at this from a web developer point of view Reading your post it's every 30 minutes you want to do the update, to me this would be a waste on a P2P application as you could achieve the same using an AJAX request and pass JSON formatted data. Couple this with something like mod_deflate on Apache and your bandwidth requirements would be almost minimal

[Lucifer]

I don't know of any games that I'd be willing to wait around on a p2p download to complete before I was even allowed to play it.

No, it's only for 'REAL-TIME-DATA', so if you don't want to use a p2p program. You also can play the game, just like 'normal-brokers' get 'financial-data' delayed. So if you don't have p2p you just get the stock price of the 'last quarter' (1 day old max, with caching Last-modified date this day and expires on 04:00). If you download the exe, your get the data as if you where 'IN-THE BULDING-OF-STOCK-MARKET', almost no delay. So the game doesn't change with the p2p connection, it's only a tool to 'follow' the price more.

Million active players!? For a browser game!? Is there even one game that have such volume cerrently? Well, if you have a million players you don't care about mere TBs of bandwidth, this would be the last of your problems then, really  You could just rake an insignificant $10000 a month to pay for some insanely good connectivity and there is no problem anymore

It's a free-game... No commerce.... Im not a capitalist, it's just a hobby to make it.
And writing a 'normal' web-page isn't one of my hobbies. Writing something that is difficult is more fun for me. And if 1 Million players can't play it on the the same time it's not a MASSIVE but a normal online game.

And you think like microsoft: 'It doesnt matter that we make crap, just buy new hardware....'
That's not my attitude. I know you can do everything if you throw money at it. The "US Federal Reserve" is a good example of that....   

A very long time ago I wrote and marketed a program called Magic Link that allowed player's to play Magic The Gathering online. The program itself was written in C++ and used UDP sockets to communicate between the players or watchers of the game in progress. I personally think this is a great exercise for any programmer to undertake, you'll learn a lot of very valuable information from the process.

That was my idiea also in the beginning to make it with Delphi and just use UDP and TCP Socket's.
But a friend of my has a company who uses PHP. I'm using this now ('browser-based') for creating the game. Because more people with 'less' programming experiance can help with programming than.
But i also can make the Server's and Engines with Delphi and use PHP as 'Scripting Agent'.

For now i just make it a browser-game with php, so a lot of (young) people can help with it. Without a high IT degree.

After all, the first player must download the data from somewhere. And you'd have to handle cases where no players are currently running their p2p clients. I suppose it would be possible to have the client send a current number of active connections when it makes a connection request which would allow your server to deny requests if they're already connected to another source.

If the client doesn't find a P2P network it wil go to one of the web-servers but only 2 time a minute or so.
And also get from the web-server some ip adresses if there people playing with p2p. And if no one is playing or less than 10 people the chance that prices changes drasticly is very low of course. If no one is playing prices can't change of course.


But the tip to use a normal torrent file for downloading is a good one...
And that people dont want to download i dont understand. But your right about that i think.


Thanx for reading and replaying... Have some good ideas to solve some problems now.
But making the p2p client is not the biggest preority at the moment. Still writing the game and technical documents at the moment.

Thanx, and good luck all...

[Chris]

It's a free-game... No commerce.... Im not a capitalist, it's just a hobby to make it.
And writing a 'normal' web-page isn't one of my hobbies. Writing something that is difficult is more fun for me. And if 1 Million players can't play it on the the same time it's not a MASSIVE but a normal online game.

You mean 1 million concurrent players? In one world? Poor Eve-online, they are so small with their punny 50k that they can not even qualify as a free hobby massive game
http://www.gamasutra.com/php-bin/news_index.php?story=21738
You really should lower your expectations, I mean, it is good to target high, but still 20 times more than the current world record is a bit too ambitious IMVHO

And you think like microsoft: 'It doesnt matter that we make crap, just buy new hardware....'
That's not my attitude. I know you can do everything if you throw money at it. The "US Federal Reserve" is a good example of that....   

Well, I use all the tricks: hardware, software, money, humans, everything I do not discriminate among them