Exploits

[Helderic]

How would you find and deal with exploits?

Lets say player A finds a exploit to duplicate a item and decides to exploit it. He continues to exploit it for a few weeks then player B finds the bug and reports it.
How would you catch player A? And what are some systems to watch/catch exploits/bugs/cheaters.

[Harkins]

Logging, logging, logging.

When a player advances a level, log it. When a player buys something substantial, log it. When a player transfers items or money to another player, log it. When a player makes any significant achievement, log it. Then do things like count how many items the average player gets in a day or the average time it takes to go from level 9 to 10. Look at whoever's doing things way faster or stronger.

On top of that, do a daily snapshot of a player's stats (money, level, items, etc.) in case your other logs fail to notice where the resources are coming from. Then you do the same sort of analysis.

If you don't have any logs and you want to find who abused an exploit you've only just realized, you're going to have a hard time. If duping an item requires visiting a particular page, or visiting two pages in quick succession, your web server logs will have *something* you can look at, you can compare player IP to any suspicious entries in your access.log. If only a particular item can be duped, look in the db for who has lots of them -- actually, you can do this for *any* item.

What you always have in your favor is that cheaters quickly get greedy when they realize what they've got and think they're getting away with it. They won't just go up a level now and then, they'll go up ten today. They won't produce a handful of extra gold for the occasional purchase, they'll produce a million. Look for the outliers.

[Helderic]

Thanks but how would you log this much? 

[travo]

I would probably use a csv for something like that... That way I can sort through all the records by type, date, etc. and easily backup data, without making the db massive.

[Harkins]

I'll probably use Hadoop to do it, but I'm not yet certain. Ask me again in a month when I've built it myself, heh.

But you don't have to get the fullest level of detail to start noticing trends.

[Chris]

You ask "how do I prepare for unexpected", the answer is simple, "you can't"
Exploits are that thing, it is something you never expected. The most reasonable thing is to focus on not letting the expoit be possible in the first place instead of features to track it.

As for dealing with exploits there are 2 solutions (after you fixed the exploitable code).
1) Modify/delete accounts involved
2) Restore the game from backups (if you are at stage where player B did it, it is probably the only option)

I also find that a decent staff is very useful in situations like that.
This is what happen once in my game:
I logged to the game and saw a big announcement on the front page. Then I went to my private message box and started reading. First admin messaged me what happended (one player got a huge amount of gold) and that he posted an announcement that it is under investigation and that he contacted leaders of major clans easing their worries, second admin messaged that he logged in, saw the player and put him immediately under protection to not let the gold spread out, the next message was from a moderator who interrogated the player and provided me the outcome, the fourth message was from a tester who recreated the exploit on a test server (providing me information how it can be recreated and under what circumstances). So, I started my dev tools, fixed the exploit, uploaded it, then made some quick tools to modify player's account (the exploit was discovered by chance, not by ill will and the player cooperated as the interrogation stated so we wanted to avoid deleting that player). Next I messaged the admin that it is secured and that new tools are available (without giving him any further instructions since it was unnecessary), next asked tester to confirm if the bug is no more and... went to bed Next day I read in my inbox a confirmation from tester that the bug is fixed and went to the forum to make some additional PR (which wasn't really needed since one of the admin already did it). That time I was really pround of my staff

[Helderic]

If you don't have any logs and you want to find who abused an exploit you've only just realized, you're going to have a hard time. If duping an item requires visiting a particular page, or visiting two pages in quick succession, your web server logs will have *something* you can look at, you can compare player IP to any suspicious entries in your access.log. If only a particular item can be duped, look in the db for who has lots of them -- actually, you can do this for *any* item.

What you always have in your favor is that cheaters quickly get greedy when they realize what they've got and think they're getting away with it. They won't just go up a level now and then, they'll go up ten today. They won't produce a handful of extra gold for the occasional purchase, they'll produce a million. Look for the outliers.

Could you use this approach rather than the method you outlined in your blog post? Obviously it would leave a few blind spots, such as how fast a person is leveling but is better overall to log or not?

[Harkins]

If you don't have any logs and you want to find who abused an exploit you've only just realized, you're going to have a hard time. If duping an item requires visiting a particular page, or visiting two pages in quick succession, your web server logs will have *something* you can look at, you can compare player IP to any suspicious entries in your access.log. If only a particular item can be duped, look in the db for who has lots of them -- actually, you can do this for *any* item.

Could you use this approach rather than the method you outlined in your blog post? Obviously it would leave a few blind spots, such as how fast a person is leveling but is better overall to log or not?

(For those who didn't see it, what I outlined in my blog post was 1. Hadoop to do big processing of big custom logs and 2. Getting clients to log directly to S3 to save bandwidth.)

I don't see this as "either-or" but as a series of steps you can take or not.

You already have your basic Apache logs to start from and that beats nothing, yep. If you add custom logging of game data alongside it, great, that gives you more potential ways to find cheaters. If you get into stuff like Hadoop,  you can deal with logging tons of data for tons of players and get actionable business analytics out of it. If you have clients log to S3, you can save some bandwidth.

[Barrikor]

One thing to log is any malformed GETs and POSTs.

Hackers will malform the input far more than normal users ever do

[Helderic]

If you don't have any logs and you want to find who abused an exploit you've only just realized, you're going to have a hard time. If duping an item requires visiting a particular page, or visiting two pages in quick succession, your web server logs will have *something* you can look at, you can compare player IP to any suspicious entries in your access.log. If only a particular item can be duped, look in the db for who has lots of them -- actually, you can do this for *any* item.

Could you use this approach rather than the method you outlined in your blog post? Obviously it would leave a few blind spots, such as how fast a person is leveling but is better overall to log or not?

(For those who didn't see it, what I outlined in my blog post was 1. Hadoop to do big processing of big custom logs and 2. Getting clients to log directly to S3 to save bandwidth.)

I don't see this as "either-or" but as a series of steps you can take or not.

You already have your basic Apache logs to start from and that beats nothing, yep. If you add custom logging of game data alongside it, great, that gives you more potential ways to find cheaters. If you get into stuff like Hadoop,  you can deal with logging tons of data for tons of players and get actionable business analytics out of it. If you have clients log to S3, you can save some bandwidth.

Ah, that cleared a little up for me. Thanks!

One thing to log is any malformed GETs and POSTs.

Hackers will malform the input far more than normal users ever do

What do you mean by that? $_REQUEST all data and if it's not something the script wanted log it?

[Scion]

One thing to log is any malformed GETs and POSTs.

Hackers will malform the input far more than normal users ever do

What do you mean by that? $_REQUEST all data and if it's not something the script wanted log it?

This is to keep a record of people that are probing your website for holes....you can assume that most hackers have a similar level of programing ability to yourself. that means that they will be able to view your page source and see how youve set up communication between the browser and the webserver....often theri next step is to start playing around with modifying some of the parameters, or using different values to see what happens....this is an early indication that someone is trying to find exploits.

[Quantz]

I use

Code:

if($HTTP_REFERER)
{
CODEHERE
}
else
{
echo "We don't allow you to navigate through your browser! Please login again.";
$account_class->logout();
}

Doesn't this dimish any ability to mess around with your $_POST's and $_GETS's?

I'm not advanced programmer, so any clarification would be nice. As I rely on this system for my own game.

[Crazy-T]

I use

Code:

if($HTTP_REFERER)
{
CODEHERE
}
else
{
echo "We don't allow you to navigate through your browser! Please login again.";
$account_class->logout();
}

Doesn't this dimish any ability to mess around with your $_POST's and $_GETS's?

I'm not advanced programmer, so any clarification would be nice. As I rely on this system for my own game.

$HTTP_REFERER

Code:

<?php // for colors!!
//Ain't
$HTTP_REFERER //the same as
$_SERVER['HTTP_REFFER']; //could you do
getenv('HTTP_REFFER');

?

[Harkins]

Quantz, if you have Firefox, install the Tamper Data plugin. It makes it trivial to spoof referer and other HTTP headers.

This would just be annoying, it would mean people could never bookmark pages or type in a URL to go to it.

[Crazy-T]

This is what i use to spoof X-Forwarded-for, https://addons.mozilla.org/en-US/firefox/addon/5948