Automated tests for game mechanics

[dsheroh]

I write software In Real Life and first-hand experience has sold me on the value of using unit tests and automated testing in general, so I naturally want to reap these benefits in my side projects as well, including games.

Unfortunately, automated tests tend to rely on knowing the expected results when the test is written.  While this is fine for business applications and infrastructure code, where there is One Right Answer that should always be produced, it doesn't work so well for game mechanics.  Game mechanics tend to have random elements specifically so that you won't know the result of an action in advance and, even if they don't, there are lots of details which are being constantly tweaked so that they'll end up feeling right.  (e.g., Market pricing code may be deterministic, but that doesn't mean you won't be fiddling with the rate at which available quantities, trade skills, etc. cause the final price to change.)

Using a fixed seed for the PRNG can help with the "testing randomness" part somewhat, by ensuring that the same set of random numbers will be returned every time through the test, but that still doesn't address the issue of the correct final results changing constantly as you adjust the calculations being done on them.  It also doesn't feel right to me that it should be getting tested only with a single, specific sequence of "random" inputs (although, admittedly, any other automated test will also be testing with only a single set of inputs, so this isn't really all that different).

Does anyone know of a good solution for this which will allow for automated testing of game mechanics which produces results more meaningful than "well, it didn't crash..."?

[Harkins]

I know the Angband series of roguelike games test with a simulated player. The developer has blogged about how he keeps an eye on how it plays new versions, if it tends to die unreasonably often to new things, etc.

Certainly not as simple and easy as unit tests, though.

[Chris]

"I have a solution, now I need a problem"

Try balancing items, monsters and attributes. You want to let player progress 1 level each day. Generate several sets of monsters/items/exp formulas and run bots that will slay enemies and buy equipment. Select the set that was nearest your desired progress pacing (be aware of overbalancing, if you make a perfect balance the game will be boring since player won't be able to discover the "best method" if all methods became equal).

Of course this is purely theoretical, autotesting via real players who play your game is far superior, simplier and cheaper

[jannesiera]

I don't know much about automated tests and such but using it for game mechanics sounds like trying to automate the stock market. It doesn't have the human feel, the fun factor or the designer's insight.

Anyway, Chris is probably right. Testing with real players is simpler and much cheaper.

[lolninja]

Game mechanics tend to have random elements specifically so that you won't know the result of an action in advance...

The first though that enters my head is to have function somewhere that generates your random numbers, this way using reflection you can just replace the random number generator with a "return 15;" job, or alternatively always pass in your random number, so your functions look like...

Code: [Select]

function hit(attacker, defender, random) {
    ...
}

Though this falls down when you need lots of random values, so I think using reflection to replace the random generation function is best.

[Nox]

A way to test such random things might be to do many passes on the same set, maybe with recording stat data like min/max/avg... although it won't be precise, it might give some insight

[Marek]

"I have a solution, now I need a problem"

I think the thread ended right there. </cynical>

[dsheroh]

"I have a solution, now I need a problem"

Not exactly...

Try balancing items, monsters and attributes. You want to let player progress 1 level each day. Generate several sets of monsters/items/exp formulas and run bots that will slay enemies and buy equipment. Select the set that was nearest your desired progress pacing (be aware of overbalancing, if you make a perfect balance the game will be boring since player won't be able to discover the "best method" if all methods became equal).

Of course this is purely theoretical, autotesting via real players who play your game is far superior, simplier and cheaper

I don't know much about automated tests and such but using it for game mechanics sounds like trying to automate the stock market. It doesn't have the human feel, the fun factor or the designer's insight.

Given these answers, I see that I wasn't clear about what I want to do tests on...  Fun gameplay and good balance absolutely have to be evaluated by live humans.  The automated/unit tests I'm talking about are to ensure that the code works, that it works correctly, and that any changes I've made have not introduced any bugs.

[bbgames]

Game mechanics tend to have random elements specifically so that you won't know the result of an action in advance...

The first though that enters my head is to have function somewhere that generates your random numbers, this way using reflection you can just replace the random number generator with a "return 15;" job, or alternatively always pass in your random number, so your functions look like...

Code: [Select]

function hit(attacker, defender, random) {
    ...
}

Though this falls down when you need lots of random values, so I think using reflection to replace the random generation function is best.

I tend to do something like this for my games when I want to do tests on them - when I'm testing, things get monkeypatched so that only certain monsters come back, items get dropped, etc. I don't get to test everything that way (although I suppose I could, if I walked through every single monster), but it works well enough for making sure things are working the way they should.

Also, an automated player (like Harkins) described would probably work too - if your game was turn-based, and you wanted players to be able to complete the game in <x> turns, just set up your tester and make it play like someone who has no idea how to play your game, and keep track of how many turns it takes. Sure, things in your game are random - but there's a certain number of turns that they're supposed to finish in, right?

[famulus]

I've only read about unit testing so please correct me if I'm wrong, but won't making sure you have unit tests at the very fine levels of granularity mean that a unit test for an algorithm ends up being a collection of unit tests consisting of tests for the underlying classes/functions that are being used along with a few tests for algorithm specific functions which would return a predictable result from a given input?  In the case of algorithms that require a random element, can't you test a) that the random number generator works, b) the functions produce the desired output for specific inputs, and c) test that you get valid results from the lower and upper bounds for your random element?

[Chris]

The automated/unit tests I'm talking about are to ensure that the code works, that it works correctly, and that any changes I've made have not introduced any bugs.

It is almost useless for BBGs.
- you can select a few dozen of players, give them awesome tester badge, setup a separate server with newest version and they will do it for you (almost 0 coding neded, only the code to display the awesome tester badge )
- traditional bugs are very rare in BBGs (compared to C products), also if you made it there is low danger (dividing by zero will not crash anything, just produce an error) also testers/players can quickly catch it also you can path it extremely fast (as opposed to retail games).
- the biggest threat are balance/gameplay "bugs" which are impossible/very hard to test via automated script.
- you will need human testers for emergency situation (like there is a serious bug and you need to fix it within 24 hours at most and you are already tired because you were writing the fix whole night and have no stamina or time left to write/adjust the auto testing script). In such case real testers are far superior and you simply need them anyway.

Assuming you made a perfect auto test to catch bugs, you still will need to use real testers to evaluate balance, layout, new interface, ease of understaning of the new manual describing the new features, colours choice... If you could make the automated thing with zero cost and absolute perfection, why not. Personally, I would not let any new code to my main games without human testing, no matter how good the auto catch bug script would be (like if your auto tester would caught 90% of bugs it would be useless to my standards as the only testing method).

[bbgames]

The automated/unit tests I'm talking about are to ensure that the code works, that it works correctly, and that any changes I've made have not introduced any bugs.

It is almost useless for BBGs.

I think you might have missed the point of automated testing - it's not so that you can make sure that your game is perfectly balanced and the layout works, it's so that you can avoid the 'emergency situations' you described.

Tests aren't there to make you do more work or spend more time - they're there to remove the fear from deployments.

[aerosuidae]

Unfortunately, automated tests tend to rely on knowing the expected results when the test is written.  While this is fine for business applications and infrastructure code, where there is One Right Answer that should always be produced, it doesn't work so well for game mechanics.

You may not know the One Right Answer for game mechanics, but you can often (always?) define the bounds within which the answer should fall.  Rather than test for "exactly right", test for "not wrong".  Unit testing at the module/class/method levels is still perfectly relevant to games.

Does anyone know of a good solution for this which will allow for automated testing of game mechanics which produces results more meaningful than "well, it didn't crash..."?

For overall game mechanics testing, I find it interesting to try to write a bot or AI to play a game.  Granted, you can never create an AI to rival the unpredictability of human testers, but even a crude AI can be useful for finding edge case bugs (stuff overflowing or loopholes in game play) as well as verifying the overall game balances remain in the right ballpark.

1. Change some code.
2. Run unit tests looking at limits and bounds.
3. Set a couple AIs to play many times overnight, logging activity.
4. If #2 passed, and #3 stats didn't go crazy, release to a test server and some humans.

my $0.02  

[dsheroh]

The automated/unit tests I'm talking about are to ensure that the code works, that it works correctly, and that any changes I've made have not introduced any bugs.

It is almost useless for BBGs.

I think you might have missed the point of automated testing - it's not so that you can make sure that your game is perfectly balanced and the layout works, it's so that you can avoid the 'emergency situations' you described.

Tests aren't there to make you do more work or spend more time - they're there to remove the fear from deployments.

Yep, exactly!

Unfortunately, automated tests tend to rely on knowing the expected results when the test is written.  While this is fine for business applications and infrastructure code, where there is One Right Answer that should always be produced, it doesn't work so well for game mechanics.

You may not know the One Right Answer for game mechanics, but you can often (always?) define the bounds within which the answer should fall.  Rather than test for "exactly right", test for "not wrong".  Unit testing at the module/class/method levels is still perfectly relevant to games.

*facepalm*

Thanks.  Why didn't I think of that?    Instead of testing "price with 5 in stock == X and price with 10 in stock == Y", I can test "price with 5 in stock > price with 10 in stock".  Instead of testing "level 2 character fights level 1 monster and walks away with Z hp left", I can test "level 2 character beats level 1 monster at least 9 times out of 10".

Works for me!

For overall game mechanics testing, I find it interesting to try to write a bot or AI to play a game.  Granted, you can never create an AI to rival the unpredictability of human testers, but even a crude AI can be useful for finding edge case bugs (stuff overflowing or loopholes in game play) as well as verifying the overall game balances remain in the right ballpark.

1. Change some code.
2. Run unit tests looking at limits and bounds.
3. Set a couple AIs to play many times overnight, logging activity.
4. If #2 passed, and #3 stats didn't go crazy, release to a test server and some humans.

That's actually something I need to be careful with, personally...  I tend to start by writing automated play, then add in the ability for human players to interact with the game rather than doing it the other way around.  I ended up abandoning one of my earlier game attempts (Crystomere, a web-based persistent RTS, more or less) when I realized that I'd been so focused on the AI players that I'd created something which couldn't deal with players who weren't logged on and active 24/7/365...

[Chris]

Tests aren't there to make you do more work or spend more time - they're there to remove the fear from deployments.

Nothing is going to remove the fear from deployments for BBG owners

[JGadrow]

Nothing is going to remove the fear from deployments for BBG owners

If you're afraid of deploying new code, then you haven't tested it well enough. Again, they're speaking only of the fear that they have coded something that is a programmatic impossibility in their application. Something they might have missed in manual testing but would cause major bugs or even break the application.

Automated unit tests are great because once you code the test, it's there forever. I dunno about you but the hardest thing about manual testing it to specifically stat out all the cases that must be tested. And, as I've said before, then someone has to test it and human time cost more than machine time. Perhaps your games work well with these "tester badges" but I would feel better for having a suite of automated tests run before having a human test phase. That way you can avoid getting egg on your face for stupid errors that should have been caught during development. Not balance-related issues or things of that nature.

*Edit - Grammar edit.

[dsheroh]

Automated unit tests are great because once you code the test, it's there forever.

That's definitely my favorite part!

The Psi Rangers test suite currently stands at 311 test cases, which means 311 things that I know work after I run the tests - or, if you want to look at it the other way around, 311 things that, if I break them, I'll be immediately notified the next time I run the test suite.  Given that the tests take a total of around 30 seconds to run (I just ran them and they were timed at 29.027 seconds), there's no reason not to run the tests at least daily when working on the project and I usually run them before and after each time I make significant changes to the code so that, if any of those 311 things break, I immediately know exactly where the problem was introduced without having to think about it - if the tests ran OK, then I changed three lines of code and the tests broke, then it had to be something in those three lines that caused the problem.