Xbox LIVE Indie Games
Sort Discussions: Previous Discussion Next Discussion
Page 1 of 1 (2 posts)

The Evil Checklist for playtest/peer review

Last post 4/29/2009 5:58 PM by The ZMan. 1 replies.
  • 10/31/2008 11:09 PM

    The Evil Checklist for playtest/peer review

    The Evil checklist is now 'official' and maintained by Microsoft... the current version can be found here

    http://creators.xna.com/en-US/help/peerreview_evilchecklist











    Historical archive below...








    See also the 'Not so Evil list of things NOT to fail for' http://forums.xna.com/forums/t/30487.aspx
    See also the 'Legal FAQ' for legal reasons you should and should not fail for wrt IP issues http://forums.xna.com/forums/t/17830.aspx
    See also the 'language FAQ' for fail reasons related to translations and languges http://forums.xna.com/forums/t/26198.aspx

    The following is a list of the most common reasons because of which games fail peer review, as they have emerged over the first months of XBLCG, plus a list of test cases that are designed to flush out these problems. This list is meant both for developers, who want to test their own game before submitting it to review, and for peer-reviewers, as a suggestion of how they could test a game they are reviewing, and what cases to think of.

    Most Common Review Failure Reasons

    • The game contains any kind of the forbidden content as defined by Microsoft in the review rules.
    • The game crashes anywhere (yes, even crashes in obscure situations will result in a failure if found by a reviewer).
    • The game hangs or appears to hang (again, even in obscure situations this means a failure) or exhibits other game defects.
    • The game can not be played with any controller index (for example it forces you to play with controller #1, or it allows you to start the game with controller #2 and navigate the menus, but then doesn't react to input from that controller in the game, and similar stuff). A user should NEVER have to switch controllers from the one they have in their hand for any part of the game including purchasing screens.
    • The game uses fonts so small, that important parts of the text are unreadable (on some TVs).
    • The game does not observe the title safe area which causes important information or important parts of the gameplay to be off-screen (on some TVs).
    • Misrepresentation of the game or promotional materials that are not appropriate for all ages (screenshots, box art, etc.).
    • The game does not have a trial mode that showcases the actual game experience, or the trial mode is misused to provide the game for free.
    • The game is not a game, but only a wrapper for a video (or several videos), misusing XBLIG as a video hosting service.

     Ratings/Classification Issues

    • 1080p - if your game sets the back buffer to anything other than 1920x1080, then your game does NOT support this. 1280x720 is NOT 1080p. Note that if you do 1920x1080, then the xbox will not scale to 480i for you. You need an extra code path for that scenario.
    • Online players - if your game supports online data sharing (levels, high scores or whatever) then that does NOT mean it's a 2-31 player game. This classification appears on the marketplace and gamers will expect a multiplayer experience. This is a valid fail reason. See this thread/post.

    Test Cases

    The following test cases are meant to flush out the kind of issues mentioned above. Note, that these test cases are meant as a guideline and inspiration only, as a basis for your own testing procedure. They are not meant to be followed slavishly and are not meant as a complete set of tests. A game may very well fail review even if it passes all tests mentioned here, if the reviewer finds some other problem! So be creative when testing and don't rely solely on this list!

    Basic Tests

    Test case 1: Multiple controller support. Start the game with the second (or third/fourth) controller. Verify that the game is playable in various scenarios, for example:
    a) Have controller 1 switched off with no profile signed in during the test (and your CC-profile logged in on controller #2).
    b) Have controller 1 switched on, again with no profile signed in during the test (and your CC-profile logged in on controller #2)
    c) Have controller 1 switched on, with your CC-profile logged in, and no profile logged in on controller #2. The game must not simply ignore controller #2 in this situation, but must still allow you to play with controller #2 (but may ask you to login a profile on #2 first).
    ...or other multi-controller scenarios...
    (Developers: Support for switching controllers mid-game is not required. The Player usually implicitly "declares" the active controller at the "Press Start" or "Main Menu" screen. See "Caveat #6" below for more controller issues you should be aware of.)

    Multicontroller support in multiplayer games:
    d) Verify that the multiplayer game is playable with various controller combinations (and not simply assumes controllers 1+2 for players 1+2, or similar).
    e) If the multiplayer game supports a varying number of players, play the game with the minimum required number of players while leaving an additional controller connected (try both with and without a signed in profile on this controller). Verify that the game does not "force" this additional controller to partake in the game. (Developers: See "Caveat #6" below for more details.)

    Test case 2: Profile logout. Log in with a CC-premium membership gamertag on the first controller. Log in with another gamertag on the second controller*). Play the game with the second controller. Then at various locations in the game (for example in the main menu, or in the middle of the gameplay), use the guide button on the second controller and sign out the second gamertag. Verify that the game handles this gamertag sign-out correctly. May be repeated to check sign-out behavior at different locations in the game.

    Test case 3: Storage device support.
    a) Start game with both a hard disk and a memory unit attached as storage devices and select one of them when asked. Does is start properly?
    b) If so, start the game again and this time, close the storage selection dialog with "B", i.e. do not select any storage device. Check that the game handles this properly. For example it either tells you that you have to select a device and re-opens the selector dialog, or continues without saving (but of course also without crashing!).
    Advanced test: If you cancel the selector dialog and the game then shows a message box of some kind, verify that this message box reacts to your input even if you are not playing with controller #1.

    Test case 4: Removing the storage device at any time. If case 3 does not cause problems, select the memory unit as the storage device and remove it. Test removal prior to AND during load/save operation as well as during device selection screen.

    Test case 5: Video modes, font sizes and safe area. Start game in various video modes, whatever the TV supports. Check if the game crashes or if anything is drawn in such a way that the game is unplayable (important stuff outside of titlesafe area, fonts too small to read, etc.).

    Test case 6: Marketplace. If the game attempts to show the marketplace page for purchasing the full version while only a local gamertag is logged in (or no gamertag is logged in at all), an exception is thrown. If this exception is not handled correctly, it causes a crash.
    To test if this is handled or not, log in with either a local gamertag or with no gamertag at all (if the game allows this) and run the game in simulated trial mode. Then check, that any menu items which would normally allow you to purchase the full game are either hidden or, if selected, do not cause a crash.
    Note: Do not use a Live gamertag that simply has restricted permissions (for example a child account) for this test. Such an account will not trigger the exception/crash that this test is attempting to provoke.

    Test case 7: Guide open during startup/dialog. Start the game and immediately open the Guide (i.e. select "Play Game" on the dashboard and immediately hit the Guide button while the game is still starting up). The guide will flicker on/off a few times. Wait until the flickering is over and the XNA startup screen in the backup is no longer visible, then close the guide. Does this cause a crash? Its also possible to crash if you time the Guide when another dialog (keyboard/device selector) is about to appear. Checking Guide.IsVisible is not 100% - always wrap your fialog code with try/catch.
    (In case you are testing your own game and actually are getting a crash, see the following thread for a discussion about how to avoid this: http://forums.xna.com/forums/p/19874/103843.aspx .)

    Test case 14Guide.SimulateTrialMode. Games that leave the Guide.SimulateTrialMode flag on during review should be failed even though it has no effect in the marketplace because reviewers cannot follow all code paths to determine crashes/hangs or bad content. See http://forums.xna.com/forums/p/30287/224320.aspx#224320

    Advanced/Special Tests

    Test case 8: SpriteFont issues. If the game has some sort of text input by the player (for example to enter your player name for a highscore list), try entering a non-standard symbol and/or foreign character. This can cause a crash if the font chosen by the developer does not support this symbol/character. (Developers: Use SpriteFont.Characters.Contains(char) to check if your font supports all chars entered by the player.)

    Test case 9: Locale issues. Start/play the game while your Xbox dashboard is set to a non English language, to check for local specific bugs:
    a) Set the Xbox to a language which is associated with a locale where a "," is used as the decimal separator in float values (instead of the "." used in English/US type locales). This allows you to check for the floating point parsing bug (see "Caveat 1" below). Games that are affected by this bug exhibit all kinds of strange behavior (I've seen problems like not even starting up correctly, hanging at some point during data loading, drawing everything totally out of scale, etc.).
    Languages that can be used for this are for example Spanish, French and German.
    b) If the game outputs number values in formatted form using thousands separators, then set the Xbox to a language which is associated with a locale where a non-breaking space is used for the thousands separator (instead of the "," used in English/US type locales, so that for example "1,234,567" would look like "1 234 567" instead). Then do something which causes the game to output a number >= 1000 (so that the thousands separator is required). Make sure that this doesn't crash the game (see "Caveat 4" below).
    Languages that seem to use the non-breaking space separator seem to be Russian, Polish and French.

    Test case 10Network/Multiplayer issues. Special case for LIVE multiplayer games: With an established multiplayer network session, try pausing the game (both with the game's own pausing functionality, if it has one, and by opening the Guide). Does the game handle this correctly? If the game indeed pauses (and not simply continues in the background), then this should not interfere with the other players (at least not in an unexpected fashion) and it must also not lead to a network timeout if you leave the game in this paused stated for a while (which happens if the developer forgot to update the network session in paused mode).

    Test case 11: Verify claimed capabilities. When a game is submitted, the creator checks a set of "capabilities", for example how many players the game supports, if it supports co-op play, the supported resolutions, if it supports "Custom SoundTracks", etc. Check if all these claims are indeed true (as far as this is possible for you).
    Tip: To check if "Custom SoundTracks" are indeed supported, try playing some music through the Xbox (from a CD, your music library, etc.) while the game is running. The game must then automatically switch off its own music and only play sound effects.

    Test Case 12: Screen resolutions. If the game follows the best practices and simply sets the screen to 1280x720 (720p) then there are no reported issues. However, if the game uses 1920x1080 (1080p) then the code must have a special code path to deal with 640x480 as the Xbox cannot scale 1080p down to that size. In addition, note that any user with a VGA cable can set many different resolutions, some with odd aspect ratios. If the developer has written code to change behaviour based on screen size then one of the VGA resolutions may not have been tested. (See http://forums.xbox.com/28011158/ShowPost.aspx for a case 'in the wild').
    To test, set 640x480 and other odd resolutions in the dashboard. See http://hardware.teamxbox.com/reviews/xbox-360/40/Xbox-360-VGA-HD-AV-Cable/p1/ for the resolutions and aspect ratios.

    Test case 13: 'Restricted' XBLA terms. Microsoft has asked that creators do not use the words "Achievement" or "Leaderboard" for features with similar functionality. See http://blogs.msdn.com/xna/archive/2008/12/12/creators-club-communiqu-11.aspx and see this thread for discussion and alternate names: http://forums.xna.com/forums/t/21124.aspx. Check that the game does not use any of these terms either in-game or in the description, screenhots or video. If they do then FAIL for 'Innapropriate for xbox live'

    Test Case 15: Test Media libary APIs with a media server/fileshare If your game uses the Media Library APIs to iterate through songs or albums then ensure you test with your xbox connected to externall file stores such as Media Server or other file shares. While the APIs will eventually return there are orders of magnitude in speed compared to your xbox hard drive or USB sticks. It is likley that your game will appear to have hung/crashed to end users with this setup.

    *): For easier testing of "test case 2", the second gamertag should either be a local gamertag (not Xbox Live enabled) or a gamertag that also has a CC-premium membership.
    This is because if the second gamertag is a Live gamertag without CC-premium membership, then you won't be able to start the game while this gamertag is signed in. If you want to use such a gamertag anyway, the process is a bit more difficult, because you need to do a "late sign-in": Keep the second gamertag logged out at first, then start the game with the first controller. When the usual start screen that says "Press Start to Begin" appears, use the second controller to finally log in the second gamertag now (when the game is already started). Then press Start on the second controller to play the game with it.
    Note again: This "late sign-in" is only necessary if the second gamertag is a Live gamertag without CC-premium membership. For a local gamertag or a CC-premium membership gamertag, this "late sign-in" is not necessary.

    Development Caveats

    The following are not really test cases for reviewers, but common caveats that have already been encountered by developers several times and which have required bug-fix releases or various resubmit-cycles during review. So the caveats are listed here more for the benefit of developers who use my little "evil" checklist to test their own games before release, than for reviewers.

    Remember to also have a look into the Best Practices for Community Games!

    Caveat 1: Floating point number parsing. If your game uses textual representations of floating point numbers in its save data (either in game data or in game saves) and then needs to parse this text back into a float value while loading, then you must be careful to avoid culture/locale-based parsing problems. (The "test case 9" of above is an attempt to find bugs caused by this.)
    Background: Different countries use different characters as the decimal separator. For example in the US and UK, a dot "." is used. I.e. "1.5" or similar. But for example Spain and Germany use a comma "," - i.e. "1,5" or similar.
    Whenever you convert a float into a string with "value.ToString()" or parse a text into a float with "float.Parse(text)", the .NET framework will use the decimal separator that matches the currently selected locale. On the Xbox, the locale in turn is defined by the language that the Xbox dashboard is set to. So if you have a value of 1/2 and do "value.ToString()" on it, then on a Xbox set to English, the resulting string will be "0.5" but on a Xbox set to Spanish, the resulting string will be "0,5". Similarly, "float.Parse(text)" will also expect the separator that matches the currently language, so if the text "0.5" is parsed on a Spanish Xbox (which expects a "," as the separator), the result will be incorrect.
    This has a lot of potential for bugs: Assume that you deploy your game with data that uses the "." and this game is played on a Spanish Xbox. Loading your data will fail, or at least give unexpected results. Or assume that someone plays your game on a Spanish Xbox and saves his game. In the game save, a "," will be used as the separator. Now the player changes his Xbox language to English and restarts the game. When is game save is loaded, the parser will expect a "." as the separator, which does not match the "," that is actually in the data and will thus result in a loading error.
    Therefore, you should under all circumstances avoid plain calls to value.ToString() and float.Parse(text).
    Instead, use value.ToString(CultureInfo.InvariantCulture) whenever your need to convert a float to a text and float.Parse(text, CultureInfo.InvariantCulture) whenever you need to parse this text back into a float.
    This will have the effect that the data is always written and parsed with a dot "." as the separator.
    Of course you then also need to make sure that all data that you deploy with the game also uses the dot "." as the decimal separator.
    Note: Simply "plussing" a float to a string is comparable to using the plain value.ToString(). So you should avoid doing stuff like "this is text" + floatValue + "more text" when saving data. Instead use "this is text " + floatValue.ToString(CultureInfo.InvariantCulture) + "more text".
    And of course the above applies only to saving data. If you want to convert a float into a text for display on screen, you should continue to use the normal value.ToString(), so that each player sees the value with the correct decimal separator for his country.
    Note, that the XMLSerializer is not affected by this problem. It is only something that you need to look out for if you are writing/parsing float values manually.

    Caveat 2: Corrupting saved data by overwriting an existing file with shorter data without truncating it. When you open a file for writing with FileMode.OpenOrCreate and the file does already exist, then the file will not be truncated, but will simply be opened. If you then write data into this file, but the new data is shorter than the data that is already in the file, then at the end of the file some trailing bytes from the previous data will remain. Depending on your save-format, this may corrupt the saved data, if during a subsequent loading of the data, the trailing "old" bytes will cause a loading error. This is for example the case if the save data is serialized XML (as there then will be superfluous XML tags from the old data after the actual end of the new XML data). Unfortunately, the XML serialization example in the XNA documentation actually tells you to use FileMode.OpenOrCreate, which is however not a good advice, because of the above problem. Instead, you should use FileMode.Create (this will create the file if it does not exist, or truncate it if it already exists, so only the new data remains in the file after you write it). According to what Shawn says here, this should be good enough when coding for Xbox and Zune. The "safe save" method recommended elsewhere is not necessary then, but is still recommended when coding for Windows.
    (There's not really a good test case that can be described for finding bugs caused by this caveat. If you want to test for this, your best bet would be to force the game to save and load data repeatedly. For example, do something that requires saving, then restart the game to force loading, repeat a few times.)

    Caveat 3: Storage device handling. Incorrect handling of the storage device selection is one of the main reasons why games fail peer review. The developer should be aware of a few facts about storage device selection:

    • The device selector dialog appears only on an Xbox that has more than one storage device (i.e. a hard disk and a MU). If the Xbox has only one storage device, no dialog ever appears, but the single device is selected automatically. Therefore, to test the various selector specific cases, you really need a MU.
    • The player is able to cancel the device selector dialog with "B", which means that the game proceeds without a selected device. The game must handle this or it will crash as soon as it attempts to load or save.
    • If the player selects a MU, he is able to later disconnect it in the middle of the game. If the game does not handle this, it will crash as soon as it attempts to load from or save to this device.

    Caveat 4: Number output with thousands separator. Your game may crash if you are trying to output formatted number values >= 1000 in a format that includes thousands separators, if your SpriteFont does not include the thousands separator character that is used by the locale that the Xbox is currently set to.
    Background: This caveat is related to caveat 1 in so far, as it also is caused by locale specific differences in number formats. Depending on the locale that the Xbox is currently set to, the thousands separator may be a different character than the "," used by the English/US-locales. Some locales seem to be using a non-breaking space as the thousands separator. This looks like a normal space, but has an ASCII-value of 160 (0xa0), instead of 32 (0x20). So a number that looks like "1,234,567" in an English/US-locale may look like "1 234 567" in a different locale. Now, if your SpriteFont does not contain the non-breaking space (for example because you only specified a character range up to 128 when you created the SpriteFont, so that the code 160 was not included), then your game will crash if it tries to output a formatted number that is >= 1000 on a Xbox with such a locale.
    To avoid this problem, either make sure that your SpriteFont contains the necessary character (160 / 0xa0) and/or when formatting your numbers, make sure to use a format that only generates characters that are supported by the SpriteFont(s) you want to use to output them.

    Caveat 5: Using Guide.BeginShowMessageBox before PlayerIndex is known. For a call to Guide.BeginShowMessageBox, you must specify a PlayerIndex. This means, that only the controller for the given index will be able to control the dialog. This again means, that you should never use this to open a dialog before you have actually detected the correct PlayerIndex. Such a detection is usually done via a "Press Start to Play" screen or something similar. So you should not use Guide.BeginShowMessageBox before this screen, because if you do, then you have no idea what PlayerIndex to supply. And if you supply the wrong one, then the player will not be able to control the dialog (which looks as if the game is no longer responding to input and is hanging).
    A common case where this is forgotton is the following: First the storage selector is shown (player independent), if the player cancels it, the game then tries to show a message box (to inform the player that saving is disabled, or that he has to select a device, or to ask him if he wants to select a device or continue without saving, etc.). If all this happens before the PlayerIndex is detected, then most likely the default PlayerIndex.One will be supplied when showing this message box, so for anyone not playing on controller #1, the message box will not react to input.

    Caveat 6: Controller Caveats. The following items are some controller related issues that have led to problems and review fails which a developer should be aware of:
    a) Note that wired guitars and drum kits will often grab controller #1 and #2 and the auto logged in profile, so people often play on controller #3 or higher with their normal controller and may not be logged in.
    b) Note that thumbsticks on some controllers do not register [0,0] even when not actively moved, and guitars (or other controllers) will make the right thumbstick register != [0,0] due to whammy bar and tilt. Therefore, such controllers and guitars can cause trouble if you try to determine the active controller not only by polling button presses, but also by polling thumbsticks (see http://forums.xna.com/forums/t/40701.aspx). The general advice is therefore not to do this. Instead, make a start screen that checks for "Start" and "A" button presses to determine the active controller.
    c) Beware of the incorrect condition in "for (PlayerIndex p = PlayerIndex.One; p < PlayerIndex.Four; p++)" or similar, because it will miss controller #4. There's been several games that don't respond to #4 as a result. Test with a fully loaded console! (The correct code would use "<=" in the condition instead of "<").
    d) Regarding multiplayer games: Sometimes people just want to watch but not play, or a wired guitar or drum kit or similar may still be connected. So in a multiplayer game, you can not simply assume that every connected controller equals a player who wants to partake in the game (even if a profile is signed in on that controller!). Therefore, you can not simply "force" every connected controller to be a player in your multiplayer game. Instead, to select the active controllers that want to partake in the game, implement an "[A] to join, [B] to leave, [Start] to play" UI to allow any controller to play or not. It's OK to assume that the controller that started the game wants to play, so feel free to already include it if you want.
    e) Do not use a profile that is logged in another controller to let a player do something that would not be possible with the profile that is logged in on the controller the player is using! This circumvents security features and also can be very confusing for the player. Or more generally: To perform an action that requires a PlayerIndex, do only use the PlayerIndex of the controller that triggers/executes the action. If the action is not possible with the profile logged in on that controller (if the profile does not have the necessary right, or no profile is logged in at all), then handle this accordingly, in a way that is not confusing to the player. (For example disable the menu item, or show an info message, or bring up the sign-in dialog, etc.).
    This may seem self-evident, but there have been games that ignored this, for example to let a player into an online match even though he was playing on a controller without a Gold-profile (by using the PlayerIndex of a second controller for all network access, a second controller on which a Gold-profile happened to be logged in). Do not do this!
    Note that the 8 minute trial screen in XNA Game Studio connect is an EMULATION for dev purposes. It will let anyone purchase the game without any checks and is outside of a developers control. Do NOT fail for this. After release, the marketplace will check correctly.

  • 4/29/2009 5:58 PM In reply to

    Re: My little "evil" checklist for peer review stress testing...

    This post is now a sticky FAQ type post.

    Any discussions on it should be in this thread: http://forums.xna.com/forums/t/30287.aspx. SypnDoctor or the mods will update this post as required.

Page 1 of 1 (2 posts) Previous Discussion Next Discussion