XNA Peer Review System
July 26, 2011 10 Comments
Normally at this time I would be posting my latest Indie review. But today I have something different to talk about. Edward Di Geronimo of Saturnine Games sent me a request to review his latest game, Antipole. And although I really had a good time with it (expect a full review later today) I had some pretty major technical issues while playing it.
I had gotten about forty-five minutes into the game and was really enjoying it. I did notice an occasional hiccup in the frame rate when I would destroy an enemy, but thought nothing of it. And then, later in the game, the hiccup suddenly became a major headache that caused the game to skip like a scratched DVD. Combined with acid pits that zap all your life instantly, I had a legitimate problem on hand.
Since Ed at Saturnine Games requested I review Antipole, I figured I would just ask him if he was aware of the problem. He told me that he had heard of it from two other people. After comparing notes with his previous reports, I tried the game again and it briefly seemed better, but right before the final boss fight the skips returned. I did manage to beat the game, but did so only by learning to work within the skipping issue. I suppose Ed could just roll with it and call it a feature. It works for Microsoft.
Having been told previously with The Cannon that some games are compatible on some Xboxes and not on others, I once again compared notes with Ed. I learned that both previous reports he had came from guys who were using completely different hard drives and systems than I was. I eliminated my hard drive as the cause and switched it over to my Xbox Elite circa 2007. I was easily able to recreate the same problems. Ed reported to me that others had brought up slowdown but didn’t find it significant. And perhaps it’s not, as I was still able to finish the game and despite being frustrated by the cheap deaths the slowdown caused, I still really, really enjoyed Antipole. But suddenly what I was told had been reported by only two people now sounded like more than two.
Ed was more than gracious when I brought my concerns to him, and we are both working together to figure out what’s going on. Shortly before this piece was posted, Ed contacted me to let me know that he himself found the slowdown issue. This was disappointing for him, but at least we have established something is wrong. Which is odd because Ed assured me that Antipole passed Peer Review, and that the problems with it were isolated to myself and two other reviewers. Oh, and all those other people who mentioned it to him as well.
I’m going to chalk this whole episode up to the peer review system being fundamentally broken. Some developers have confided to me that they believe the system doesn’t work because it creates an atmosphere of “you scratch my back and I’ll scratch yours.” I’m not a developer of XNA, but as an outsider looking it I’m going to say that people who practice this are just screwing themselves.
When you do your peer reviews, holding back on any criticisms related to functionality of games is not going to help you, the developer you’re reviewing, or the XBLIG platform. Developers have told me they feel pressure to give positive reviews out to help their own games pass peer review with minimal resistance. Some have told me that before this became the norm, games could end up waiting in line to pass review for months.
I’m certainly not blaming developers for the system being pretty sucky. That’s on Microsoft and nobody else. But for now it’s all you guys have, so you should make the best of it. And that doesn’t include letting glitches or other issues slide by because you’re afraid that telling someone will render them butthurt and they will retaliate against your game. You guys are a community and this means helping one another. I seriously doubt anyone’s feelings will be hurt if they are told there is a major flaw in their game that hampers it playability. And yet seven separate developers have told me that they feel that making too much noise puts their own game in danger.
And I want to say that very few of those who I’ve spoken with have said that it’s a hostile atmosphere. Instead, they say that developers create pressure on themselves upon reaching Peer Review. At this point I’m sure seeing the light at the end of the tunnel is an exciting time. You’ve worked hard for months or maybe even years and finally the game is near completion. Developers I’ve spoken with have said that Peer Reviewers have told them that they would give a game good marks even if there was a design flaw and that they could just patch it at a later date. Others have said they feel XNA developers sympathize with being in the same situation and give a game passing marks whether it merits it or not. Or, I’m told that Indie games are just held to a lower standard and glitches are expected by people who do the Peer Reviews. Some believe a lot of the Peer Reviewers don’t even play the games at all.
I’m told this has led to alliances within the XNA community. The “back scratching” scenario I mentioned above, which helps assure that games pass peer review in a more expedient manner and reach the marketplace in all likelihood well before they’ve been subjected to true play testing. Developers make deals to give each-other good marks and pass the review. This isn’t Survivor, guys. The only thing this is going to do is get XBLIG voted off the Microsoft island. Permanently.
I would also like to say that developers should not depend on Peer Review alone. Play through the game yourself. Get your friends to do it. Take notes. Ask questions. Try to break it yourself. Not every little glitch will get discovered, but letting the big ones slip by reflects poorly on the XBLIG market as a whole.
This isn’t me trying to publicly call anyone out. This is me issuing a challenge. Despite all it’s flaws and it’s status as the red-headed step child of the Xbox 360, the Indie platform is a wonderful opportunity for all of you. But abusing the system Microsoft set up for you is not going to be the foot in the door of the game industry that you want it to be. The odds that your game is going to make enough profit to propel your entry into the industry are slim. Developers should treat XNA as a hobby and a chance to build a resume and nothing more. Don’t operate under the assumption you’re going to make a career out of this. The vast majority of you will not. In that spirit, look out for each other. Don’t be afraid to hurt feelings, and don’t worry about someone taking exception to finding a glitch in your game. If someone does threaten retaliation for a poor Peer Review (and I’ve been told there have been some thinly veiled hints of such things), the community at large will deal with them. At least I hope so.
In my month of doing Indie Gamer Chick, the amount of talent I’ve seen floors me. There’s some really amazingly gifted game developers in your community. So in closing, help each other. Nit pick. Offer advice. But do not blow smoke up each others asses. You’re all adults. Albeit adults who make games about rampaging squirrels, flatulence on dates, or God knows what else. Actually I wouldn’t be shocked if someone reads this and green lights a game where you literally blow smoke up something’s ass. It’s just so Xbox Live Indie Game.
Before the publication of this piece, copies of it were sent out to many developers who had not previously voiced concerns to me about issues with XNA. Of the ten people who received a copy of this, eight signed off on it as being completely accurate, with two not responding at all. So don’t shoot the messenger.
Indie Gamer Chick 
Peer review is not for finding problems with gameplay, it is for making sure that none of Microsoft’s rules are broken.
There’s a playtesting section to the apphub site, but most people put their games straight into review because they want their games out as fast as possible.
“Peer review is not for finding problems with gameplay, it is for making sure that none of Microsoft’s rules are broken. ”
If that’s the case, you might want to have a word with developers, because without fail whenever I tell a developer about a major glitch in their game, their response is “Well, it passed Peer Review.”
That is the case. But most reviewers don’t put in the time really required to find all those glitches / bugs, especially ones that come about from playing too long. Devs shouldn’t have that kind of attitude either and reporting major glitches / issues that should not have gotten through peer review could lead to those games being pulled from the Marketplace.
To clarify something, the MS rules that CSR speaks of largely cover most of the bugs and glitches that you would want to complain to the developer about. So to a lot of developers saying something passed peer review to them means it’s free of serious bugs. And generally that’s true, but as Kris mentions many developers do a half-ass job of peer reviewing and so bugs, even major ones, get missed.
Just look at FortressCraft, which got through peer review with a bug that completely wiped out your saves. Obviously that wasn’t as thoroughly reviewed as needed, and should have failed. Other games make it through with bugs, but they’re not bugs that MS considers the failable type. Still other games fail in review for things that most reasonable people don’t think are an issue, but since it breaks a MS rule it’s a fail (if the reviewer is being diligent anyway).
I’d like to clarify this point:
“Ed reported to me that others had brought up slowdown but didn’t find it significant.”
These reports were specifically clarified as being skipping a frame or two on very rare occasions and only being noticeable if you were looking for it. That matches the behavior I had seen myself for the first two builds that went on the marketplace – I’d see a barely noticable frame skip once or twice in a complete playthrough of the game. I believe that I only noticed it because I was actively looking for it. I consider this totally normal.
This current build is different. I took some stabs in the dark at fixing the issue and didn’t play through the entire game to test it. I submitted to playtest and received no responses. Peer review went quick with only a few replies. Now that I’ve played it completely, I see that I’ve clearly made things worse. On the bright side, it gives me an idea where to look.
Sounds to me like Ed didn’t spend enough time play-testing it on a real console; It’s usually very easy to fall into the trap of just doing all your testing on the development PC itself, but it is a bad habit as some issues only present after long term aggressive play (playing the game as it WASN’T intended to be played) on a real life console.
Even then there is no guarantee that you’ll find it, but then that’s what the play testing section is for…
I’m not saying that the dev was being lazy or deliberately dismissive, I’m just saying that he may need to amend his approach to play testing and quality control.
(And don’t even get me started on dodgy reviewers… I copped all sorts of flack for legitimately failing FortressCraft only to see if pass in minutes (complete with major bug) on its second attempt which means that none of the passing reviewers could possibly have done a legitimate test cycle before passing the game… grumble grumble grumble)
The game was tested *very* extensively on multiple consoles and went through play test and peer review multiple times.
At this point, it looks like the root of the issue seems to be that on a small percent of Xbox systems the garbage collector takes far longer to run than it does on most systems.
Fair enough then; I was simply taking a guess based on personal experience.
I once had a play test build that Code 4’d on a particular menu command on the Xbox that I was completely unaware of as it didn’t cause any problems when running on my PC.
I didn’t even think of the Xbox’s GC problem (though in hindsight it is extremely obvious) as I tend to steer clear of dynamic lists for my game objects, preferring instead to simply recycle slots of a static array.
Peer Review has a serious lack of “hoops” for developers to jump through, especially when it’s their mate holding the hoop at ground level.
It’s not all a problem with peer review though, most people rush through the development phases for their game and don’t even bother to play test. Test with your mates, play test THEN peer review. 🙂
This is exactly what is happening with the peer review process. There is a group of people on there that work together and will fail your game without even playing it. So at that point what do you do? You have to play the game that they have designed, there is no other way. I did this for awhile and got 3 games out. After dealing with these guys I gave up and decided to not put any more games on Xbox at all. I started all over and will take a year off to wait for the Xbox One and pray that they get rid of these guys and come up with a system that stops all the nonsense. This is the reason that MS has changed the process they are not stupid. If the Xbox One has this same sort of system then I will not make games for them. Whenever I made a game I put in in play test and nobody tested it. When they did test my game they just told me what to change in the game so they can enjoy it better. When the game would get to review after months of working on the game (months from when I put the game in play test, not the years it took me to make the game) my first game I was told to pull it off and start over. I asked why do you always tell me to pull the game? I was told the title safe area was not correct?? It was correct I changed it 5 times for them, which means 2 months just to get that done. The guy got mad at me and told me another reason which showed me that he did not ever play the game because he was wrong. I called him on it but he still failed the game. I said ok this is just a jerk but all his friends ganged up on me and failed all my games whenever I would put it in review. I sent many emails to Microsoft complaining about this. Long story short many of the games that are on there should not be, including 2 of my own. When I saw that I was starting to make games to pas review and not for my own enjoyment and that of the people who buy the game I stopped making games. If MS can get rid of that part of the process then I again an do what I enjoy and that is to create games. I was not there to sell as many games as possible but to make games that were good and people enjoyed. After dealing with those guys( the group) I started to make games to pass the process as fast as possible. I received many offers of pass my game and I will pass yours, this I never did and never will!
Thank you MS for changing that Bullshit system it really sucks!!