"Andrew Gozzard" <an*r*w*g*z*a*d@u*a*e*u*a*> wrote: I believe this issue happens regardless of that assumption. The expected winrate converges to its true value as more runs are done, so in the initial 250 game sample it is difficult to properly distinguish whether your agent would outperform other agents, compared to a 1000 game sample. By the end of the 1000 games, my non-learning agent is consistently 1st, but it's a little less clear with less games. By the 100 mark the agent is often 2nd. Given that I've heard most agents having a <55% winrate in the tournaments, I'm not sure I would place my bet on any of them for the first game. As you say, it is impossible to win every single game (with random agent allies and satisfactory agent spies, for instance, your chances of winning are no longer decided by you). I think some of us are not sure how to evaluate our agents (though this is probably intentional as we are to write a report doing such evaluations?)