I have implemented a learning agent and my agent only wins if more than 250 games are done; if it's less than that, the satisfactory agent wins. If I run 10000 games in the tournament, my agents will lose the first 250 but win the rest of the 9750 games. Will this be considered consistently beating other agents? Or does it have to win every single game?

ANONYMOUS wrote: As has been addressed before, your agent should not assume it remains between games. To be considered to consistently outperform another agent, then your agent, when dropped into an identical random situation as the other agent, should have a higher expected win rate. You are welcome to have your agent learn more if it is put through multiple rounds to improve its play, but there is no guarantee given in the specification that it will be given time to learn before being assessed. So in effect, this is asking: Why are you making things harder for yourself by making your agent learn from zero while being assessed? Why are you throwing out all the learning it has already done? It is impossible to win every single game even with expert play, so that is not the requirement. But your agent should be effective in any situation it is put in, and that includes its first game.

"Andrew Gozzard" <an*r*w*g*z*a*d@u*a*e*u*a*> wrote: I believe this issue happens regardless of that assumption. The expected winrate converges to its true value as more runs are done, so in the initial 250 game sample it is difficult to properly distinguish whether your agent would outperform other agents, compared to a 1000 game sample. By the end of the 1000 games, my non-learning agent is consistently 1st, but it's a little less clear with less games. By the 100 mark the agent is often 2nd. Given that I've heard most agents having a <55% winrate in the tournaments, I'm not sure I would place my bet on any of them for the first game. As you say, it is impossible to win every single game (with random agent allies and satisfactory agent spies, for instance, your chances of winning are no longer decided by you). I think some of us are not sure how to evaluate our agents (though this is probably intentional as we are to write a report doing such evaluations?)

ANONYMOUS wrote: The measured win rate converges to the expected win rate as more samples are taken. The expected win rate is the intrinsic true value to which it converges. We will take a reasonable statistical sample. If your agent learns more an its expected win rate improves over the course of multiple games, then that still means that it was initially bad and the empirical win rate will reflect that. I am saying that you should be aiming to maximize your expected win rate in whatever random scenario you are put in. That is what it means to be an effective game agent. The question is not how much you bet but which one would you bet on if you were required to do so. Even if it's a small margin, if your agent has a higher expected win rate (probability that it wins the game) than the benchmark agent, the only logical decision is to bet on the one with the higher win rate. If we imagine a pair of parallel universes where the only difference is that one of the players in universe A is agent A but in universe B is agent B, then if A consistently outperforms B, we would expect it to be more likely for A to win than for B to win. The whole purpose of a game agent is to win the game. Your objective is to maximize the likelihood that your agent wins any game it is put into. Remember that part of the project is a report, part of which asks for you to discuss how you assessed your agent's performance to determine that it was actually performing well. Hope that helps! Cheers, Gozz