ANONYMOUS wrote:
> I believe this issue happens regardless of that assumption. The expected winrate converges to its true value as more runs are done, so in the initial 250 game sample it is difficult to properly distinguish whether your agent would outperform other agents, compared to a 1000 game sample. By the end of the 1000 games, my non-learning agent is consistently 1st, but it's a little less clear with less games. By the 100 mark the agent is often 2nd.
The measured win rate converges to the expected win rate as more samples are taken. The expected win rate is the intrinsic true value to which it converges. We will take a reasonable statistical sample. If your agent learns more an its expected win rate improves over the course of multiple games, then that still means that it was initially bad and the empirical win rate will reflect that. I am saying that you should be aiming to maximize your expected win rate in whatever random scenario you are put in. That is what it means to be an effective game agent.
> Given that I've heard most agents having a <55% winrate in the tournaments, I'm not sure I would place my bet on any of them for the first game. As you say, it is impossible to win every single game (with random agent allies and satisfactory agent spies, for instance, your chances of winning are no longer decided by you). I think some of us are not sure how to evaluate our agents (though this is probably intentional as we are to write a report doing such evaluations?)
The question is not how much you bet but which one would you bet on if you were required to do so. Even if it's a small margin, if your agent has a higher expected win rate (probability that it wins the game) than the benchmark agent, the only logical decision is to bet on the one with the higher win rate.
If we imagine a pair of parallel universes where the only difference is that one of the players in universe A is agent A but in universe B is agent B, then if A consistently outperforms B, we would expect it to be more likely for A to win than for B to win.
The whole purpose of a game agent is to win the game. Your objective is to maximize the likelihood that your agent wins any game it is put into.
Remember that part of the project is a report, part of which asks for you to discuss how you assessed your agent's performance to determine that it was actually performing well.
Hope that helps!
Cheers,
Gozz
