TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game

Starcraft II (SCII) is widely considered as the most challenging Real Time Strategy (RTS) game as of now, due to large observation space, huge (continuous and infinite) action space, partial observation, multi-player simultaneous game model, long time horizon decision, etc. To push the frontier of AI's capability, Deepmind and Blizzard jointly present the StarCraft II Learning Environment (SC2LE) --- a testbench for designing complex decision making systems. While SC2LE provides a few mini games such as \textit{MoveToBeacon}, \textit{CollectMineralShards}, and \textit{DefeatRoaches} where some AI agents achieve the professional player's level, it is still far away from achieving the professional level in a \emph{full} game. To initialize the research and investigation in the full game, we develop two AI agents --- the AI agent TStarBot1 is based on deep reinforcement learning over flat action structure, and the AI agent TStarBot2 is based on rule controller over hierarchical action structure. Both TStarBot1 and TStarBot2 are able to defeat the builtin AI agents from level 1 to level 10 in a full game (1v1 \textit{Zerg}-vs-\textit{Zerg} game on the AbyssalReef map), noting that level 8, level 9, and level 10 are cheating agents with full vision on the whole map, with resource harvest boosting, and with both, respectively \footnote{According to some informal discussions from the StarCraft II forum, level 10 builtin AI is estimated to be Platinum to Diamond~\cite{scii-forum}, which are equivalent to top 50\% - 30\% human players in the ranking system of Battle.net Leagues~\cite{liquid}. }.