Online Ranking

We build Online GRF Ranking to better evaluate football AI by incorporating contributions from the community. We invite participants to register on our online evaluation platform JIDI and submit their GRF agents on our Daily Evaluation section or participate the Competition Evaluation.

JIDI Intro 

JIDI gets its name from a Chinese word 及第 (jí dì) which refers to Passing an imperial examination in ancient China. JIDI aims to address the lack of public leaderboards in the field of decision-making AI by providing online evaluation and ranking services for a wide range of reinforcement learning environments. Currently, JIDI supports nearly 100 environments, encompassing both single-agent games like classic-control games in the gym framework, as well as multi-agent games such as Texas Hold’em and Google Research Football. The online ranking system allows agents to compete against unseen opponents, which is crucial for studying generalization ability, an important aspect of algorithm emphasized by many works.

The mechanism behind JIDI is illustrated below. When an user submits his customized decision-making agent to the agent storage, the back-end evaluation process executes parallel evaluation tasks distributedly on computing nodes. The evaluation results are updated to the user’s score and the online ranking is updated accordingly. Users can view replays of their matches.

JIDI offers two types of evaluation: Daily Evaluation and Competition Evaluation.

Daily Evaluation 

In Daily Evaluation (Link), users can submit their agents at any time, triggering the background evaluation processes. This allows for continuous assessment and ranking of agents based on their performance. JIDI provide Daily Evaluation on three GRF full-game scenarios:

Competition Evaluation 

Competition Evaluation (Link) usually associates with academic conferences or university coursework. The background evaluation processes are initiated once the deadline is reached. JIDI has previously held a GRF online competition on two tracks:

and is holding the same competition this year:

Feel free to participant and hope you have fun ! (competition description link)