The calculation method of TP Bench GPA:<\/strong><\/p>\n\n\n\nFirstly, to each problem, we assign a numerical weight \\(w\\) from 1 to 5 based on its difficulty level. Next, we translate the color coding used above to assign a letter and a corresponding numerical grade \\(g\\) as follows:<\/p>\n\n\n\n
(a) Green: Letter grade A which is assigned a numerical value of \\(g=4\\).
(b) Blue: Letter grade B and is assigned a value of \\(g=3\\).
(c) Yellow: Letter grade C and \\(g=2\\).
(d) Red: Letter grade F which is set to \\(g=0\\).<\/p>\n\n\n\n
Finally, we apply the following formula to determine GPA (out of 4) \\[ {\\rm GPA}=\\frac{\\sum_{i=1}^{N}w_{i}\\times g_{i}}{\\sum_{i=1}^{N}w_{i}} \\] where \\(N\\) represents the total number of attempts across all problems.<\/p>\n\n\n\n
L<\/strong>ooking ahead, we plan to improve and introduce additional evaluation metrics, one of which could incorporate confidence and precision to better capture the variability of model responses across multiple attempts on the same problem. This metric would impose penalties on models that exhibit high inconsistency in their responses, ensuring a more nuanced assessment of a model’s reliability and robustness. More intriguingly, as we enable models to learn and improve incrementally from their previous attempts, rather than treating each attempt as independent, we will need to develop more sophisticated metrics to evaluate their performance accurately.<\/p>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Public Problems and Model Solutions Download our current public problems, their expert solutions, and model results in PDF format: Download the chain-of-thought report for DeepSeek-R1: The model performance on public problems is as follows: The meaning of colors in the plot: The calculation method of TP Bench GPA: Firstly, to each problem, we assign a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":1,"comment_status":"closed","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/tpbench.org\/index.php?rest_route=\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tpbench.org\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/tpbench.org\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/tpbench.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tpbench.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2"}],"version-history":[{"count":47,"href":"https:\/\/tpbench.org\/index.php?rest_route=\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":503,"href":"https:\/\/tpbench.org\/index.php?rest_route=\/wp\/v2\/pages\/2\/revisions\/503"}],"wp:attachment":[{"href":"https:\/\/tpbench.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/tpbench.org\/wp-content\/uploads\/2026\/05\/website_figure-787x1024.png\")
<\/figure>\n\n\n\n