Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets.

Thamme Gowda Tom Kocmi Ekaterina Artemova Eleftherios Avramidis Rachel Bawden Ondrej Bojar Konstantin Dranch Anton Dvorkovich Sergey Dukanov Mark Fishel Markus Freitag Thamme Gowda Roman Grundkiewicz Barry Haddow Marzena Karpinska Philipp Koehn Howard Lakougna Jessica Lundin Christof Monz Kenton Murray Masaaki Nagata Stefano Perrella Lorenzo Proietti 0002 Martin Popel Maja Popovic Parker Riley Mariya Shmatova Steinthór Steingrímsson Lisa Yankovskaya Vilém Zouhar Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets. WMT 2025 355-413 conf/wmt/2025 https://doi.org/10.18653/v1/2025.wmt-1.22 https://aclanthology.org/2025.wmt-1.22/ db/conf/wmt/wmt2025.html#KocmiAABBDDDFFGGHKKLLMM25 streams/conf/wmt Marco Gaido Roman Grundkiewicz Thamme Gowda Matteo Negri Findings of the WMT 2025 Shared Task on Model Compression: Early Insights on Compressing LLMs for Machine Translation. WMT 2025 484-494 conf/wmt/2025 https://doi.org/10.18653/v1/2025.wmt-1.25 https://aclanthology.org/2025.wmt-1.25/ db/conf/wmt/wmt2025.html#GaidoGGN25 streams/conf/wmt

Tom Kocmi Eleftherios Avramidis Rachel Bawden Ondrej Bojar Konstantin Dranch Anton Dvorkovich Sergey Dukanov Natalia Fedorova Mark Fishel Markus Freitag Thamme Gowda Roman Grundkiewicz Barry Haddow Marzena Karpinska Philipp Koehn Howard Lakougna Jessica Lundin Kenton Murray Masaaki Nagata Stefano Perrella Lorenzo Proietti 0002 Martin Popel Maja Popovic Parker Riley Mariya Shmatova Steinþór Steingrímsson Lisa Yankovskaya Vilém Zouhar Preliminary Ranking of WMT25 General Machine Translation Systems. 2025 August abs/2508.14909 CoRR https://doi.org/10.48550/arXiv.2508.14909 db/journals/corr/corr2508.html#abs-2508-14909 streams/journals/corr

Hyundong Cho Thamme Gowda Yuyang Huang Zixun Lu Tianli Tong Jonathan May BotEval: Facilitating Interactive Human Evaluation. 107-116 2024 ACL (3) https://doi.org/10.18653/v1/2024.acl-demos.11 conf/acl/2024-3 db/conf/acl/acl2024-3.html#ChoGHLTM24 Thamme Gowda Roman Grundkiewicz Elijah Rippeth Matt Post Marcin Junczys-Dowmunt PyMarian: Fast Neural Machine Translation and Evaluation in Python. 328-335 2024 EMNLP (System Demonstrations) https://doi.org/10.18653/v1/2024.emnlp-demo.34 conf/emnlp/2024d db/conf/emnlp/emnlp2024d.html#GowdaGRPJ24 Tom Kocmi Eleftherios Avramidis Rachel Bawden Ondrej Bojar Anton Dvorkovich Christian Federmann Mark Fishel Markus Freitag Thamme Gowda Roman Grundkiewicz Barry Haddow Marzena Karpinska Philipp Koehn Benjamin Marie Christof Monz Kenton Murray Masaaki Nagata Martin Popel Maja Popovic Mariya Shmatova Steinthór Steingrímsson Vilém Zouhar Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet. 1-46 2024 WMT https://doi.org/10.18653/v1/2024.wmt-1.1 https://aclanthology.org/2024.wmt-1.1 conf/wmt/2024 db/conf/wmt/wmt2024.html#KocmiABBDFFFGGH24

Hyundong Cho Thamme Gowda Yuyang Huang Zixun Lu Tianli Tong Jonathan May BotEval: Facilitating Interactive Human Evaluation. 2024 abs/2407.17770 CoRR https://doi.org/10.48550/arXiv.2407.17770 db/journals/corr/corr2407.html#abs-2407-17770 streams/journals/corr

Tom Kocmi Eleftherios Avramidis Rachel Bawden Ondrej Bojar Anton Dvorkovich Christian Federmann Mark Fishel Markus Freitag Thamme Gowda Roman Grundkiewicz Barry Haddow Marzena Karpinska Philipp Koehn Benjamin Marie Kenton Murray Masaaki Nagata Martin Popel Maja Popovic Mariya Shmatova Steinþór Steingrímsson Vilém Zouhar Preliminary WMT24 Ranking of General MT Systems and LLMs. 2024 abs/2407.19884 CoRR https://doi.org/10.48550/arXiv.2407.19884 db/journals/corr/corr2407.html#abs-2407-19884 streams/journals/corr

Thamme Gowda Roman Grundkiewicz Elijah Rippeth Matt Post Marcin Junczys-Dowmunt PyMarian: Fast Neural Machine Translation and Evaluation in Python. 2024 abs/2408.11853 CoRR https://doi.org/10.48550/arXiv.2408.11853 db/journals/corr/corr2408.html#abs-2408-11853 streams/journals/corr

Tom Kocmi Eleftherios Avramidis Rachel Bawden Ondrej Bojar Anton Dvorkovich Christian Federmann Mark Fishel Markus Freitag Thamme Gowda Roman Grundkiewicz Barry Haddow Philipp Koehn Benjamin Marie Christof Monz Makoto Morishita Kenton Murray Makoto Nagata Toshiaki Nakazawa Martin Popel Maja Popovic Mariya Shmatova Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet. 1-42 2023 WMT https://doi.org/10.18653/v1/2023.wmt-1.1 https://aclanthology.org/2023.wmt-1.1 conf/wmt/2023 db/conf/wmt/wmt2023.html#KocmiABBDFFFGGH23 Steve Sloto Brian Thompson 0001 Huda Khayrallah Tobias Domhan Thamme Gowda Philipp Koehn Findings of the WMT 2023 Shared Task on Parallel Data Curation. 95-102 2023 WMT https://doi.org/10.18653/v1/2023.wmt-1.5 https://aclanthology.org/2023.wmt-1.5 conf/wmt/2023 db/conf/wmt/wmt2023.html#Sloto0KDGK23 Thamme Gowda Tom Kocmi Marcin Junczys-Dowmunt Cometoid: Distilling Strong Reference-based Machine Translation Metrics into Even Stronger Quality Estimation Metrics. 751-755 2023 WMT https://doi.org/10.18653/v1/2023.wmt-1.62 https://aclanthology.org/2023.wmt-1.62 conf/wmt/2023 db/conf/wmt/wmt2023.html#GowdaKJ23

Matt Post Thamme Gowda Roman Grundkiewicz Huda Khayrallah Rohit Jain Marcin Junczys-Dowmunt SOTASTREAM: A Streaming Approach to Machine Translation Training. 2023 abs/2308.07489 CoRR https://doi.org/10.48550/arXiv.2308.07489 db/journals/corr/corr2308.html#abs-2308-07489

Tom Kocmi Rachel Bawden Ondrej Bojar Anton Dvorkovich Christian Federmann Mark Fishel Thamme Gowda Yvette Graham Roman Grundkiewicz Barry Haddow Rebecca Knowles Philipp Koehn Christof Monz Makoto Morishita Masaaki Nagata Toshiaki Nakazawa Michal Novák 0001 Martin Popel Maja Popovic Findings of the 2022 Conference on Machine Translation (WMT22). 1-45 2022 WMT https://doi.org/10.18653/v1/2022.wmt-1.1 https://aclanthology.org/2022.wmt-1.1 conf/wmt/2022 db/conf/wmt/wmt2022.html#KocmiBBDFFGGGHKKMMNNNPP22

Thamme Gowda Mozhdeh Gheini Jonathan May Checks and Strategies for Enabling Code-Switched Machine Translation. 2022 abs/2210.05096 CoRR https://doi.org/10.48550/arXiv.2210.05096 db/journals/corr/corr2210.html#abs-2210-05096

Thamme Gowda Zhao Zhang Chris Mattmann Jonathan May Many-to-English Machine Translation Tools, Data, and Pretrained Models. 306-316 2021 ACL (demo) https://doi.org/10.18653/v1/2021.acl-demo.37 https://aclanthology.org/2021.acl-demo.37 conf/acl/2021d db/conf/acl/acl2021-d.html#GowdaZMM21 Thamme Gowda Weiqiu You Constantine Lignos Jonathan May Macro-Average: Rare Types Are Important Too. 1138-1157 2021 NAACL-HLT https://doi.org/10.18653/v1/2021.naacl-main.90 https://aclanthology.org/2021.naacl-main.90/ conf/naacl/2021 db/conf/naacl/naacl2021.html#GowdaYLM21

Thamme Gowda Zhao Zhang Chris A. Mattmann Jonathan May Many-to-English Machine Translation Tools, Data, and Pretrained Models. 2021 abs/2104.00290 CoRR https://arxiv.org/abs/2104.00290 db/journals/corr/corr2104.html#abs-2104-00290

Thamme Gowda Weiqiu You Constantine Lignos Jonathan May Macro-Average: Rare Types Are Important Too. 2021 abs/2104.05700 CoRR https://arxiv.org/abs/2104.05700 db/journals/corr/corr2104.html#abs-2104-05700

Thamme Gowda Jonathan May Finding the Optimal Vocabulary Size for Neural Machine Translation. 3955-3964 2020 EMNLP (Findings) https://doi.org/10.18653/v1/2020.findings-emnlp.352 https://aclanthology.org/2020.findings-emnlp.352/ conf/emnlp/2020f db/conf/emnlp/emnlp2020f.html#GowdaM20 Ninareh Mehrabi Thamme Gowda Fred Morstatter Nanyun Peng 0001 Aram Galstyan Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition. 231-232 2020 HT https://doi.org/10.1145/3372923.3404804 conf/ht/2020 db/conf/ht/ht2020.html#MehrabiGMPG20

Thamme Gowda Jonathan May Neural Machine Translation with Imbalanced Classes. 2020 abs/2004.02334 CoRR https://arxiv.org/abs/2004.02334 db/journals/corr/corr2004.html#abs-2004-02334

Elizabeth Boschee Joel Barry Jayadev Billa Marjorie Freedman Thamme Gowda Constantine Lignos Chester Palen-Michel Michael Pust Banriskhem K. Khonglah Srikanth R. Madikeri Jonathan May Scott Miller SARAL: A Low-Resource Cross-Lingual Domain-Focused Information Retrieval System for Effective Rapid Document Triage. 19-24 2019 ACL (3) https://doi.org/10.18653/v1/p19-3004 https://aclanthology.org/P19-3004/ conf/acl/2019-3 db/conf/acl/acl2019-3.html#BoscheeBBFGLPPK19 Xiaoman Pan Thamme Gowda Heng Ji 0001 Jonathan May Scott Miller Cross-lingual Joint Entity and Word Embedding to Improve Entity Linking and Parallel Sentence Mining. 56-66 2019 DeepLo@EMNLP-IJCNLP https://doi.org/10.18653/v1/D19-6107 conf/acl-deeplo/2019 db/conf/acl-deeplo/acl-deeplo2019.html#PanGJMM19

Ninareh Mehrabi Thamme Gowda Fred Morstatter Nanyun Peng 0001 Aram Galstyan Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition. 2019 abs/1910.10872 CoRR http://arxiv.org/abs/1910.10872 db/journals/corr/corr1910.html#abs-1910-10872

Kiri L. Wagstaff Raymond Francis Thamme Gowda You Lu Ellen Riloff Karanjeet Singh 0002 Nina L. Lanza Mars Target Encyclopedia: Rock and Soil Composition Extracted From the Literature. 2018 AAAI https://doi.org/10.1609/aaai.v32i1.11412 conf/aaai/2018 db/conf/aaai/aaai2018.html#WagstaffFGLRSL18 7861-7866 Kiri L. Wagstaff You Lu Alice Stanboli Kevin Grimes Thamme Gowda Jordan Padams Deep Mars: CNN Classification of Mars Imagery for the PDS Imaging Atlas. 2018 AAAI https://doi.org/10.1609/aaai.v32i1.11404 conf/aaai/2018 db/conf/aaai/aaai2018.html#WagstaffLSGGP18 7867-7872 Kyle Hundman Thamme Gowda Mayank Kejriwal Benedikt Boecking Always Lurking: Understanding and Mitigating Bias in Online Human Trafficking Detection. 137-143 2018 AIES https://doi.org/10.1145/3278721.3278782 conf/aies/2018 db/conf/aies/aies2018.html#HundmanGKB18 Thamme Gowda Kyle Hundman Chris A. Mattmann An Approach for Automatic and Large Scale Image Forensics. 16-20 2017 MFSec@ICMR https://doi.org/10.1145/3078897.3080536 conf/mir/2017mfsec db/conf/mir/mfsec2017.html#GowdaHM17

Kyle Hundman Thamme Gowda Mayank Kejriwal Benedikt Boecking Always Lurking: Understanding and Mitigating Bias in Online Human Trafficking Detection. 2017 abs/1712.00846 CoRR http://arxiv.org/abs/1712.00846 db/journals/corr/corr1712.html#abs-1712-00846

Thamme Gowda Chris A. Mattmann Clustering Web Pages Based on Structure and Style Similarity (Application Paper). 175-180 2016 IRI https://doi.org/10.1109/IRI.2016.30 https://doi.ieeecomputersociety.org/10.1109/IRI.2016.30 conf/iri/2016 db/conf/iri/iri2016.html#GowdaM16 Ekaterina Artemova Eleftherios Avramidis Joel Barry Rachel Bawden Jayadev Billa Benedikt Boecking Ondrej Bojar Elizabeth Boschee Hyundong Cho Tobias Domhan Konstantin Dranch Sergey Dukanov Anton Dvorkovich Christian Federmann Natalia Fedorova Mark Fishel Raymond Francis Marjorie Freedman Markus Freitag Marco Gaido Aram Galstyan Mozhdeh Gheini Yvette Graham Kevin Grimes Roman Grundkiewicz Barry Haddow Yuyang Huang Kyle Hundman Rohit Jain Heng Ji 0001 Marcin Junczys-Dowmunt Marzena Karpinska Mayank Kejriwal Huda Khayrallah Banriskhem K. Khonglah Rebecca Knowles Tom Kocmi Philipp Koehn Howard Lakougna Nina L. Lanza Constantine Lignos You Lu Zixun Lu Jessica Lundin Srikanth R. Madikeri Benjamin Marie Chris MattmannChris A. Mattmann Jonathan May Ninareh Mehrabi Scott Miller Christof Monz Makoto Morishita Fred Morstatter Kenton Murray Makoto Nagata Masaaki Nagata Toshiaki Nakazawa Matteo Negri Michal Novák 0001 Jordan Padams Chester Palen-Michel Xiaoman Pan Nanyun Peng 0001 Stefano Perrella Martin Popel Maja Popovic Matt Post Lorenzo Proietti 0002 Michael Pust Parker Riley Ellen Riloff Elijah Rippeth Mariya Shmatova Karanjeet Singh 0002 Steve Sloto Alice Stanboli Steinþór SteingrímssonSteinthór Steingrímsson Brian Thompson 0001 Tianli Tong Kiri WagstaffKiri L. Wagstaff Lisa Yankovskaya Weiqiu You Zhao Zhang Vilém Zouhar