Resultado da Busca
Sacrebleu or sacre bleu is a French profanity used as a cry of surprise, irritation or displeasure. It is a minced oath form of the profane sacré Dieu (holy God), which, by some religions, is considered profane, due to one of the Ten Commandments in the Bible, which reads " Thou shalt not take the name of the Lord thy God in vain ."
SacreBLEU provides hassle-free computation of shareable, comparable, and reproducible BLEU scores. Inspired by Rico Sennrich's `multi-bleu-detok.perl`, it produces the official WMT scores but works...
- Overview
- Motivation
- Features
- v2.0.0
- Installation
- Command-line Usage
- Basics
- Translationese Support
- Languages & Preprocessing
- Multi-reference Evaluation
- GeneratedCaptionsTabForHeroSec
SacreBLEU (Post, 2018) provides hassle-free computation of shareable, comparable, and reproducible BLEU scores. Inspired by Rico Sennrich's multi-bleu-detok.perl, it produces the official WMT scores but works with plain text. It also knows all the standard test sets and handles downloading, processing, and tokenization for you.
The official version is hosted at https://github.com/mjpost/sacrebleu.
Comparing BLEU scores is harder than it should be. Every decoder has its own implementation, often borrowed from Moses, but maybe with subtle changes. Moses itself has a number of implementations as standalone scripts, with little indication of how they differ (note: they mostly don't, but multi-bleu.pl expects tokenized input). Different flags passed to each of these scripts can produce wide swings in the final score. All of these may handle tokenization in different ways. On top of this, downloading and managing test sets is a moderate annoyance.
Sacre bleu! What a mess.
•It automatically downloads common WMT test sets and processes them to plain text
•It produces a short version string that facilitates cross-paper comparisons
•It properly computes scores on detokenized outputs, using WMT (Conference on Machine Translation) standard tokenization
•It produces the same values as the official script (mteval-v13a.pl) used by WMT
•It outputs the BLEU score without the comma, so you don't have to remove it with sed (Looking at you, multi-bleu.perl)
•It supports different tokenizers for BLEU including support for Japanese and Chinese
As of v2.0.0, the default output format is changed to json for less painful parsing experience. This means that software that parse the output of sacreBLEU should be modified to either (i) parse the JSON using for example the jq utility or (ii) pass -f text to sacreBLEU to preserve the old textual output. The latter change can also be made persistently by exporting SACREBLEU_FORMAT=text in relevant shell configuration files.
Here's an example of parsing the score key of the JSON output using jq:
Install the official Python module from PyPI (Python>=3.6 only):
In order to install Japanese tokenizer support through mecab-python3, you need to run the following command instead, to perform a full installation with dependencies:
You can get a list of available test sets with sacrebleu --list. Please see DATASETS.md for an up-to-date list of supported datasets. You can also list available test sets for a given language pair with sacrebleu --list -l en-fr.
Downloading test sets
Downloading is triggered when you request a test set. If the dataset is not available, it is downloaded and unpacked. E.g., you can use the following commands to download the source, pass it through your translation system in translate.sh, and then score it: Some test sets also have the outputs of systems that were submitted to the task. For example, the wmt/systems test set. This provides a convenient way to score: $ sacrebleu -i output.detok.txt -t wmt17 -l en-de If you want to keep the old behavior, you can pass -f text or export SACREBLEU_FORMAT=text:
Scoring
(All examples below assume old-style text output for a compact representation that save space) Let's say that you just translated the en-de test set of WMT17 with your fancy MT system and the detokenized translations are in a file called output.detok.txt: You can obtain a short version of the signature with --short/-sh: If you only want the score to be printed, you can use the --score-only/-b flag: The precision of the scores can be configured via the --width/-w flag:
Using your own reference file
SacreBLEU knows about common test sets (as detailed in the --list example above), but you can also use it to score system outputs with arbitrary references. In this case, do not forget to provide detokenized reference and hypotheses files:
If you are interested in the translationese effect, you can evaluate BLEU on a subset of sentences with a given original language (identified based on the origlang tag in the raw SGM files). E.g., to evaluate only against originally German sentences translated to English use:
and to evaluate against the complement (in this case origlang en, fr, cs, ru, de) use:
BLEU
•You can compute case-insensitive BLEU by passing --lowercase to sacreBLEU •The default tokenizer for BLEU is 13a which mimics the mteval-v13a script from Moses. •Other tokenizers are: •none which will not apply any kind of tokenization at all •char for language-agnostic character-level tokenization •intl applies international tokenization and mimics the mteval-v14 script from Moses •zh separates out Chinese characters and tokenizes the non-Chinese parts using 13a tokenizer •ja-mecab tokenizes Japanese inputs using the MeCab morphological analyzer •ko-mecab tokenizes Korean inputs using the MeCab-ko morphological analyzer •flores101 and flores200 uses the SentencePiece model built from the Flores-101 and Flores-200 dataset, respectively. Note: the canonical .spm file will be automatically fetched if not found locally. •You can switch tokenizers using the --tokenize flag of sacreBLEU. Alternatively, if you provide language-pair strings using --language-pair/-l, zh, ja-mecab and ko-mecab tokenizers will be used if the target language is zh or ja or ko, respectively. •Note that there's no automatic language detection from the hypotheses so you need to make sure that you are correctly selecting the tokenizer for Japanese, Korean and Chinese. Default 13a tokenizer will produce poor results for Japanese: Let's use the ja-mecab tokenizer: If you provide the language-pair, sacreBLEU will use ja-mecab automatically:
chrF / chrF++
chrF applies minimum to none pre-processing as it deals with character n-grams: •If you pass --chrf-whitespace, whitespace characters will be preserved when computing character n-grams. •If you pass --chrf-lowercase, sacreBLEU will compute case-insensitive chrF. •If you enable non-zero --chrf-word-order (pass 2 for chrF++), a very simple punctuation tokenization will be internally applied.
TER
Translation Error Rate (TER) has its own special tokenizer that you can configure through the command line. The defaults provided are compatible with the upstream TER implementation (TERCOM) but you can nevertheless modify the behavior through the command-line: •TER is by default case-insensitive. Pass --ter-case-sensitive to enable case-sensitivity. •Pass --ter-normalize to apply a general Western tokenization •Pass --ter-asian-support to enable the tokenization of Asian characters. If provided with --ter-normalize, both will be applied. •Pass --ter-no-punct to strip punctuation.
All three metrics support the use of multiple references during evaluation. Let's first pass all references as positional arguments:
Alternatively (less recommended), we can concatenate references using tabs as delimiters as well. Don't forget to pass --num-refs/-nr in this case!
sacreBLEU is a reference implementation of BLEU that auto-downloads test sets and reports a version string for cross-lab comparisons. It supports different tokenizers, metrics, and statistical tests for machine translation evaluation.
12 de abr. de 2024 · sacreBLEU is a hassle-free and reproducible tool for computing BLEU scores, a common metric for machine translation evaluation. It supports various test sets, tokenizers, and output formats, and provides statistical significance tests.
6 de mai. de 2017 · Resposta destacada. Francês (França) It's an old expression that could nowadays be translated to "Oh mon Dieu !" or "Ça alors !" ("Oh my God !" "Dang it" "Damn it"...). It comes from "Sacre de Dieu" (which could be translated to "Blessing of God"), but "Dieu" became "bleu" to avoid blasphemy. Veja uma tradução.
- Kobayashi Bldg 4F, 2-6-14 Ebisu-minami, Shibuya-ku Tokyo 150-0022, Japan, 1500022, JP
- CEO of Lang-8, Inc.
tradução sacrebleu em Português, dicionário Francês - Português, consulte também 'sacrilège, sac, serviable, sacrifice', definição, exemplos, definição.
20 de nov. de 2018 · Sacré bleu is a dated, stereotypical French exclamation that means \"Holy blue!\" or \"My God!\" It is often used in English to show how French something is, or to mock French culture. Learn its origin, usage, and examples.
