[aliasgerovs]

What this adds

This PR adds E.T. Bench (Liu et al., NeurIPS 2024), a benchmark for event-level, time-sensitive video understanding. It has 7,289 samples in 12 tasks over 7K videos (251 hours, 8 domains), grouped into four abilities:

Ability	Tasks	Score
Referring	RAR (action at a given time), ECA (event-caption alignment), RVQ (question about a given moment)	accuracy
Grounding	TVG (sentence grounding), EPM (episodic memory), TAL (action localization), EVS (summarization), VHD (highlight)	F1
Dense captioning	DVC (dense video captioning), SLC (step localization and captioning)	F1 and sentence similarity
Complex understanding	TEM (temporal matching), GVQ (grounded question answering)	recall

There is also ETBench-Subset, the official 470-sample subset that the paper uses for commercial models.

Data: https://huggingface.co/datasets/PolyU-ChenLab/ETBench (CC BY-NC-SA 4.0).

How it works

• The annotations are downloaded the first time, and then the video archive of each source the chosen split needs (about 139 GB for the full benchmark), into LMUData/ETBench.
• Every sample has its own prompt (the official annotations with timestamps written as text), and the model answers in free text.
• The answers are scored with the rules of the official script (evaluation/compute_metrics.py). The result has the 14 numbers of the paper's main table, the five averages (referring accuracy, grounding F1, captioning F1 and similarity, complex recall), and the details for every task and source.
• Like the official script, the caption similarity uses the sentence-transformers model all-MiniLM-L6-v2. Captions are PTB-tokenized with pycocoevalcap when it is installed (it needs Java), and with a simple tokenizer otherwise.
• The official script compares a predicted caption that matches no event with a random string, so its similarity changes a little between runs. Here those strings come from a fixed seed, so the result is reproducible.
• One video in the official archives, summe/cooking.mp4, has broken timestamps: decord cannot seek in it, so loading it fails (and the torchvision fallback in qwen-vl-utils fails too with recent PyAV). It is re-encoded once after extraction with PyAV, frame for frame at the same frame rate, and the original is kept next to it. I checked all 7,002 videos with decord and this is the only one.
• The benchmark also ships its prompts in a second style (etbench_vid_v1.0.json) for models that read timestamps as special tokens, such as the paper's E.T. Chat: timestamps in the prompt are written as <vid> tokens and their values are given separately. The _vid presets use these prompts; the text item then also carries the values (src) and the task. For ECA the two files write the options differently ("13.7 - 16.8 seconds" or "13.7 - 16.8"), and the official scorer compares answers with the option texts when the letter is wrong or missing, so the _vid presets are scored with that file's options, as the official script would.
• Dataset names: ETBench_1fps, ETBench_2fps, ETBench-Subset_1fps, ETBench-Subset_2fps, and ETBench_1fps_vid, ETBench-Subset_1fps_vid.

Check against the official script

The official script is under CC BY-NC-SA, so the scoring is written fresh following its rules. To check it, I generated an answer for every one of the 7,289 samples in all the formats the official parser handles (with noise in the times, wrong options, shuffled captions, and some unreadable answers) and scored them with both. All 20 task/source results are identical: accuracy, F1, recall, mIoU, the number of unreadable answers, and the caption similarity when both use the same seed for the random strings. Unit tests are in tests/test_etbench_metrics.py.

Reference result

Qwen3-VL-8B-Instruct on ETBench_1fps, with the same model settings as #1715 (64 to 14,336 visual tokens, greedy, up to 1,024 new tokens):

RAR	ECA	RVQ	TVG	EPM	TAL	EVS	VHD	DVC F1	DVC Sim	SLC F1	SLC Sim	TEM	GVQ
54.4	63.8	61.0	56.8	9.1	33.2	15.5	65.0	32.0	15.9	25.2	16.8	23.5	6.4

The five averages are 59.7 (referring accuracy), 35.9 (grounding F1), 28.6 / 16.4 (captioning F1 / similarity) and 15.0 (complex recall). All but 2 of the 7,289 answers could be read. For context, the paper's best open model, E.T. Chat, has 44.6 / 37.0 / 33.6 on RAR / ECA / RVQ, 38.6 TVG F1 and 62.5 VHD F1. The newer Qwen3-VL-8B is clearly ahead on referring and sentence grounding, but still weak on episodic memory (EPM) and grounded question answering (GVQ), as in the paper.

Scoring these Qwen3-VL-8B answers with the official compute_metrics.py gives exactly the same numbers for all 20 task/source results, including the 2 unreadable answers.

Check with a model from the paper

E.T. Chat, the model of the E.T. Bench paper, added in #1815 with the official inference settings, run end to end on ETBench_1fps_vid and compared with the paper's Table 1:

	RAR	ECA	RVQ	TVG	EPM	TAL	EVS	VHD	DVC F1	DVC Sim	SLC F1	SLC Sim	TEM	GVQ
Paper, all 7,289 samples	44.6	37.0	33.6	38.6	10.2	30.8	25.4	62.5	38.4	19.7	24.4	14.6	16.5	3.7
VLMEvalKit, all 7,289 samples	43.2	40.0	34.0	39.3	9.1	32.3	23.1	62.0	37.7	18.6	24.7	14.9	16.7	3.8
Paper, 470-sample subset	33.3	31.8	30.8	32.9	8.9	28.1	15.3	60.9	39.8	19.5	23.7	14.8	14.8	1.3
VLMEvalKit, 470-sample subset	33.3	31.8	30.8	36.7	7.1	28.9	24.4	58.9	34.8	18.2	25.8	16.1	18.0	2.1

On the full set the 14 numbers are on average 1.0 point from the paper, and the averages of the five groups are all within 0.7 points. E.T. Chat samples its answers (temperature 0.2, as in the official script), so two runs differ a little; the largest gaps are ECA +3.0 and EVS -2.3 (with the text-style option list ECA would be 37.6). The subset numbers come from the same answers; with only about 40 samples per task, they move more.

Checks

• The repo pre-commit hooks (flake8, isort, yapf and the others) pass on the changed files.
• pytest tests/test_etbench_metrics.py: 16 passed (parsing, every scoring rule, end-to-end scoring, the video repair, both prompt styles).
• pytest tests/test_dataset_alias.py: 29 passed (the new datasets are in the shortcut manifest).

[aliasgerovs]

@ssiq @TianhaoLiang2000 Could you review this when you have a moment? It adds E.T. Bench (12 tasks, plus the official 470-sample subset), and the scoring gives exactly the same numbers as the official compute_metrics.py on the reference run. Thanks!

[ssiq]

Thanks for the additional results. Qwen3-VL-8B is not included in the official E.T. Bench report, so I cannot verify whether the scores reported in this PR align with the official results.

Could you provide results for a model that is already included in the official report, evaluated using the same benchmark protocol? That would give us a verifiable reference point and allow us to confirm whether this integration reproduces the official scores.

[aliasgerovs]

@ssiq Thanks, that makes sense. I added E.T. Chat, the model from the E.T. Bench paper, in #1815, using its official code, weights and inference settings, and ran it end to end with this PR's data and scoring:

	RAR	ECA	RVQ	TVG	EPM	TAL	EVS	VHD	DVC F1	DVC Sim	SLC F1	SLC Sim	TEM	GVQ
Paper, all 7,289 samples	44.6	37.0	33.6	38.6	10.2	30.8	25.4	62.5	38.4	19.7	24.4	14.6	16.5	3.7
VLMEvalKit, all 7,289 samples	43.2	40.0	34.0	39.3	9.1	32.3	23.1	62.0	37.7	18.6	24.7	14.9	16.7	3.8
Paper, 470-sample subset	33.3	31.8	30.8	32.9	8.9	28.1	15.3	60.9	39.8	19.5	23.7	14.8	14.8	1.3
VLMEvalKit, 470-sample subset	33.3	31.8	30.8	36.7	7.1	28.9	24.4	58.9	34.8	18.2	25.8	16.1	18.0	2.1

On the full set the 14 numbers are on average 1.0 point from the paper, and the averages of the five groups are all within 0.7 points. E.T. Chat samples its answers (temperature 0.2, as in the official script), so two runs differ a little; the largest gaps are ECA +3.0 and EVS -2.3. The subset numbers come from the same answers, but with only about 40 samples per task they move more.

E.T. Chat reads timestamps as special tokens, so it needs the benchmark's second official prompt style (etbench_vid_v1.0.json, where the times in the prompt are <vid> tokens). I added it to this PR as ETBench_1fps_vid and ETBench-Subset_1fps_vid; the existing presets are unchanged. The PR description is updated.