Skip to content

AIME'24 is broken w/ Openai-API backend #3497

@Lyken17

Description

@Lyken17

Reproduce

# launch openai-api server
python -m sglang.launch_server \
    --model Qwen/qwen3-8B --port 8000 \
    --dtype auto --data-parallel-size 8 --tensor-parallel-size 1

lm_eval --model local-completions \
    --tasks aime25 \
    --model_args model=Qwen/qwen3-8B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=1,max_retries=3 

it then returns

2026-01-16:00:13:19 WARNING  [models.api_models:479] API request failed with error message: {"object":"error","message":"Prompt cannot be empty","type":"BadRequestError","param":null,"code":400}. Retrying...
2026-01-16:00:13:20 WARNING  [models.api_models:479] API request failed with error message: {"object":"error","message":"Prompt cannot be empty","type":"BadRequestError","param":null,"code":400}. Retrying...
2026-01-16:00:13:21 WARNING  [models.api_models:479] API request failed with error message: {"object":"error","message":"Prompt cannot be empty","type":"BadRequestError","param":null,"code":400}. Retrying...```

while for other tasks like gsm8k hellaswag the above scripts work as expected. Seems there is something wrong with the AIME config.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions