-
Notifications
You must be signed in to change notification settings - Fork 3k
Open
Description
Reproduce
# launch openai-api server
python -m sglang.launch_server \
--model Qwen/qwen3-8B --port 8000 \
--dtype auto --data-parallel-size 8 --tensor-parallel-size 1
lm_eval --model local-completions \
--tasks aime25 \
--model_args model=Qwen/qwen3-8B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=1,max_retries=3
it then returns
2026-01-16:00:13:19 WARNING [models.api_models:479] API request failed with error message: {"object":"error","message":"Prompt cannot be empty","type":"BadRequestError","param":null,"code":400}. Retrying...
2026-01-16:00:13:20 WARNING [models.api_models:479] API request failed with error message: {"object":"error","message":"Prompt cannot be empty","type":"BadRequestError","param":null,"code":400}. Retrying...
2026-01-16:00:13:21 WARNING [models.api_models:479] API request failed with error message: {"object":"error","message":"Prompt cannot be empty","type":"BadRequestError","param":null,"code":400}. Retrying...```
while for other tasks like gsm8k hellaswag the above scripts work as expected. Seems there is something wrong with the AIME config.
Metadata
Metadata
Assignees
Labels
No labels