Nightly-A3

Nightly-A3 #129

Sign in to view logs

Triggered via schedule February 3, 2026 16:37

wangxiyuan

main

Status Failure

Total duration 6h 20m 52s

Artifacts 11

schedule_nightly_test_a3.yaml

on: schedule

Matrix: multi-node

Matrix: test ops

Matrix: single-node

Annotations

42 errors

multi-node (multi-node-deepseek-pd, DeepSeek-V3.yaml, 2) / DeepSeek-V3.yaml

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

multi-node (multi-node-deepseek-pd, DeepSeek-V3.yaml, 2) / DeepSeek-V3.yaml

Process completed with exit code 1.

multi-node (multi-node-deepseek-pd, DeepSeek-V3.yaml, 2) / DeepSeek-V3.yaml

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

multi-node (multi-node-qwenw8a8-2node-eplb, Qwen3-235B-W8A8-EPLB.yaml, 2) / Qwen3-235B-W8A8-EPLB.yaml

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

multi-node (multi-node-qwenw8a8-2node-eplb, Qwen3-235B-W8A8-EPLB.yaml, 2) / Qwen3-235B-W8A8-EPLB.yaml

Process completed with exit code 1.

multi-node (multi-node-qwenw8a8-2node-eplb, Qwen3-235B-W8A8-EPLB.yaml, 2) / Qwen3-235B-W8A8-EPLB.yaml

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

multi-node (multi-node-dpsk3.2-2node, DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml, 2) / DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

multi-node (multi-node-dpsk3.2-2node, DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml, 2) / DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml

Process completed with exit code 1.

multi-node (multi-node-dpsk3.2-2node, DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml, 2) / DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

multi-node (multi-node-deepseek-r1-w8a8-longseq, DeepSeek-R1-W8A8-longseq.yaml, 2) / DeepSeek-R1-W8A8-longseq.yaml

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

multi-node (multi-node-deepseek-r1-w8a8-longseq, DeepSeek-R1-W8A8-longseq.yaml, 2) / DeepSeek-R1-W8A8-longseq.yaml

Process completed with exit code 1.

multi-node (multi-node-deepseek-r1-w8a8-longseq, DeepSeek-R1-W8A8-longseq.yaml, 2) / DeepSeek-R1-W8A8-longseq.yaml

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

multi-node (multi-node-kimi-k2-instruct-w8a8, Kimi-K2-Instruct-W8A8.yaml, 2) / Kimi-K2-Instruct-W8A8.yaml

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

multi-node (multi-node-kimi-k2-instruct-w8a8, Kimi-K2-Instruct-W8A8.yaml, 2) / Kimi-K2-Instruct-W8A8.yaml

Process completed with exit code 1.

multi-node (multi-node-kimi-k2-instruct-w8a8, Kimi-K2-Instruct-W8A8.yaml, 2) / Kimi-K2-Instruct-W8A8.yaml

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

multi-node (multi-node-deepseek-v3.1, DeepSeek-V3.1-BF16.yaml, 2) / DeepSeek-V3.1-BF16.yaml

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

multi-node (multi-node-deepseek-v3.1, DeepSeek-V3.1-BF16.yaml, 2) / DeepSeek-V3.1-BF16.yaml

Process completed with exit code 1.

multi-node (multi-node-deepseek-v3.1, DeepSeek-V3.1-BF16.yaml, 2) / DeepSeek-V3.1-BF16.yaml

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (deepseek3_2-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test_dee... / tests/e2e/nightly/single_node/models/test_deepseek_v3_2_w8a8.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (deepseek3_2-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test_dee... / tests/e2e/nightly/single_node/models/test_deepseek_v3_2_w8a8.py

Process completed with exit code 1.

single-node (deepseek3_2-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test_dee... / tests/e2e/nightly/single_node/models/test_deepseek_v3_2_w8a8.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (qwen3-235b-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test_qwen... / tests/e2e/nightly/single_node/models/test_qwen3_235b_w8a8.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (qwen3-235b-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test_qwen... / tests/e2e/nightly/single_node/models/test_qwen3_235b_w8a8.py

Process completed with exit code 1.

single-node (qwen3-235b-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test_qwen... / tests/e2e/nightly/single_node/models/test_qwen3_235b_w8a8.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (qwen3-235b-a22b-w8a8-eplb, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models... / tests/e2e/nightly/single_node/models/test_qwen3_235b_a22b_w8a8_eplb.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (qwen3-235b-a22b-w8a8-eplb, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models... / tests/e2e/nightly/single_node/models/test_qwen3_235b_a22b_w8a8_eplb.py

Process completed with exit code 1.

single-node (qwen3-235b-a22b-w8a8-eplb, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models... / tests/e2e/nightly/single_node/models/test_qwen3_235b_a22b_w8a8_eplb.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (deepseek-r1-w8a8-hbm, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test... / tests/e2e/nightly/single_node/models/test_deepseek_r1_w8a8_hbm.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (deepseek-r1-w8a8-hbm, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test... / tests/e2e/nightly/single_node/models/test_deepseek_r1_w8a8_hbm.py

Process completed with exit code 1.

single-node (deepseek-r1-w8a8-hbm, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/test... / tests/e2e/nightly/single_node/models/test_deepseek_r1_w8a8_hbm.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (deepseek-r1-0528-w8a8-prefix-cache, linux-aarch64-a3-16, tests/e2e/nightly/single_no... / tests/e2e/nightly/single_node/models/test_prefix_cache_deepseek_r1_0528_w8a8.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (deepseek-r1-0528-w8a8-prefix-cache, linux-aarch64-a3-16, tests/e2e/nightly/single_no... / tests/e2e/nightly/single_node/models/test_prefix_cache_deepseek_r1_0528_w8a8.py

Process completed with exit code 1.

single-node (deepseek-r1-0528-w8a8-prefix-cache, linux-aarch64-a3-16, tests/e2e/nightly/single_no... / tests/e2e/nightly/single_node/models/test_prefix_cache_deepseek_r1_0528_w8a8.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (deepseek-r1-w8a8-eplb, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_deepseek_r1_0528_w8a8_eplb.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (deepseek-r1-w8a8-eplb, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_deepseek_r1_0528_w8a8_eplb.py

Process completed with exit code 1.

single-node (deepseek-r1-w8a8-eplb, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_deepseek_r1_0528_w8a8_eplb.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (deepseek-r1-0528-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_deepseek_r1_0528_w8a8.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (deepseek-r1-0528-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_deepseek_r1_0528_w8a8.py

Process completed with exit code 1.

single-node (deepseek-r1-0528-w8a8, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_deepseek_r1_0528_w8a8.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

single-node (deepseek-r1-w8a8-mtpx, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_mtpx_deepseek_r1_0528_w8a8.py

Executing the custom container implementation failed. Please contact your self hosted runner administrator.

single-node (deepseek-r1-w8a8-mtpx, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_mtpx_deepseek_r1_0528_w8a8.py

Process completed with exit code 1.

single-node (deepseek-r1-w8a8-mtpx, linux-aarch64-a3-16, tests/e2e/nightly/single_node/models/tes... / tests/e2e/nightly/single_node/models/test_mtpx_deepseek_r1_0528_w8a8.py

Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1

Artifacts

Produced during runtime

Name	Size	Digest
DeepSeek-R1-W8A8-longseq.yaml-pod-logs	12.7 KB	`sha256:4cbe9b885be45ac64ed31a7f9670fd6e81b1198ee34e1f2760a65e80a6acde84`
DeepSeek-V3.1-BF16.yaml-pod-logs	10.6 KB	`sha256:227673f1bfd14939c7bbd41b42bb5f616c4ee3619a7204c69c6af5cd0b39e9f8`
DeepSeek-V3.yaml-pod-logs	35.7 KB	`sha256:420404ca7d984a05157b88d55cdb23d7934c2cec3cef53be4227fe04b892bd5f`
DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml-pod-logs	11.5 KB	`sha256:659f0f14652206b11170cfcbf6fa65306e4f4c20ee98aaea4955a6e659b58d78`
Kimi-K2-Instruct-W8A8.yaml-pod-logs	10.6 KB	`sha256:93dec6548d5a13a6f2a94c2de9a5b4f08092b146e0cc320682e1ad0100c44ce7`
Qwen3-235B-A22B.yaml-pod-logs	9.61 KB	`sha256:07a47486333202aeedbefbb189dc2fc04e1f9ccbebfeefa8a9086d241ea0a490`
Qwen3-235B-W8A8-EPLB.yaml-pod-logs	11.8 KB	`sha256:eea86e01b5839c97db252062c9a4eaa91f50ff1b88df0e3d099f658374ac4923`
Qwen3-235B-W8A8-longseq.yaml-pod-logs	41.3 KB	`sha256:c84f67fc1f223f39a5c9b4f53d51ea1169cfdd21d8c61c0ae9c945b4b73b3846`
Qwen3-235B-W8A8.yaml-pod-logs	39.7 KB	`sha256:5d4678f7349ebcb35a6531fbf206a1bb1562f7b2d57a31f2a3959666b4e027f9`
Qwen3-235B-disagg-pd.yaml-pod-logs	411 KB	`sha256:380795407f49269baafbe76a006f666cb2fae0a1b318d5d16c7667ca29e87caf`
Qwen3-VL-235B-disagg-pd.yaml-pod-logs	187 KB	`sha256:e50eaef7f582bd0993f66de84de1bd58c043709f060b5583f4a7d2f5b905078b`