Marvin Test Failure Analysis #2693
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Marvin Test Failure Analysis | |
| on: | |
| workflow_run: | |
| workflows: ["Tests", "Run static analysis"] | |
| types: | |
| - completed | |
| concurrency: | |
| group: marvin-test-failure-${{ github.event.workflow_run.head_branch }} | |
| cancel-in-progress: true | |
| jobs: | |
| martian-test-failure: | |
| # Only run if the test workflow failed | |
| if: ${{ github.event.workflow_run.conclusion == 'failure' }} | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| issues: read | |
| id-token: write | |
| actions: read # Required for Claude to read CI results | |
| steps: | |
| - name: Checkout repository | |
| uses: actions/checkout@v6 | |
| with: | |
| fetch-depth: 1 | |
| - name: Generate Marvin App token | |
| id: marvin-token | |
| uses: actions/create-github-app-token@v2 | |
| with: | |
| app-id: ${{ secrets.MARVIN_APP_ID }} | |
| private-key: ${{ secrets.MARVIN_APP_PRIVATE_KEY }} | |
| - name: Set up Python 3.10 | |
| uses: actions/setup-python@v6 | |
| with: | |
| python-version: "3.10" | |
| # Install UV package manager | |
| - name: Install UV | |
| uses: astral-sh/setup-uv@v7 | |
| # Install dependencies | |
| - name: Install dependencies | |
| run: uv sync --all-packages --group dev | |
| - name: Set analysis prompt | |
| id: analysis-prompt | |
| run: | | |
| cat >> $GITHUB_OUTPUT << 'EOF' | |
| PROMPT<<PROMPT_END | |
| You're a test failure analysis assistant for FastMCP, a Python framework for building Model Context Protocol servers and clients. | |
| # Your Task | |
| A GitHub Actions workflow has failed. Your job is to: | |
| 1. Analyze the test failure(s) to understand what went wrong | |
| 2. Identify the root cause of the failure(s) | |
| 3. Suggest a clear, actionable solution to fix the failure(s) | |
| # Getting Started | |
| 1. Call the generate_agents_md tool to get a high-level summary of the project | |
| 2. Get the pull request associated with this workflow run from the GitHub repository: ${{ github.repository }} | |
| - The workflow run ID is: ${{ github.event.workflow_run.id }} | |
| - The workflow run was triggered by: ${{ github.event.workflow_run.event }} | |
| - Use GitHub MCP tools to get PR details and workflow run information | |
| 3. Use the GitHub MCP tools to fetch job logs and failure information: | |
| - Use get_workflow_run to get details about the failed workflow | |
| - Use list_workflow_jobs to see which jobs failed | |
| - Use get_job_logs with failed_only=true to get logs for failed jobs | |
| - Use summarize_run_log_failures to get an AI summary of what failed | |
| 4. Analyze the failures to understand the root cause | |
| 5. Search the codebase for relevant files, tests, and implementations | |
| # Your Response | |
| Post a comment on the pull request with your analysis. Your comment should include: | |
| ## Test Failure Analysis | |
| **Summary**: A brief 1-2 sentence summary of what failed. | |
| **Root Cause**: A clear explanation of why the tests failed, based on your analysis of the logs and code. | |
| **Suggested Solution**: Specific, actionable steps to fix the failure(s). Include: | |
| - Which files need to be modified | |
| - What changes are needed | |
| - Why these changes will fix the issue | |
| <details> | |
| <summary>Detailed Analysis</summary> | |
| Include here: | |
| - Relevant log excerpts showing the failure | |
| - Code snippets that are causing the issue | |
| - Any related issues or PRs that might be relevant | |
| </details> | |
| <details> | |
| <summary>Related Files</summary> | |
| List files that are relevant to the failure with brief explanations of their relevance. | |
| </details> | |
| # Important Guidelines | |
| - Be concise and actionable - developers want to quickly understand and fix the issue. Provide | |
| additional context, references, etc in collapsible details blocks to ensure that the comment you're adding | |
| is short and easy to read but additional information is a click away. | |
| - Focus on facts from the logs and code, not speculation | |
| - If you can't determine the root cause, say so clearly | |
| - If your only suggestion is a bad suggestion (disable the test, change the timeout, etc), indicate that you've run out of ideas and | |
| that they probably don't want to do that. | |
| - Provide specific file names, line numbers, and code references when possible | |
| - You can run make commands (e.g., `make lint`, `make typecheck`, `make sync`) to build, test, or lint the code | |
| - You can also run git commands (e.g., `git status`, `git log`, `git diff`) to inspect the repository | |
| - You can use WebSearch and WebFetch to research errors, stack traces, or related issues | |
| - For bash commands, you are limited to make and git commands only | |
| # CRITICAL: ANGRY USERS | |
| **IMPORTANT**: If the user is angry with you, the triage bot, don't respond. Just exit immediately without further action. | |
| If at any point in the conversation the user has asked you to stop replying to the thread, just exit immediately. | |
| If you are posting the same suggestion as you have previously made, do not post the suggestion again. | |
| # IMPORTANT: EDIT YOUR COMMENT | |
| Do not post a new comment every time you triage a failing workflow. If a previous comment has been posted by you (marvin) | |
| in a previous triage, edit that comment do not add a new comment for each failure. Be sure to include a note that you've edited | |
| your comment to reflect the latest analysis. Don't worry about keeping the old content around, there's comment history for | |
| that. | |
| # Problems Encountered | |
| If you encounter any problems during your analysis (e.g., unable to fetch logs, tools not working), document them clearly so the team knows what limitations you faced. | |
| PROMPT_END | |
| EOF | |
| - name: Setup GitHub MCP Server | |
| run: | | |
| mkdir -p /tmp/mcp-config | |
| cat > /tmp/mcp-config/mcp-servers.json << 'EOF' | |
| { | |
| "mcpServers": { | |
| "repository-summary": { | |
| "type": "http", | |
| "url": "https://agents-md-generator.fastmcp.app/mcp" | |
| }, | |
| "code-search": { | |
| "type": "http", | |
| "url": "https://public-code-search.fastmcp.app/mcp" | |
| }, | |
| "github-research": { | |
| "type": "stdio", | |
| "command": "uvx", | |
| "args": [ | |
| "github-research-mcp" | |
| ], | |
| "env": { | |
| "DISABLE_SUMMARIES": "true", | |
| "GITHUB_PERSONAL_ACCESS_TOKEN": "${{ secrets.GITHUB_TOKEN }}" | |
| } | |
| } | |
| } | |
| } | |
| EOF | |
| - name: Clean up stale Claude locks | |
| run: rm -rf ~/.claude/.locks ~/.local/state/claude/locks || true | |
| - name: Run Claude Code | |
| id: claude | |
| uses: anthropics/claude-code-action@v1 | |
| with: | |
| github_token: ${{ steps.marvin-token.outputs.token }} | |
| anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY_FOR_CI }} | |
| bot_name: "Marvin Context Protocol" | |
| claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} | |
| additional_permissions: | | |
| actions: read | |
| prompt: ${{ steps.analysis-prompt.outputs.PROMPT }} | |
| claude_args: | | |
| --allowed-tools mcp__repository-summary,mcp__code-search,mcp__github-research,WebSearch,WebFetch,Bash(make:*,git:*) | |
| --mcp-config /tmp/mcp-config/mcp-servers.json |