fix: speed up gopls lint with install optimization and optional parallelism #123
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
The gopls check was slow (~3.5 minutes) because it processed all ~2800 Go source files sequentially. This change: 1. Installs gopls once instead of using 'go run' each time 2. Uses xargs with parallel execution (-P nproc) to process files in batches of 100 across all available CPU cores This reduces the lint time from ~3.5 minutes to ~1 minute (about 68% faster). Closes #121 Co-Authored-By: Greg Slepak <contact@taoeffect.com>
devin-ai-integration
bot
force-pushed
the
devin/1769030545-fix-gopls-lint-performance
branch
from
622d3fc to
9a94079
Compare
devin-ai-integration
bot
changed the title
Add lint-frontend and lint-backend steps to run before their respective test steps, ensuring linting passes before tests run. Co-Authored-By: Greg Slepak <contact@taoeffect.com>
Linting is already handled by pull-compliance.yml which runs on all PRs. No need to duplicate lint steps in pull-tests.yml. Co-Authored-By: Greg Slepak <contact@taoeffect.com>
- Add cross-platform CPU detection (nproc for Linux, sysctl for macOS) - Add verification that gopls was installed successfully - Use null-delimited format (-0) with xargs to handle filenames with spaces - Remove stderr suppression from go install to surface installation errors Co-Authored-By: Greg Slepak <contact@taoeffect.com>
pedrogaudencio
left a comment
pedrogaudencio
left a comment
There was a problem hiding this comment.
Thank you for looking into this - switching from go run to installing gopls is a nice win and makes it cleaner.
There is one concern tho: the xargs -P "$NPROC" -n 100 approach ends up launching a bunch of gopls check processes at the same time. On my laptop that caused heavy memory/swap pressure (and lag) and actually made make lint slower overall, even tho it’s “more parallel”.
I’d suggest we keep the “install + reuse gopls” part, but run run gopls check in a single invocation (or sequential batches). If we still want parallelism, maybe make it a configurable and default to something conservative (e.g. GOPLS_PARALLEL=1 locally, and higher in CI).
I'm happy to help adjust the script to “install once + sequential check” vs “configurable parallelism”.
Address feedback from pedrogaudencio: parallel gopls processes can cause memory/swap pressure on local machines. Changed to: - Default GOPLS_PARALLEL=1 (sequential) for local development - CI can set GOPLS_PARALLEL to a higher value if desired - Removed automatic CPU detection since parallelism is now opt-in The 'install gopls once' optimization is kept, which still provides a speedup over the original 'go run' approach. Co-Authored-By: Greg Slepak <contact@taoeffect.com>
devin-ai-integration
bot
changed the title
|
Thanks for the feedback @pedrogaudencio! I've updated the script to address your concern:
The "install gopls once" optimization is kept, which still provides a speedup over the original |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| # Install gopls if not already installed, then use the installed binary for | ||
| # faster execution. Using 'go run' each time adds overhead. | ||
| "$GO" install "$GOPLS_PACKAGE" | ||
| GOPLS_BIN=$("$GO" env GOPATH)/bin/gopls |
There was a problem hiding this comment.
🔴 GOPLS_BIN path doesn't account for GOBIN environment variable
The script determines the gopls binary path using $("$GO" env GOPATH)/bin/gopls, but when the GOBIN environment variable is set, go install places binaries in $GOBIN instead of $GOPATH/bin.
When a user has GOBIN configured (a common setup for Go developers who want binaries in a custom location), the script will:
- Successfully install gopls (to
$GOBIN/gopls) - Look for gopls at
$GOPATH/bin/gopls(wrong location) - Fail the
-xcheck and exit with "Error: Failed to install gopls"
This causes the lint step to fail with a misleading error message even though gopls was installed correctly. The proper approach would be to use $("$GO" env GOBIN) if set, falling back to $("$GO" env GOPATH)/bin otherwise.
Recommendation: Use the correct binary path that accounts for GOBIN:
GOPLS_BIN=$({ "$GO" env GOBIN; } | grep . || echo "$("$GO" env GOPATH)/bin")/goplsOr more readably:
GOBIN_DIR=$("$GO" env GOBIN)
if [[ -z "$GOBIN_DIR" ]]; then
GOBIN_DIR=$("$GO" env GOPATH)/bin
fi
GOPLS_BIN="$GOBIN_DIR/gopls"
Was this helpful? React with 👍 or 👎 to provide feedback.
Closes #121
Summary
The
gopls checkstep inmake lintwas slow (~3.5 minutes) because it rango runeach time, recompiling gopls. This PR speeds it up by:go installinstead of usinggo runeach timeGOPLS_PARALLELenv var (defaults to 1 to avoid memory pressure)Before: ~3.5 minutes
After: ~3 minutes with default settings (sequential), faster with
GOPLS_PARALLELset higherChanges
tools/lint-go-gopls.shto:go installbefore running checksGOPLS_PARALLELenv var (default: 1)xargs -0 -P "$PARALLEL" -n 100with null-delimited input to handle filenames with spacesTesting
Verified that the linter still catches errors by intentionally adding a syntax error to
services/user/user.goand confirming it was detected.Updates since last revision
Addressed feedback from @pedrogaudencio: parallel gopls processes can cause memory/swap pressure on local machines. Changed parallelism to be opt-in via
GOPLS_PARALLELenv var, defaulting to 1 (sequential). CI can setGOPLS_PARALLEL=4or higher if desired.Human Review Checklist
go installapproach works correctly in CI environmentsGOPLS_PARALLELenv var is respected when set2>/dev/null) - this is intentional to filter noise but could hide some edge case errorsLink to Devin run: https://app.devin.ai/sessions/c29881ff27ee429e9b2592f7f655c998
Requested by: Greg Slepak (@taoeffect)