Skip to content

Commit 1a7bb46

Browse files
albertocavalcantealbertocavalcante
committed
docs(tools): add vendoring plan for buildtools parser
Document two approaches to eliminate the buildtools dependency: 1. Vendoring automation tool to extract build/labels/tables packages 2. Fork plan for a standalone starlark-parser module The buildtools parser (8K LOC) uses only stdlib - no external deps.
1 parent 4ee6ef9 commit 1a7bb46

File tree

1 file changed

+355
-0
lines changed

1 file changed

+355
-0
lines changed

‎tools/VENDORING_PLAN.md‎

Lines changed: 355 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,355 @@
1+
# Starlark Parser Vendoring Plan
2+
3+
This document outlines two approaches to eliminate the `bazelbuild/buildtools` dependency:
4+
5+
1. **Vendoring automation** - Tool to vendor the parser into this repo
6+
2. **Fork plan** - Create a standalone parser module
7+
8+
---
9+
10+
## Current State
11+
12+
```
13+
go-bzlmod
14+
└── depends on: github.com/bazelbuild/buildtools (full module)
15+
└── we only use: build/, labels/, tables/ packages (~8K LOC, stdlib only)
16+
```
17+
18+
The `build` package (Starlark parser) has **zero external dependencies** - it only uses stdlib.
19+
The starlark-go/protobuf deps in buildtools are for buildifier/buildozer, not the parser.
20+
21+
---
22+
23+
## Option 1: Vendoring Automation
24+
25+
### Target Structure
26+
27+
```
28+
go-bzlmod/
29+
├── internal/
30+
│ └── starlark/ # Vendored parser
31+
│ ├── build/ # AST types, parser, printer
32+
│ ├── labels/ # Bazel label parsing
33+
│ ├── tables/ # Known rules/attributes
34+
│ └── VERSION # Tracks vendored version
35+
├── tools/
36+
│ └── vendor-parser/
37+
│ ├── main.go # Vendoring tool
38+
│ └── README.md
39+
```
40+
41+
### Vendoring Tool Design
42+
43+
**Command:**
44+
45+
```bash
46+
go run ./tools/vendor-parser -version v0.0.0-20250602201422-b1e23f1025b8
47+
# or
48+
go run ./tools/vendor-parser -commit b1e23f1025b8
49+
# or
50+
go run ./tools/vendor-parser -tag v7.1.2
51+
```
52+
53+
**Tool Workflow:**
54+
55+
```
56+
┌─────────────────────────────────────────────────────────────────────┐
57+
│ 1. DOWNLOAD │
58+
│ - Fetch tarball from GitHub at specified version/commit/tag │
59+
│ - URL: github.com/bazelbuild/buildtools/archive/{ref}.tar.gz │
60+
│ - Verify download (checksum if provided) │
61+
└─────────────────────────────────────────────────────────────────────┘
62+
│
63+
▼
64+
┌─────────────────────────────────────────────────────────────────────┐
65+
│ 2. EXTRACT │
66+
│ - Extract only: build/, labels/, tables/ │
67+
│ - Skip: *_test.go files (optional, configurable) │
68+
│ - Skip: testdata/ directories │
69+
└─────────────────────────────────────────────────────────────────────┘
70+
│
71+
▼
72+
┌─────────────────────────────────────────────────────────────────────┐
73+
│ 3. REWRITE IMPORTS │
74+
│ - Old: github.com/bazelbuild/buildtools/build │
75+
│ - New: github.com/albertocavalcante/go-bzlmod/internal/starlark/build │
76+
│ - Same for: labels, tables │
77+
│ - Use go/ast or simple string replacement │
78+
└─────────────────────────────────────────────────────────────────────┘
79+
│
80+
▼
81+
┌─────────────────────────────────────────────────────────────────────┐
82+
│ 4. WRITE VERSION FILE │
83+
│ internal/starlark/VERSION: │
84+
│ { │
85+
│ "source": "github.com/bazelbuild/buildtools", │
86+
│ "version": "v0.0.0-20250602201422-b1e23f1025b8", │
87+
│ "commit": "b1e23f1025b8...", │
88+
│ "vendored_at": "2026-02-02T12:00:00Z", │
89+
│ "packages": ["build", "labels", "tables"], │
90+
│ "checksum": "sha256:..." │
91+
│ } │
92+
└─────────────────────────────────────────────────────────────────────┘
93+
│
94+
▼
95+
┌─────────────────────────────────────────────────────────────────────┐
96+
│ 5. UPDATE INTERNAL IMPORTS │
97+
│ - Update go-bzlmod source files to use internal/starlark/... │
98+
│ - Files: parser.go, ast/parser.go, internal/buildutil/attr.go │
99+
│ - Run: go mod tidy (removes buildtools dependency) │
100+
└─────────────────────────────────────────────────────────────────────┘
101+
│
102+
▼
103+
┌─────────────────────────────────────────────────────────────────────┐
104+
│ 6. VERIFY │
105+
│ - go build ./... │
106+
│ - go test ./... │
107+
│ - Confirm buildtools removed from go.mod │
108+
└─────────────────────────────────────────────────────────────────────┘
109+
```
110+
111+
### Tool Implementation Notes
112+
113+
```go
114+
// tools/vendor-parser/main.go
115+
116+
package main
117+
118+
import (
119+
"archive/tar"
120+
"compress/gzip"
121+
"flag"
122+
"go/ast"
123+
"go/parser"
124+
"go/printer"
125+
"go/token"
126+
"io"
127+
"net/http"
128+
"os"
129+
"path/filepath"
130+
"strings"
131+
)
132+
133+
const (
134+
buildtoolsRepo = "bazelbuild/buildtools"
135+
targetDir = "internal/starlark"
136+
oldImportPath = "github.com/bazelbuild/buildtools"
137+
newImportPath = "github.com/albertocavalcante/go-bzlmod/internal/starlark"
138+
)
139+
140+
var packages = []string{"build", "labels", "tables"}
141+
142+
func main() {
143+
version := flag.String("version", "", "Go module version (e.g., v0.0.0-20250602...)")
144+
commit := flag.String("commit", "", "Git commit hash")
145+
tag := flag.String("tag", "", "Git tag (e.g., v7.1.2)")
146+
keepTests := flag.Bool("keep-tests", false, "Keep test files")
147+
flag.Parse()
148+
149+
// ... implementation
150+
}
151+
```
152+
153+
### Key Functions Needed
154+
155+
1. **`downloadTarball(ref string) (io.ReadCloser, error)`**
156+
- Fetch from `https://github.com/{repo}/archive/{ref}.tar.gz`
157+
158+
2. **`extractPackages(tarball io.Reader, packages []string, dest string) error`**
159+
- Extract only specified packages from tarball
160+
161+
3. **`rewriteImports(dir string, oldPath, newPath string) error`**
162+
- Parse Go files with go/ast
163+
- Rewrite import paths
164+
- Write back with go/printer
165+
166+
4. **`writeVersionFile(dest string, meta VersionMeta) error`**
167+
- Write JSON metadata about vendored version
168+
169+
5. **`updateProjectImports(projectRoot string) error`**
170+
- Update imports in main project files
171+
172+
### Usage After Implementation
173+
174+
```bash
175+
# Vendor a specific version
176+
go run ./tools/vendor-parser -version v0.0.0-20250602201422-b1e23f1025b8
177+
178+
# Vendor from a commit
179+
go run ./tools/vendor-parser -commit b1e23f1025b8
180+
181+
# Vendor from a tag
182+
go run ./tools/vendor-parser -tag v7.1.2
183+
184+
# Check what's currently vendored
185+
cat internal/starlark/VERSION
186+
187+
# Update to latest
188+
go run ./tools/vendor-parser -tag $(gh api repos/bazelbuild/buildtools/releases/latest -q .tag_name)
189+
```
190+
191+
### Maintenance Workflow
192+
193+
```bash
194+
# 1. Check for updates
195+
gh api repos/bazelbuild/buildtools/releases/latest
196+
197+
# 2. Review changelog
198+
gh release view v7.2.0 -R bazelbuild/buildtools
199+
200+
# 3. Vendor new version
201+
go run ./tools/vendor-parser -tag v7.2.0
202+
203+
# 4. Run tests
204+
go test ./...
205+
206+
# 5. Commit
207+
git add internal/starlark/
208+
git commit -m "build: vendor buildtools parser v7.2.0"
209+
```
210+
211+
---
212+
213+
## Option 2: Fork Plan
214+
215+
Create a standalone module: `github.com/albertocavalcante/starlark-parser`
216+
217+
### Fork Structure
218+
219+
```
220+
starlark-parser/
221+
├── build/ # Parser, AST, printer (from buildtools)
222+
├── labels/ # Label parsing (from buildtools)
223+
├── tables/ # Known rules (from buildtools)
224+
├── go.mod # module github.com/albertocavalcante/starlark-parser
225+
├── LICENSE # Apache 2.0 (same as buildtools)
226+
├── README.md # Attribution, usage
227+
└── .github/
228+
└── workflows/
229+
└── sync.yml # Automation to sync from upstream
230+
```
231+
232+
### go.mod (Zero Dependencies!)
233+
234+
```go
235+
module github.com/albertocavalcante/starlark-parser
236+
237+
go 1.21
238+
```
239+
240+
### Sync Automation (.github/workflows/sync.yml)
241+
242+
```yaml
243+
name: Sync from upstream
244+
245+
on:
246+
schedule:
247+
- cron: "0 0 * * 0" # Weekly
248+
workflow_dispatch:
249+
inputs:
250+
upstream_ref:
251+
description: "Upstream ref to sync (tag, commit, or branch)"
252+
default: "master"
253+
254+
jobs:
255+
sync:
256+
runs-on: ubuntu-latest
257+
steps:
258+
- uses: actions/checkout@v4
259+
260+
- name: Fetch upstream
261+
run: |
262+
git clone --depth=1 --filter=blob:none --sparse \
263+
https://github.com/bazelbuild/buildtools.git upstream
264+
cd upstream
265+
git sparse-checkout set build labels tables
266+
267+
- name: Copy packages
268+
run: |
269+
rm -rf build labels tables
270+
cp -r upstream/build upstream/labels upstream/tables .
271+
rm -rf upstream
272+
273+
- name: Rewrite imports
274+
run: |
275+
find . -name "*.go" -exec sed -i \
276+
's|github.com/bazelbuild/buildtools|github.com/albertocavalcante/starlark-parser|g' {} +
277+
278+
- name: Test
279+
run: go test ./...
280+
281+
- name: Create PR
282+
uses: peter-evans/create-pull-request@v5
283+
with:
284+
title: "sync: update from upstream buildtools"
285+
body: "Automated sync from bazelbuild/buildtools"
286+
branch: sync-upstream
287+
```
288+
289+
### Usage in go-bzlmod
290+
291+
```go
292+
// go.mod
293+
require github.com/albertocavalcante/starlark-parser v1.0.0
294+
295+
// parser.go
296+
import "github.com/albertocavalcante/starlark-parser/build"
297+
```
298+
299+
### Fork Versioning Strategy
300+
301+
| Upstream | Fork Version | Notes |
302+
| ----------------- | ------------ | ------------ |
303+
| v7.1.2 | v1.0.0 | Initial fork |
304+
| v7.2.0 | v1.1.0 | Sync update |
305+
| v8.0.0 (breaking) | v2.0.0 | Major sync |
306+
307+
### Fork Maintenance Checklist
308+
309+
- [ ] Initial fork with packages extracted
310+
- [ ] Import paths rewritten
311+
- [ ] Tests passing
312+
- [ ] LICENSE file (Apache 2.0)
313+
- [ ] README with attribution
314+
- [ ] GitHub Actions for automated sync
315+
- [ ] Tagged release (v1.0.0)
316+
- [ ] Update go-bzlmod to use fork
317+
318+
---
319+
320+
## Comparison
321+
322+
| Aspect | Vendoring | Fork |
323+
| -------------------- | --------------------- | ---------------------- |
324+
| **Dependency count** | 0 (internal) | 1 (your module) |
325+
| **Maintenance** | Manual sync | Automated PR |
326+
| **Reusability** | Only this project | Any project |
327+
| **Import path** | Long internal path | Clean module path |
328+
| **Setup effort** | Medium (tool once) | Medium (repo setup) |
329+
| **Ongoing effort** | Run tool occasionally | Merge PRs occasionally |
330+
331+
---
332+
333+
## Recommendation
334+
335+
**Start with vendoring** (Option 1):
336+
337+
1. Faster to implement
338+
2. Zero dependencies immediately
339+
3. Can convert to fork later if needed
340+
341+
**Consider fork later** if:
342+
343+
- You have other projects needing the parser
344+
- You want to contribute improvements upstream
345+
- Community interest in a minimal parser module
346+
347+
---
348+
349+
## Next Steps
350+
351+
1. [ ] Implement `tools/vendor-parser/main.go`
352+
2. [ ] Test vendoring workflow
353+
3. [ ] Remove buildtools dependency
354+
4. [ ] Document in README
355+
5. [ ] (Future) Create fork if beneficial

0 commit comments

Comments
 (0)