feat(pdp): add SP-to-SP piece pull endpoint

[rvagg]

Implement POST /pdp/piece/fetch for fetching pieces from other SPs. Includes idempotent request handling, failure tracking with retry status, and CommP verification for external sources.

New files:

• handlers_fetch.go: fetch endpoint and database store implementation See HandleFetch() Godoc for comprehensive details about new functionality.
• fetch_types.go: request/response types with source URL validation
• piece_cid.go: PieceCIDv2 parsing utilities (consolidated existing PieceCID code)
• 20251212-pdp-fetch.sql: tracking tables for fetch requests

Changes:

• task_park_piece.go: verify CommP for external sources, reduce MaxFailures to 5
• handlers.go: refactor to use shared PieceCID utilities

Pulls in lib/ffi/piece_funcs.go from main branch as is.

[LexLuthr]

Looks good. Just need to change the ParkPiece strategy.

[snadrus]

This PR is a concern as we have SPs with huge databases, and Support is looking for ways to reduce Curio's db impact (ex: #854 ). Streaming behavior must be debugged using logs, not databases.

[snadrus]

@rvagg Can we have the spec for this ticket?
I would like the Curio team to implement it in a supportable way.

[rvagg]

@snadrus sorry I should have linked to #828 as the issue behind this ticket. There's also a longer document that the description there came out of but I think that'd be noise at this stage. The godoc I put in the handler can/should also be treated as a spec with the desired client flow.

---

I hear you re database size, but we have a clean-up task in here that keeps it minimal, that could be tightened much further than it is now - currently to 5 days, to retain data for debugging purposes, but it could even be as small as an hour if we want. The problems with not storing state for this is:

• Removes options for doing idempotency on requests
• Client polling for status to understand progress becomes more difficult and less informative, Curio is more of a cross-fingers black box
• No clean failure tracking on record

We could just rely on parked_pieces and make this a per-piece interaction - fire once for the original request then query status for individual pieces, we don't quite get the level of granularity on status provided here. Also harmony_task_history might be able to give us some of the failure information, we'd have to join with it and return whatever it has to say. But it's a potentially big table and probably not intended for this purpose.

The trade-off would be: less granularity, separate status API(s), more complex queries against harmony_task_history. The choice here was two small, short-lived tables with aggressive cleanup over complex joins against large shared tables, for a more convenient, informative, and idempotent API.

[rvagg]

@LexLuthr in a new fixup! commit I've reworked this based on your feedback. Restored task_park_piece.go and lib/ffi/piece_funcs.go to their original state, so no changes to ParkPiece or the shared piece funcs.

Created a separate PDPv0_FetchPiece in tasks/pdp/task_fetch_piece.go that handles the fetch flow independently: downloads from source URL, computes and verifies CommP while streaming, stores to custore, then creates a parked_pieces entry for StorePiece to pick up. Tracking happens via pdp_piece_fetch_items which ties into the fetch request lifecycle, cleanup still happens via the existing CASCADE delete when fetch records age out.

The task runs up to 4 concurrent downloads, each isolated so a slow/bad source doesn't block others. Has a 1 hour total fetch timeout (these are capped at 1016 MiB total) and exponential backoff on retries (5 max). Still room for verified streaming retrieval in future but I hope this should be solid enough for a starter?

[BigLep]

Curio: what are the next steps for getting this approved/merged? I want to make sure we're clear about what things need to get done now or can be improved later, particularly with the merge back to main. My understanding is that @rvagg has biased towards making the future merge back to main easier. That seems right to me. @snadrus: if Curio wants to do a different implementation, can that be handled in the pdpv0-main branch after the fact?

Context: this is the last remaining feature needed for FOC GA (as part of FilOzone/filecoin-services#357 )). There are still plenty of operational/tooling items though for FOC GA, but getting the GA durability story landed is key for us to turn focus towards bug fixing / operational improvement mode because we know more APIs won't be added to Synapse.

How do we get decisions here as quickly as person? I suggest we have a verbal sync (I can work on scheduling). Even if we end up deciding that more immediate work is required, I want to take days getting to that decision with async back and forth.

[rvagg]

Renamed everything in here from "Fetch" to "Pull".

During SDK integration I found it too hard to distinguish it from fetch() and didn't want to cause confusion, so the functionality is now called "Pull", you ask an SP to pull from another SP.

[rvagg]

@rjan90 we're going to get this landed before you cut a tag this week, keeping it open while I'm still testing on it in case something comes up, but it's ready as is, works, tested, just not heavily exercised.

outdated

[rjan90]

@rjan90 we're going to get this landed before you cut a tag this week, keeping it open while I'm still testing on it in case something comes up, but it's ready as is, works, tested, just not heavily exercised.

Discussed in sync today to merge this, so that we can cut a tag and get it shipped out to PDP SPs.