Use engine api get-blobs for block subscriber
#14513
Conversation
terencechain
force-pushed
the
get-blobs
branch
4 times, most recently
from
7ce1a18
to
f52c455
Compare
terencechain
force-pushed
the
get-blobs
branch
from
f52c455
to
7f73d11
Compare
|
|
||
| // Initialize KZG hashes and retrieve blobs | ||
| kzgHashes := make([]common.Hash, len(kzgCommitments)) | ||
| blobs, err := s.GetBlobs(ctx, kzgHashes) |
There was a problem hiding this comment.
Can we check if the EL supports this API (cached exchange capabilities result?) before attempting to call it? Otherwise the logs could be noisy for anyone who upgrades to this version with an EL that does not support getBlobs.
| defer span.End() | ||
|
|
||
| result := make([]*pb.BlobAndProof, len(versionedHashes)) | ||
| err := s.rpcClient.CallContext(ctx, &result, GetBlobsV1, versionedHashes) |
There was a problem hiding this comment.
I don't see the json methodset defined on this type, you probably need to define unmarshaling for the encoding to come through corrrectly.
| if err != nil { | ||
| return nil, errors.Wrap(err, "could not create RO blob with root") | ||
| } | ||
| verifiedBlobs = append(verifiedBlobs, blocks.NewVerifiedROBlob(roBlob)) |
There was a problem hiding this comment.
You're calling NewVerifiedROBlob and minting VerifiedROBlobs here, rather than getting those values from a verifier. That is risky. Would it be possible to define a verifier for this? Maybe something like the initial sync verifier that can deal with a batch.
On that note, I don't see you verifying the commitment inclusion proof anywhere in this PR.
Terence reminded me we are constructing the proof so we don't have to verify it, derp.
| log.WithFields(blobFields(sidecar.ROBlob)).WithError(err).Error("Failed to receive blob") | ||
| } | ||
|
|
||
| if err := s.cfg.p2p.BroadcastBlob(ctx, sidecar.Index, sidecar.BlobSidecar); err != nil { |
There was a problem hiding this comment.
Would it be safe to broadcast before we call ReceiveBlob? I think it would be best to do these steps across multiple loops, like:
- verify all of the blobs (I actually think we should do this inside ReconstructBlobSidecars since that method returns VerifiedROBlobs).
- call BroadcastBlob for all the blobs. - this ensures we get the most benefit from idontwant. I think that calling broadcast is non-blocking; io with the individual peers happens in separate libp2p threads. don't need to worry about "rebroadcasting" blobs where we have the index on disk, because libp2p handles that.
- Call ReceiveBlob for each blob. Do this after all the broadcasts because it will save the blobs to disk, which can block, esp if they have fsync enabled.
Reference: ethereum/execution-apis#559
Background
The$T_{BlockArrival} + \triangle_{BlobsOverEngineAPI} < T_{BlockArrival} + \triangle_{BlobsOverP2P}$ which simplifies to: $\triangle_{BlobsOverEngineAPI} < \triangle_{BlobsOverP2P}$
engine_getBlobsV1function allows the CL to retrieve blobs and their proofs from the EL by providing the block body's KZG version hashes. If the blobs and proofs are already available in the local mempool, the EL will return them. Once the CL receives the blobs and proofs, it can import the block and run the fork choice algorithm to update the head. In the best-case scenario, this process is faster than waiting for blobs to propagate through the network. By assuming the following condition holds:The benefits include:
Prysm Implementation
Now we will discuss how Prysm may implement this optimization. Feel free to stop reading here if you're not interested in the Prysm side of things.
One way to study this is by reviewing the following task timeline:
The best approach is likely to introduce another background task between steps 3 and 4:
3a. Request blobs from the EL based on the block's KZG commitments. If the blobs can be retrieved, construct blob sidecars and save them to the database.
This is the only change required for the optimization. In step 5, if the blob sidecars exist in the database, the block should pass.
Implementation Details
engine_getBlobsV1. If the blob sidecars don't exist, nothing happens.ReceiveBlob.Implications
Once I have the blob sidecars from the EL, should I still forward blob sidecars from P2P?
Yes, we should as good citizens of the network! We need to ensure that even if the blob sidecars already exist in the database, whether constructed from the EL or received via the network, they are forwarded. If they are constructed from the EL, we should remember to forward them at least once.
Can I forward or save blob sidecars in the DB without blob p2p gossip checks?
There are two points to this. The first is that once you import the block, you've already implicitly verified the blob sidecars using fields like slot and index. The second is that you trust the EL client to correctly construct the proof for you, which is no different from building a block and blobs using EL today. I don't think you need any further blob sidecar verification before forwarding to the network.
Could staled blob sidecar exist in DB?
This isn't much different from the current behavior, where blob sidecars can be imported without an accompanying block in Prysm. In this case, the blob sidecars will still be pruned after their retention window, and there's an upper limit on how often this can occur.