Redesign the indexing process to sync bitcoin faster #180
Labels
help wanted
Extra attention is needed
roadmap
A feature that will be developed
server
Changes required on the server project
We need to optimize the blockchain syncing process.
Expected behavior
Indexing Bitcoin should take at most 1 week.
Actual behavior
Indexing Bitcoin takes months!
Steps to reproduce the behavior
Just mount a full Bitcoin node, and follow the steps to sync the explorer with it.
Notes
This is a very complex task, which involves work from the infra side to the backend work.
On the infra side, we need to use a load balancer for the bitcoind RPC API, based on previous experiences, the minimum requirements are:
On the explorer side:
On the approach to take, the syncing process should be done in several stages (looks like a good candidate for akka-streams):
keeping 3 nodes at minimum, with 8GB on ram or more, and
As we don't require the whole data to be indexed, ideally we should be able to disable some stages to speed up the process, and save space, these are good candidates (sql tables):
All of this would affect the exposed API, because we shouldn't return blocks that aren't fully synced, also, it's important to consider potential rollbacks while syncing the data.
The text was updated successfully, but these errors were encountered: