Skip to main content

18 posts tagged with "db-sync"

View All Tags

· One min read
Kostas Dermentzis

High level summary​

The DBSync team is preparing a release which introduces schema simplifications, removes indexes, unique and foreign keys. It also provides a way to fix older values and migrates without the need to resync from genesis.

Lower level summary​

Schema simplifications​

Indexes, Unique and Foreign keys are removed in order to speedup syncing #1295 The same pr also introduces a different way to rollback, which doesn't rely on foreign keys and indexes.

Performance​

The DBSync team ran a big number of benchmarks and investigated ways to speedup syncing. A conservative number of these will be included in the next release and the rest can be found in performance view.

Migrations and resyncing​

The next release will be 13.1.0, it will enable a migration without the need to resync. It will also introduce a procedure that fixes bytes values of Datum and RedeemerData in existing databases #1294

Release​

The release has been mostly cherry-picked from master #1294 and its scope can be seen release view

· 2 min read
Kostas Dermentzis

DBSync Update​

Fast restarts​

We fixed a long overdue issue in db-sync which caused long delays on restarts 1266. This has been one of db-sync main objectives for this period. Restarts are now very fast, because db-sync deletes almost nothing from the db, it just replays the ledger rules until it reaches the tip of the db. The fix also improves reconnection speed, in cases where the node restarts or the connection is temporarily lost. It also speeds up even more in cases where due to a deployment mess up a very old snapshot or no snapshot at all is used.

Property based testing​

We added stateful property based testing, using quickcheck-state-machine 1269. These tests use empty or almost empty blocks to test the new behaviour of restarts and rollbacks.

These tests generate arbitrarily a list of symbolic commands from these:

RollForward Int
RollBack BlockNo
StopDBSync
StartDBSync
RestartNode
AssertBlockNo BlockNo

The commands are translated into real commands. For example RollForward Int will forge a new block that fits on the current chain. These real commands are executed against db-sync using the mock chain-sync server. The symbolic commands are executed against a vesy simplistic Model of db-sync which looks like this:

  Model
    { serverTip :: BlockNo
    , dbSyncTip :: BlockNo
    , dbSynsIsOn :: Bool
    , dbSynsHasSynced :: Bool
    }

Finally a number of postconditions are checked, related to the eventual block number of db-sync.

Tech Debt​

We handled a number of tech debt in 1275 This improves the code format of db-sync, deletes many queries that were never used and groups the others. This tech debt resolution not only improves the experience of working in db-sync, but can facilitate some of our other objectives, as it makes it very explicit which queries are used during syncing and which indexes are necessary.

Smash​

We worked on fixing an issue related to fetching pool metadata 1276. The issue which is described in 1270

· 2 min read
Kostas Dermentzis

DBSync Update​

New Tag​

We created a new db-sync tag 13.0.5 which addresses shortcomings of the last release 13.0.4. It is currently under testing. The Changelog is here and in more details:

  • We fixed fees for tx with phase 2 failure that didn't include a total collateral field. 1248

  • We fixed an issue that could cause db-sync to crash if a specific rollback occured. 1247

  • DBSync will now avoid reserialising data, especially Datums, which not only slows down db-sync but could result in the wrong CBOR encoding being inserted. 1217

  • All the fixes above come with unit tests which validates the fix.

  • Added support for preprod and preview from docker. DBSync no longer needs to include the configs for different networks, these are directly fetched from the cardano world. 1254

  • We added better support from docker for the new disable options and the overall documentation. 1260

All the above were also backported to the master branch

Open source​

We made the db-sync board public, so everyone can have access on the issues we prioritise. We also added new tasks to the board, some of them could be approachable to newcomers or people who want to contribute.

Progress on tech debt and new features​

  • 1223 was merged, which removes the foreign keys from the db schema. This opens the road to a number of optimizations.

  • An additional fix on top of the previous work was added 1250

  • An initial version where DBSync does not rollback on restart is done here 1266. This allows db-sync to restart much faster, without the need to delete data and reinsert them. In the future it can also facilitate migrations in cases where the ledger snapshots have a breaking change, without the need to resync everything from genesis.