Skip to main content

16 posts tagged with "db-sync"

View All Tags

· One min read
Kostas Dermentzis

High level summary​

After spending the last months on improving DBSync and releasing 13.1.0.0 the DBSync team focused the last weeks on integrating upstream changes and tech debt. This includes integrating CHaP, the new leger core and ghc-9.2.4

Lower level summary​

  • Integrate CHaP #1331
  • Integrate new ledger core 0.1.1.1 and ghc-9.2 #1332
  • Technical debt: Add code formatter fourmolu #1334
  • Preparing and tracking Conway and UTxO integration

· One min read
Kostas Dermentzis

High level summary​

The DBSync team continued testing release 13.1.0.0. The QA team has reported that no issues have been found. The DBSync team also worked on cherry-picks back to master and on fixing bugs.

Lower level summary​

  • Release is cherry-picked back to master, which uses the new rollback mechanism which uses reverse indexes, same as the release #1320 This also fixes a bug number of issues on master.
  • Depenencies upgrade and CHaP integration #1324
  • AdaPots fix #1323. This fixes an issue where the per epoch AdaPots didn't match the epoch boundary, but they also included changes from the first block of the epoch.
  • Deposits Event fix #3212. This pr adjusts the Deposits ledger events, so that it can be better used by db-sync. This can reduce the number of queries that db-sync does during syncing an make syncing faster.

· One min read
Kostas Dermentzis

High level summary​

The DB Sync team prepared a release 13.1.0.0-rc2 which includes many improvements for db-sync, it makes rollbacks and syncing much faster, simplifies the schema, fixes bugs and introduces migrations. This release finalises the objectives that were set for db-sync for the previous 3 months period and part of the syncing speed objective set for the next period Changelog

Lower level summary​

  • Branch release/13.1.0.x includes all the improvements related to the release. The release is passing through the testing phase and a number of bugs and issues have been fixed, like #1312 #1311. Also many new unit tests have been added.

  • Part of the release branch is cherry-picked back into master, in a way that it respects the new release and development process, so that it takes into account migrations release process

  • The DB Sync team has also tagged release 13.0.6 which better supports preview and preprod for docker.

· One min read
Kostas Dermentzis

High level summary​

The DBSync team is preparing a release which introduces schema simplifications, removes indexes, unique and foreign keys. It also provides a way to fix older values and migrates without the need to resync from genesis.

Lower level summary​

Schema simplifications​

Indexes, Unique and Foreign keys are removed in order to speedup syncing #1295 The same pr also introduces a different way to rollback, which doesn't rely on foreign keys and indexes.

Performance​

The DBSync team ran a big number of benchmarks and investigated ways to speedup syncing. A conservative number of these will be included in the next release and the rest can be found in performance view.

Migrations and resyncing​

The next release will be 13.1.0, it will enable a migration without the need to resync. It will also introduce a procedure that fixes bytes values of Datum and RedeemerData in existing databases #1294

Release​

The release has been mostly cherry-picked from master #1294 and its scope can be seen release view

· 2 min read
Kostas Dermentzis

DBSync Update​

Fast restarts​

We fixed a long overdue issue in db-sync which caused long delays on restarts 1266. This has been one of db-sync main objectives for this period. Restarts are now very fast, because db-sync deletes almost nothing from the db, it just replays the ledger rules until it reaches the tip of the db. The fix also improves reconnection speed, in cases where the node restarts or the connection is temporarily lost. It also speeds up even more in cases where due to a deployment mess up a very old snapshot or no snapshot at all is used.

Property based testing​

We added stateful property based testing, using quickcheck-state-machine 1269. These tests use empty or almost empty blocks to test the new behaviour of restarts and rollbacks.

These tests generate arbitrarily a list of symbolic commands from these:

RollForward Int
RollBack BlockNo
StopDBSync
StartDBSync
RestartNode
AssertBlockNo BlockNo

The commands are translated into real commands. For example RollForward Int will forge a new block that fits on the current chain. These real commands are executed against db-sync using the mock chain-sync server. The symbolic commands are executed against a vesy simplistic Model of db-sync which looks like this:

  Model
    { serverTip :: BlockNo
    , dbSyncTip :: BlockNo
    , dbSynsIsOn :: Bool
    , dbSynsHasSynced :: Bool
    }

Finally a number of postconditions are checked, related to the eventual block number of db-sync.

Tech Debt​

We handled a number of tech debt in 1275 This improves the code format of db-sync, deletes many queries that were never used and groups the others. This tech debt resolution not only improves the experience of working in db-sync, but can facilitate some of our other objectives, as it makes it very explicit which queries are used during syncing and which indexes are necessary.

Smash​

We worked on fixing an issue related to fetching pool metadata 1276. The issue which is described in 1270