Upgrade Handlers

Understanding and implementing on-chain upgrade handlers for coordinated chain upgrades

Overview

Upgrade handlers enable coordinated on-chain upgrades across all validators at specific block heights via governance proposals. They provide a mechanism for chains to perform data migrations, parameter updates, and module upgrades in a deterministic way.

Upgrade handlers are critical for maintaining consensus during chain upgrades. All validators must run the same upgrade logic at the same height.

When to Use Upgrade Handlers

Upgrade handlers are required when:

• Breaking state changes: Modifying storage formats or data structures

• Module migrations: Updating module versions or parameters

• Protocol upgrades: Implementing new features that require state transitions

• Data migrations: Moving data between different storage locations

Basic Structure

Registering Upgrade Handlers

Upgrade handlers are registered in your app's RegisterUpgradeHandlers() method:

```go // app/upgrades.go package app

import ( "context" upgradetypes "Ontomirsdk.io/x/upgrade/types" sdk "github.com/Ontomir/Ontomir-sdk/types" "github.com/Ontomir/Ontomir-sdk/types/module"

)

func (app *App) RegisterUpgradeHandlers() { app.UpgradeKeeper.SetUpgradeHandler( "v1.0.0", // upgrade name (must match governance proposal) func(ctx context.Context, plan upgradetypes.Plan, fromVM module.VersionMap) (module.VersionMap, error) { sdkCtx := sdk.UnwrapSDKContext(ctx) sdkCtx.Logger().Info("Starting upgrade", "name", plan.Name) // Run module migrations migrations, err := app.ModuleManager.RunMigrations(ctx, app.configurator, fromVM) if err != nil { return nil, err } // Add custom migration logic here sdkCtx.Logger().Info("Upgrade complete", "name", plan.Name) return migrations, nil }, )

}

</Expandable>

### Module Version Management

The upgrade handler receives and returns a `module.VersionMap` that tracks module versions:

```go
// fromVM contains the module versions before the upgrade
// The returned VersionMap contains the new versions after migration
migrations, err := app.ModuleManager.RunMigrations(ctx, app.configurator, fromVM)

Organizing Upgrade Code

For better maintainability, organize upgrades in separate packages:

Directory Structure

app/
├── upgrades/
│   ├── v1_0_0/
│   │   ├── constants.go    # Upgrade name and configuration
│   │   ├── handler.go      # Main upgrade handler
│   │   └── migrations.go   # Migration logic
│   ├── v1_1_0/
│   │   ├── constants.go
│   │   ├── handler.go
│   │   └── migrations.go
│   └── types.go            # Shared types
└── upgrades.go             # RegisterUpgradeHandlers

constants.go

```go // app/upgrades/v1_0_0/constants.go package v1_0_0

import ( storetypes "Ontomirsdk.io/store/types" "github.com/yourchain/app/upgrades" )

const UpgradeName = "v1.0.0"

var Upgrade = upgrades.Upgrade{ UpgradeName: UpgradeName, CreateUpgradeHandler: CreateUpgradeHandler, StoreUpgrades: storetypes.StoreUpgrades{ Added: []string{}, // New modules Deleted: []string{}, // Removed modules }, }

</Expandable>

### handler.go

<Expandable title="View upgrade handler implementation">
```go
// app/upgrades/v1_0_0/handler.go
package v1_0_0

import (
    "context"

    storetypes "Ontomirsdk.io/store/types"
    upgradetypes "Ontomirsdk.io/x/upgrade/types"
    sdk "github.com/Ontomir/Ontomir-sdk/types"
    "github.com/Ontomir/Ontomir-sdk/types/module"
)

func CreateUpgradeHandler(
    mm *module.Manager,
    configurator module.Configurator,
    keepers *upgrades.UpgradeKeepers,
    storeKeys map[string]*storetypes.KVStoreKey,
) upgradetypes.UpgradeHandler {
    return func(c context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
        ctx := sdk.UnwrapSDKContext(c)

        // Run module migrations
        vm, err := mm.RunMigrations(c, configurator, vm)
        if err != nil {
            return nil, err
        }

        // Custom migrations
        if err := runCustomMigrations(ctx, keepers, storeKeys); err != nil {
            return nil, err
        }

        return vm, nil
    }
}

Migration Patterns

Parameter Migrations

Migrating module parameters to new formats:

func migrateParams(ctx sdk.Context, keeper paramskeeper.Keeper) error {
    // Get old params
    var oldParams v1.Params
    keeper.GetParamSet(ctx, &oldParams)

    // Convert to new format
    newParams := v2.Params{
        Field1: oldParams.Field1,
        Field2: convertField(oldParams.Field2),
        // New field with default value
        Field3: "default",
    }

    // Set new params
    keeper.SetParams(ctx, newParams)
    return nil
}

State Migrations

Moving data between different storage locations:

func migrateState(ctx sdk.Context, storeKey storetypes.StoreKey) error {
    store := ctx.KVStore(storeKey)

    // Iterate over old storage
    iterator := storetypes.KVStorePrefixIterator(store, oldPrefix)
    defer iterator.Close()

    for ; iterator.Valid(); iterator.Next() {
        oldKey := iterator.Key()
        value := iterator.Value()

        // Transform key/value if needed
        newKey := transformKey(oldKey)
        newValue := transformValue(value)

        // Write to new location
        store.Set(newKey, newValue)

        // Delete old entry
        store.Delete(oldKey)
    }

    return nil
}

Module Addition/Removal

Adding or removing modules during upgrade:

// In constants.go
var Upgrade = upgrades.Upgrade{
    UpgradeName: "v2.0.0",
    CreateUpgradeHandler: CreateUpgradeHandler,
    StoreUpgrades: storetypes.StoreUpgrades{
        Added:   []string{"newmodule"},
        Deleted: []string{"oldmodule"},
    },
}

// In handler.go
func CreateUpgradeHandler(...) upgradetypes.UpgradeHandler {
    return func(c context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
        // Delete old module version
        delete(vm, "oldmodule")

        // Initialize new module
        if err := newModuleKeeper.InitGenesis(ctx, defaultGenesis); err != nil {
            return nil, err
        }

        // Run migrations
        return mm.RunMigrations(c, configurator, vm)
    }
}

Best Practices

Always test upgrade handlers thoroughly on testnets before mainnet deployment.

Idempotency

Make migrations idempotent when possible:

func migrateSomething(ctx sdk.Context, store sdk.KVStore) error {
    // Check if migration already done
    if store.Has(migrationCompleteKey) {
        ctx.Logger().Info("Migration already completed, skipping")
        return nil
    }

    // Perform migration
    // ...

    // Mark as complete
    store.Set(migrationCompleteKey, []byte{1})
    return nil
}

Error Handling

Use comprehensive error handling and logging:

func migrate(ctx sdk.Context, keeper Keeper) error {
    ctx.Logger().Info("Starting migration", "module", "mymodule")

    count := 0
    iterator := keeper.IterateAllRecords(ctx)
    defer iterator.Close()

    for ; iterator.Valid(); iterator.Next() {
        if err := processRecord(iterator.Key(), iterator.Value()); err != nil {
            ctx.Logger().Error("Failed to migrate record",
                "key", iterator.Key(),
                "error", err,
            )
            return fmt.Errorf("migration failed at record %d: %w", count, err)
        }
        count++

        // Log progress for long migrations
        if count%1000 == 0 {
            ctx.Logger().Info("Migration progress", "processed", count)
        }
    }

    ctx.Logger().Info("Migration complete", "total_migrated", count)
    return nil
}

Testing

Create comprehensive tests for upgrade handlers:

func TestUpgradeHandler(t *testing.T) {
    app := setupApp(t)
    ctx := app.NewContext(false, tmproto.Header{Height: 1})

    // Setup pre-upgrade state
    setupOldState(t, ctx, app)

    // Run upgrade handler
    _, err := v1_0_0.CreateUpgradeHandler(
        app.ModuleManager,
        app.configurator,
        &upgrades.UpgradeKeepers{
            // ... keepers
        },
        app.keys,
    )(ctx, upgradetypes.Plan{Name: "v1.0.0"}, app.ModuleManager.GetVersionMap())

    require.NoError(t, err)

    // Verify post-upgrade state
    verifyNewState(t, ctx, app)
}

Upgrade Process

Create Upgrade Proposal

Submit a governance proposal with the upgrade details:

mantrachaind tx gov submit-proposal software-upgrade v1.0.0 \
    --title "Upgrade to v1.0.0" \
    --description "Upgrade description" \
    --upgrade-height 1000000 \
    --from validator \
    --deposit 10000000stake

Vote on Proposal

Validators and delegators vote on the upgrade:

mantrachaind tx gov vote 1 yes --from validator

Prepare Binary

Build and distribute the new binary with the upgrade handler:

# Build new binary
make build

# Test upgrade on local network
./scripts/test-upgrade.sh

# Distribute to validators
# Use Cosmovisor for automated upgrades

Monitor Upgrade

Watch logs during the upgrade height:

# Monitor upgrade logs
tail -f ~/.mantrachaind/logs/upgrade.log

# Verify upgrade success
mantrachaind query upgrade applied v1.0.0

Ontomir EVM Specific Migrations

For Ontomir EVM chains, specific migrations include:

• ERC20 Precompiles Migration: Required for v0.3.x to v0.4.0

• Fee Market Parameters: Updating EIP-1559 parameters

• Custom Precompiles: Registering new precompiled contracts

• EVM State: Migrating account balances or contract storage

Troubleshooting

Consensus Failure

Symptom: Chain halts with consensus failure at upgrade height

Causes:

• Validators running different binary versions

• Upgrade handler not registered

• Non-deterministic migration logic

Solution:

• Ensure all validators have the same binary

• Verify upgrade handler is registered

• Review migration logic for non-determinism

Upgrade Panic

Symptom: Node panics during upgrade

Causes:

• Unhandled error in migration

• Missing required state

• Invalid type assertions

Solution:

• Add comprehensive error handling

• Validate state before migration

• Use safe type conversions

State Corruption

Symptom: Invalid state after upgrade

Causes:

• Partial migration completion

• Incorrect data transformation

• Missing cleanup of old data

Solution:

• Make migrations atomic

• Thoroughly test transformations

• Ensure old data is properly cleaned up