Technical Note: Sei Labs Engineers Discover IBC Indexing Discrepancy

If you would like to be notified about future technical blog posts, product news and developer initiatives, you can sign up to our developer mailing list using the form at the bottom of this article

The Inter-Blockchain Communication (IBC) protocol is a groundbreaking framework designed to facilitate secure and seamless communication between independent blockchains, enabling them to transfer tokens and data with each other. IBC is a complex and challenging technology to work with.

In the past few months, the Sei Labs engineering team observed a discrepancy between the total inflow and outflow of OSMO tokens on the Sei blockchain on several third party services. An investigation and root cause analysis performed by the Sei Labs team identified a likely cause for the issue: the mismatches are due to how these third party services index and interpret data. It is important to note that this was not due to any chain-level issues. 

This blog post will detail how this issue was discovered, investigated and presents two potential solutions to it.

NB The team at Flipside Crypto were informed of this issue after the investigation. They promptly fixed the issue and we are grateful for their cooperation and support.

Discovery

The issue initially came to light through a Flipside report which appeared to show that the total OSMO inflow appeared smaller than the total outflow starting the second day of the launch. This doesn’t make sense - tokens have to be put in to be transacted out, and therefore the total inflow must be equal to or greater than the total outflow. The only way this can legitimately happen is if there’s a bug in the IBC logic on both Sei and Osmosis’s side

The Sei Labs engineering team checked on Mintscan and found a similar, but distinct result. The discrepancies were: 

• Mintscan: ~4917796 more OSMO outflow since launch
• Flipside: ~3219878 more OSMO outflow since launch

In order to isolate the origin of these misalignments, Sei Labs Engineers re-indexed data from the first few days following the Sei Mainnet launch. Their findings confirmed that both inflow and outflow totals align:

(Net_Inflow_18520000_to_19300000 - Net_Outflow_18520000_to_19300000) + Bank Supply of IBC Uosmo on height 18520000 = total supply of uosmo at height 19300000.

In this equation: Net_Inflow_18520000_to_19300000 = 289607886926903
Net_Outflow_18520000_to_19300000 = 288034713858306 
Bank Supply of IBC Uosmo on height 18520000 = 20685000 
Bank supply of uosmo at height 19300000 = 1573193753597

Investigation

The Sei Labs engineers had now discovered the issue of seemingly misaligned inflow and outflow totals on two different third party data providers, and conducted a root cause analysis.

Having confirmed that the on-chain totals did indeed match properly as expected, focus turned to how the third party services were indexing IBC data. Flipside have open sourced their data models on GitHub, so this was a good place to start.

The Flipside data model computed IBC outflow by observing ‘ibc_transfer’ events on the blockchain and IBC inflow by observing ‘write_acknowledgement’ msg and the ‘packet_data’ attribute.

Let’s take a look at how successful IBC transfers play out:

1. Packet Creation on Chain A (Source Chain):
   1. A user on Chain A initiates a token transfer to Chain B.
   2. This action results in an IBC packet being created on Chain A, representing the token transfer. The tokens on Chain A are held in the IBC escrow account, ensuring they can't be double-spent. To be noted here each channel corresponds to a unique escrow account, for example the escrow account osmo<>sei channel is different from that of the axelar<>sei channel.
   3. Transaction #1: 1 transaction on Chain A to initiate the transfer.
2. Relaying the Packet to Chain B:
   1. A relayer notices the new packet on Chain A and submits a transaction to Chain B, relaying the packet data.
   2. Transaction #2: 1 transaction on Chain B to receive the packet.
3. Processing the Packet on Chain B (Destination Chain):
   1. Chain B processes the packet, mints the equivalent tokens on Chain B for the intended recipient, and then generates an acknowledgment.
4. Relaying the Acknowledgment to Chain A:
   1. A relayer notices the acknowledgment on Chain B and submits a transaction to Chain A to relay this acknowledgment.
   2. Transaction #3: 1 transaction on Chain A to process the acknowledgment.

So, if all the ibc_transfer messages are recorded, and all the transactions proceed as outlined above, then the totals match up, right? In theory, yes.

In practice, there's a spanner in the works: timeouts. Some IBC transfers are created with a timeout parameter, which can be specified as a block height, timestamp, or sometimes both. After this time or block height, the IBC packet will be considered timed out… so what happens next?

If a relayer attempts to relay a timed out IBC packet to Chain B’s IBC module, that module will return a timeout error. The relayer informs the sending module on Chain A that the packet has timed out (with a TimeoutPacket). The tokens which were to be transferred are sent back to the transfer initiator, and the IBC transfer has failed. 

Now that you have seen the lifecycle of an IBC transfer transaction, and learned about how these can sometimes fail due to timing out, you can understand the likely cause of the IBC inflow/outflow discrepancies observed in the data. The Sei Labs engineering team looked back at the transaction data from the days following the launch of the Sei blockchain. There were indeed frequent instances where outbound IBC transfers timed out. 

The third party data indexers were counting every ibc_transfer event towards IBC outflows, even if that transfer subsequently timed out - even if a transfer transaction is initially successful on the source chain, there remains a possibility of it timing out during the relaying period. The total IBC outflow calculated by these data providers therefore represented successful IBC transfers, but also those which had timed out. This is how the IBC outflow was greater than the inflow.

Proposed Solution

Instead of processing IBC transfer events to get inflow / outflow amount, the Sei Labs engineering team have proposed to count the total amount of non-native tokens transferred in or out from the IBC module account, as it accounts for all scenarios described above (i.e. transfer in, transfer out and timeout then refund).

They have provided the following code snippet to parse the amount with this approach:

// IBC Transactions: Check tx event for event type 'transfer' between ibc-transfer module account and sei account
// Example: {"type":"transfer","attributes":
// [{"key":"recipient","value":"sei1z3g0ccd0gpe6m4afjjmtxka269yyrymhwwvf3x"},
//   {"key":"sender","value":"sei1yl6hdjhmkf37639730gffanpzndzdpmhrn8l3z"}, 
// {"key":"amount","value":"7898600ibc/2CC0B1B7A981ACC748547..."}]}
func ParseBridgeTx(txResponse *sdktypes.TxResponse, chainId string, msgTypes []string) ([]types.RawBridgeTx, error) {

	// txResponse here is the result the same as `seid q txs --events`
	for _, log := range txResponse.Logs {
		for _, event := range log.Events {
...
			if event.Type != transferEventType {
				continue
			}
...
			amountCoins, err := sdktypes.ParseCoinsNormalized(amount)
			for _, amountCoin := range amountCoins {

				// Case: IBC Transfer Bridge In or IBC Timeout Refund
				if sender == IBCTransferModuleAccount {
					bridgeUserAddress = recipient
					bridgeAddress = sender
					bridgeTxType = IBCTransferBridgeInType
				}

				// Case: IBC Transfer Bridge Out
				if recipient == IBCTransferModuleAccount {
					bridgeUserAddress = sender
					bridgeAddress = recipient
					bridgeTxType = IBCTransferBridgeOutType
				}
...
			}
		}
	}

Indexing the inflow and outflow amount of native tokens on the source chain works with the same approach as the non-native token indexing. The only subtle difference here is instead of counting the transfer information from the IBC module account, you want to count it from the specific IBC escrow account for your channel.

Conclusion

Throughout this report, we have elucidated the complex nuances of the Inter-Blockchain Communication (IBC) protocol and its intricate inflow and outflow indexing processes. 

Anomalies in the indexing, as evidenced by the discrepancy observed in the Flipside dashboard, can lead to misconceptions within the community. It's imperative that all stakeholders are aware of the meticulousness required when working with IBC, particularly during moments of high network congestion or when encountering unexpected scenarios like packet timeouts.

The Sei Labs Engineering team wants to express their gratitude to all community members and stakeholders for their patience and trust. The team is fully equipped and committed to ensuring the transparency, accuracy, and efficiency of all IBC transactions on our chain. We remain steadfast in our dedication to fostering trust and upholding the integrity of the IBC ecosystem.

If you would like to be notified about future technical blog posts, product news and developer initiatives, you can sign up to our developer mailing list using the form below (current members will not see this option) :