hyperswitch
feat(core): add redis lock during insertion of event in event table during initial attempt of outgoing webhook delivery
#7579
Open

feat(core): add redis lock during insertion of event in event table during initial attempt of outgoing webhook delivery #7579

cookieg13 wants to merge 1 commit into main from eventsLocking
cookieg13
cookieg1310 days ago (edited 10 days ago)

Type of Change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring
  • Dependency updates
  • Documentation
  • CI/CD

Description

The current implementation of insertion of an event in event table does not handle the case where multiple requests with the same idempotent_event_id might be processed concurrently, leading to multiple unique constraint violation errors in the database.

This was reported by a merchant who was facing high volumes of unique constraint violation errors.

NOTE: this unique constraint log error issue is only applicable for theinitial attempt of outgoing webhook delivery, not for retry attempts

Why does this happen?

  1. Multiple requests with the same idempotent_event_id might reach the database at the same time.
  2. Each request checks for the event, finds nothing (since it’s not committed yet), and proceeds to insert.
  3. The first request succeeds, while the others fail due to the unique constraint on idempotent_event_id.

Solution (Using Redis Lock)
Before querying the database, acquire a lock in Redis for the given idempotent_event_id.
Only one process/thread will hold the lock, preventing others from inserting at the same time.

If a lock is acquired:

  • Check if the event already exists in the database.
  • If it exists, return early
  • Otherwise, insert the event.
  • Once done, release the lock

Additional Changes

  • This PR modifies the API contract
  • This PR modifies the database schema
  • This PR modifies application configuration/environment variables

Motivation and Context

How did you test it?

Checklist

  • I formatted the code cargo +nightly fmt --all
  • I addressed lints thrown by cargo clippy
  • I reviewed the submitted code
  • I added unit tests for my changes where possible
cookieg13 cookieg13 assigned cookieg13 cookieg13 10 days ago
cookieg13 cookieg13 requested a review 10 days ago
semanticdiff-com
semanticdiff-com10 days ago (edited 10 days ago)

Review changes with  SemanticDiff

Changed Files
File Status
  crates/router/src/core/webhooks/outgoing.rs  1% smaller
cookieg13 cookieg13 added A-core
cookieg13 feat(core): add redis lock during insertion of event in event table d…
5738c1f2
cookieg13 cookieg13 force pushed from 882433c3 to 5738c1f2 10 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
Labels
Milestone