PR #7905 feat(router): adding generic tokenization endpoint

su-shivanshmathur37 days ago (edited 10 days ago)

Type of Change

Description

Create a new set of endpoints for giving tokenization as a service for generic usage, current motivation of the service is to be used in proxy payment service.

Additional Changes

This PR modifies the API contract
This PR modifies the database schema
This PR modifies application configuration/environment variables

Motivation and Context

How did you test it?

Create a token

curl --location 'http: //localhost:8080/v2/tokenize' \
--header 'x-profile-id: pro_neU1Qn8gnIich0LAaQXm' \
--header 'Authorization: publishable-key=PUBLISHABLE_KEY,client-secret=CLIENT_SECRET' \
--header 'Content-Type: application/json' \
--data '{
    "customer_id": customer_id,
    "token_request": {
        "payment_method_data": {
            "card": {
                "card_holder_name": "Narayan Bhat"
            }
        }
    }
}'

Response

{
    "id": "12345_tok_01966c6515157c3193db1c2502435ad5",
    "created_at": "2025-04-25 10:01:26.037519",
    "flag": "enabled"
}

Checklist

I formatted the code cargo +nightly fmt --all
I addressed lints thrown by cargo clippy
I reviewed the submitted code
I added unit tests for my changes where possible

inital commit

2a1384c3

updates

8a1ffeff

GlobalTokenId intoduce

59027b8e

Changes

4076f3cb

model changes

b50f333c

insert changes

967ab76f

Changes

4cb5a50b

Working changes

c3681e1c

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

96d98c7c

Working commits

84859456

Sanity

b4d4435f

Retrieve endpoint for tokenization

de7bc0f2

development.toml changes

d65dd5d1

migrations

dcdc5d9b

flag check

978ca250

su-shivanshmathur assigned

su-shivanshmathur 37 days ago

su-shivanshmathur requested a review 37 days ago

semanticdiff-com37 days ago (edited 10 days ago)

Review changes with

Changed Files

File	Status
crates/router/src/db.rs	80% smaller
crates/router/src/types/payment_methods.rs	78% smaller
crates/diesel_models/src/enums.rs	41% smaller
crates/hyperswitch_domain_models/src/payment_methods.rs	22% smaller
crates/router/src/core/payment_methods.rs	1% smaller
crates/common_enums/src/enums.rs	1% smaller
api-reference-v2/api-reference/tokenization/tokenization--create.mdx	Unsupported file format
api-reference-v2/mint.json	0% smaller
api-reference-v2/openapi_spec.json	0% smaller
config/development.toml	Unsupported file format
crates/api_models/Cargo.toml	Unsupported file format
crates/api_models/src/events.rs	0% smaller
crates/api_models/src/lib.rs	0% smaller
crates/api_models/src/payment_methods.rs	0% smaller
crates/api_models/src/tokenization.rs	0% smaller
crates/common_types/Cargo.toml	Unsupported file format
crates/common_utils/Cargo.toml	Unsupported file format
crates/common_utils/src/consts.rs	0% smaller
crates/common_utils/src/events.rs	0% smaller
crates/common_utils/src/id_type.rs	0% smaller
crates/common_utils/src/id_type/global_id.rs	0% smaller
crates/common_utils/src/id_type/global_id/token.rs	0% smaller
crates/common_utils/src/lib.rs	0% smaller
crates/common_utils/src/tokenization.rs	0% smaller
crates/diesel_models/Cargo.toml	Unsupported file format
crates/diesel_models/src/lib.rs	0% smaller
crates/diesel_models/src/mod.rs	0% smaller
crates/diesel_models/src/payment_methods_session.rs	0% smaller
crates/diesel_models/src/query.rs	0% smaller
crates/diesel_models/src/query/tokenization.rs	0% smaller
crates/diesel_models/src/schema_v2.rs	0% smaller
crates/diesel_models/src/tokenization.rs	0% smaller
crates/diesel_models/src/types.rs	0% smaller
crates/hyperswitch_domain_models/Cargo.toml	Unsupported file format
crates/hyperswitch_domain_models/src/lib.rs	0% smaller
crates/hyperswitch_domain_models/src/tokenization.rs	0% smaller
crates/masking/src/lib.rs	0% smaller
crates/masking/src/secret.rs	0% smaller
crates/openapi/Cargo.toml	Unsupported file format
crates/openapi/src/openapi_v2.rs	0% smaller
crates/openapi/src/routes.rs	0% smaller
crates/openapi/src/routes/tokenization.rs	0% smaller
crates/router/Cargo.toml	Unsupported file format
crates/router/src/core.rs	0% smaller
crates/router/src/core/payment_methods/transformers.rs	0% smaller
crates/router/src/core/payment_methods/vault.rs	0% smaller
crates/router/src/core/tokenization.rs	0% smaller
crates/router/src/db/kafka_store.rs	0% smaller
crates/router/src/lib.rs	0% smaller
crates/router/src/routes.rs	0% smaller
crates/router/src/routes/app.rs	0% smaller
crates/router/src/routes/lock_utils.rs	0% smaller
crates/router/src/routes/tokenization.rs	0% smaller
crates/router_env/src/logger/types.rs	0% smaller
crates/storage_impl/Cargo.toml	Unsupported file format
crates/storage_impl/src/lib.rs	0% smaller
crates/storage_impl/src/mock_db.rs	0% smaller
crates/storage_impl/src/tokenization.rs	0% smaller
v2_migrations/2025-04-25-105138_tokenization_service_creation/down.sql	Unsupported file format
v2_migrations/2025-04-25-105138_tokenization_service_creation/up.sql	Unsupported file format

hyperswitch-bot added M-database-changes

chore: run formatter

6141225a

clippy errors

e8b43012

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

87765707

Conflict resolve

78615649

chore: run formatter

1d33d0c9

clippy changes

84ee28c6

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

3e9937a2

chore: run formatter

f817c02c

clippy error

428ac804

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

d3a6717f

chore: run formatter

46460edd

Changes

23eb33d0

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

219f9d72

Clippy changes

096dbf79

chore: run formatter

27c74f23

schema changes

cbf61a2a

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

2f11f33f

changes in down.sql

a6bd4b21

storage_impl tokenization file removal

71442c8c

chore: run formatter

08923448

jarnura requested changes on 2025-04-29

config/development.toml

69	69	applepay_endpoint = "DOMAIN SPECIFIC ENDPOINT"
70	70
71	71	[locker]
72		host = "http://127.0.0.1:3000"
73		host_rs = "http://127.0.0.1:3000"
	72	host = "http://127.0.0.1:3001"

jarnura33 days ago

why we are changing these ports?

su-shivanshmathur32 days ago

Because these are the ones which are being used in card-vault

jarnura30 days ago

can we change the port in card-vault, i don't know that the places we need to change(basically documentations and other places) this port.

crates/router/src/core/tokenization.rs

	160		}
	161
	162		#[cfg(all(feature = "v2", feature = "tokenization_v2"))]
	163		fn mask_sensitive_data(value: serde_json::Value) -> serde_json::Value {

jarnura32 days ago

why we are using masking like this? there is secret which does the same right?

su-shivanshmathur32 days ago

The function is needed for masking on all the keys present but yes the masking value could be used from the secret thing

usage of this is if I just want the key structure which I would have so that could create the proxy payment request in that format

jarnura30 days ago

We can introduce a masking strategy which does the same and using masking itself. So that there will be only one approach for masking

development.toml changes

9926d7b4

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

1ace00e3

maverox dismissed these changes on 2025-05-02

prasunna09 commented on 2025-04-30

config/development.toml

84	84	redis_ttl_in_seconds = 172800
85	85
86	86	[jwekey]
87		vault_encryption_key = """
88		-----BEGIN PUBLIC KEY-----
89		MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwa6siKaSYqD1o4J3AbHq
90		Km8oVTvep7GoN/C45qY60C7DO72H1O7Ujt6ZsSiK83EyI0CaUg3ORPS3ayobFNmu
91		zR366ckK8GIf3BG7sVI6u/9751z4OvBHZMM9JFWa7Bx/RCPQ8aeM+iJoqf9auuQm
92		3NCTlfaZJif45pShswR+xuZTR/bqnsOSP/MFROI9ch0NE7KRogy0tvrZe21lP24i
93		Ro2LJJG+bYshxBddhxQf2ryJ85+/Trxdu16PunodGzCl6EMT3bvb4ZC41i15omqU
94		aXXV1Z1wYUhlsO0jyd1bVvjyuE/KE1TbBS0gfR/RkacODmmE2zEdZ0EyyiXwqkmc
95		oQIDAQAB
96		-----END PUBLIC KEY-----
97		"""
98		rust_locker_encryption_key = """
99		-----BEGIN PUBLIC KEY-----
100		MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwa6siKaSYqD1o4J3AbHq
101		Km8oVTvep7GoN/C45qY60C7DO72H1O7Ujt6ZsSiK83EyI0CaUg3ORPS3ayobFNmu
102		zR366ckK8GIf3BG7sVI6u/9751z4OvBHZMM9JFWa7Bx/RCPQ8aeM+iJoqf9auuQm
103		3NCTlfaZJif45pShswR+xuZTR/bqnsOSP/MFROI9ch0NE7KRogy0tvrZe21lP24i
104		Ro2LJJG+bYshxBddhxQf2ryJ85+/Trxdu16PunodGzCl6EMT3bvb4ZC41i15omqU
105		aXXV1Z1wYUhlsO0jyd1bVvjyuE/KE1TbBS0gfR/RkacODmmE2zEdZ0EyyiXwqkmc
106		oQIDAQAB
107		-----END PUBLIC KEY-----
108		"""
	87	vault_encryption_key = ""
	88	rust_locker_encryption_key = ""

prasunna0932 days ago

why did we remove these keys? is this intentional?

su-shivanshmathur14 days ago

Yes

Payment method session integration for tokenization service

5ff31ee5

su-shivanshmathur dismissed their stale review via 5ff31ee5 26 days ago

chore: run formatter

b53a889f

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

a3d7559e

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

1625353d

Added client secret based auth

4b20cfee

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

448de98c

chore: run formatter

14a26f02

jarnura requested changes on 2025-05-14

Conversation is marked as resolved

Show resolved

jarnura requested changes on 2025-05-14

Conversation is marked as resolved

Show resolved

crates/storage_impl/src/lib.rs

	510		fn unique_constraints(&self) -> Vec<String> {
	511		vec![
	512		format!("id_{}", self.id.get_string_repr()),
	513		format!("locker_id_{}", self.locker_id),

jarnura18 days ago

Global id itself will be a unique key right, is the composite unqiue key is required here

su-shivanshmathur18 days ago

I hope its fine now ?

crates/router/src/db/tokenization.rs

1

#[cfg(all(feature = "v2", feature = "tokenization_v2"))]

jarnura18 days ago

move this to storage impl instead of keeping this in router, for can refer other table in storage_impl crate for reference

masking data and adding storage_impl

1fed1010

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

7e55476c

Comment resolution

a9993afa

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

9ec96398

jarnura requested changes on 2025-05-15

storage_imple usage inplace of db

d48cb21f

chore: run formatter

4d1ec7d8

JSON mask startegy introduce

2d2e0a20

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

e459fbeb

chore: run formatter

a1006d34

adding unit test case

a3c39153

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

6a617d03

chore: run formatter

749221b5

test case changes

97b19040

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

b0501ca1

chore: run formatter

95042704

prasunna09 requested changes on 2025-05-16

crates/api_models/src/payment_methods.rs

	2850
	2851		/// The json to be used for tokeniation
	2852		#[schema(value_type = Option<serde_json::Value>)]
	2853		pub tokenization_data: Option<masking::Secret<serde_json::Value>>,

prasunna0915 days ago

Suggested change

      
                pub tokenization_data: Option<masking::Secret<serde_json::Value>>,
          
                pub tokenization_data: Option<pii::SecretSerdeValue>,

Conversation is marked as resolved

Show resolved

crates/api_models/src/payment_methods.rs

	2870
	2871		/// The json to be used for tokeniation
	2872		#[schema(value_type = Option<serde_json::Value>)]
	2873		pub tokenization_data: Option<masking::Secret<serde_json::Value>>,

prasunna0915 days ago

please use this pii::SecretSerdeValue and in other places

crates/common_utils/src/tokenization.rs

	13		#[cfg(all(feature = "v2", feature = "tokenization_v2"))]
	14		/// Response structure for tokenization operations
	15		#[derive(Debug, Serialize, Deserialize)]
	16		pub struct TokenizationResponse {
	17		/// Unique identifier for the tokenized data
	18		pub id: GlobalTokenId,
	19		/// Reference to the vault/locker where the actual data is stored
	20		pub locker_id: String,
	21		/// Timestamp when the token was created
	22		pub created_at: PrimitiveDateTime,
	23		/// Timestamp when the token was last updated
	24		pub updated_at: PrimitiveDateTime,
	25		/// Current status of the token
	26		pub flag: TokenizationFlag,

prasunna0915 days ago

why do we need this tokenization.rs in common_utils crate?
tokenization types are present both in api, domain, and diesel, why do we need this here?

crates/common_utils/src/tokenization.rs

	40		/// Enum representing the status of a tokenized payment method
	41		#[derive(Debug, Clone, Serialize, Deserialize, strum::Display, strum::EnumString)]
	42		#[strum(serialize_all = "snake_case")]
	43		pub enum TokenizationFlag {
	44		/// Token is active and can be used for payments
	45		Enabled,
	46		/// Token is inactive and cannot be used for payments
	47		Disabled,

prasunna0915 days ago

why do we need this here? we have already added in common enums ryt?

crates/diesel_models/src/tokenization.rs

	67		}
	68
	69		#[cfg(all(feature = "v2", feature = "tokenization_v2"))]
	70		impl Tokenization {
	71		pub async fn insert(self, conn: &PgPooledConn) -> StorageResult<Self> {
	72		generics::generic_insert(conn, self).await
	73		}
	74		}

prasunna0915 days ago

this too.

su-shivanshmathur15 days ago

Because this is there in storage_impl
That's why?

Comment and clippy changes

82556934

clippy errors

006e520e

common imports

dc4b1ac6

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

27c652f5

chore: run formatter

bd87f89f

Comments

fde74444

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

b33d18d8

openapi changes

3bb4e0b1

hyperswitch-bot added M-api-contract-changes

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

58a1d673

su-shivanshmathur requested a review 13 days ago

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

90b6050a

remove the api for get token

1c562782

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

2051cc4f

Format error

8be1cfa1

chore: run formatter

5da9217c

Removing unwanted imports and enums

89088e2b

JsonMaskStrategy under serde feature flag

b6808b16

chore: run formatter

810ae36f

jarnura dismissed these changes on 2025-05-22

maverox dismissed these changes on 2025-05-22

tsdk02 dismissed these changes on 2025-05-22

Comment changes and OpenApi spec changes

a3baaf40

su-shivanshmathur dismissed their stale review via a3baaf40 10 days ago

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

54cf6d35

tsdk02 dismissed these changes on 2025-05-22

maverox dismissed these changes on 2025-05-22

chore: run formatter

cfc23d18

hyperswitch-bot dismissed their stale review via cfc23d18 10 days ago

tsdk02 dismissed these changes on 2025-05-22

Merge branch 'main' of github.com:juspay/hyperswitch into feature/tok…

3daa21f8

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

ea3aee5a

su-shivanshmathur dismissed their stale review via ea3aee5a 10 days ago

prasunna09 dismissed these changes on 2025-05-22

Conversation is marked as resolved

Show resolved

crates/router/src/core/tokenization.rs

	108
	109		#[instrument(skip_all)]
	110		#[cfg(all(feature = "v2", feature = "tokenization_v2"))]
	111		pub async fn get_token_vault_core(

prasunna0910 days ago

nit: this name could be better.

ps: not able to get the functionality from the name.

su-shivanshmathur10 days ago

Can take up later when we'll use this for a handler fuction

Conversation is marked as resolved

Show resolved

Conversation is marked as resolved

Show resolved

jarnura dismissed these changes on 2025-05-22

Comment resolution

332f7f1f

su-shivanshmathur dismissed their stale review via 332f7f1f 10 days ago

chore: run formatter

bf87db45

OpenApi Changes

3f5fd1bb

open-api changes

874ea798

Merge branch 'feature/tokenize' of github.com:juspay/hyperswitch into…

daa0ea90

tsdk02 approved these changes on 2025-05-22

jarnura approved these changes on 2025-05-22

prasunna09 approved these changes on 2025-05-22

crates/router/src/core/tokenization.rs

	18		use error_stack::ResultExt;
	19		#[cfg(all(feature = "v2", feature = "tokenization_v2"))]
	20		use hyperswitch_domain_models;
	21		#[cfg(all(feature = "v2", feature = "tokenization_v2"))]

prasunna099 days ago

nit: instead of having this on every import n method, this can be moved to parent level. core.rs in the same crate.

crates/router/src/routes/tokenization.rs

	52		\|state, auth: auth::AuthenticationData, request, _\| async move {
	53		tokenization::create_vault_token_core(
	54		state,
	55		&auth.merchant_account,

prasunna099 days ago

merchantContext has been introduced recently. we should use that ig

CC: @maverox

likhinbopanna merged 49a178ed into main 5 days ago

likhinbopanna deleted the feature/tokenize branch 5 days ago

	7		customer_id VARCHAR(64) NOT NULL,
	8		created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
	9		updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
	10		locker_id VARCHAR(255) NOT NULL,

	307		// Clone,
	308		// Copy,
	309		// Debug,
	310		// Eq,

hyperswitch
feat(router): adding generic tokenization endpoint
#7905

Merged

feat(router): adding generic tokenization endpoint #7905

Type of Change

Description

Additional Changes

Motivation and Context

How did you test it?

Checklist

	pub tokenization_data: Option<masking::Secret<serde_json::Value>>,
	pub tokenization_data: Option<pii::SecretSerdeValue>,

	28		#[derive(Debug, Serialize, Deserialize, ToSchema)]
	29		pub struct GenericTokenizationRequest {
	30		#[schema(value_type = String, example = "12345_cus_01926c58bc6e77c09e809964e72af8c8")]
	31		pub customer_id: GlobalCustomerId,

	96		{
	97		let conn = connection::pg_connection_read(self).await?;
	98
	99		// Use the find_by_id method we just defined

hyperswitch feat(router): adding generic tokenization endpoint #7905 Merged

feat(router): adding generic tokenization endpoint #7905

Type of Change

Description

Additional Changes

Motivation and Context

How did you test it?

Checklist

hyperswitch
feat(router): adding generic tokenization endpoint
#7905

Merged