Load Compliant Data with Airbyte Mappings

When it comes to modern data integration, securing and managing sensitive data isn't just about ticking compliance boxes—it's about maintaining customer trust while keeping your data usable and organized. Whether you're moving data between regions, migrating to the cloud, or connecting on-premise systems, ensuring data compliance and security throughout the journey is crucial.

Airbyte now introduces a powerful solution: Mappings. This comprehensive suite of features lets you:

• Hash sensitive data for one-way protection
• Encrypt fields that need secure access later
• Rename fields for clarity and consistency
• Filter rows to control exactly what data moves

Available across Airbyte's entire catalog of 550+ connectors, Mappings transform your data while it moves, ensuring compliance and security without sacrificing usability. Let's explore how these features work together to protect your data pipeline.

To get started, open the Mappings tab for in your workspace for any Airbyte connection. We demonstrate Mappings with an example connection moving customer data from an on-premise Oracle database to a cloud Databricks instance running in AWS.

Flexible Hashing: Protecting Sensitive Data When Needed

When you need to secure sensitive information, Airbyte's hashing capabilities let you configure exactly which fields to protect. For example, if you set up hashing for a customer ID field like C_D_ID, Airbyte will transform values using your chosen hashing algorithm - turning them into secure hashes. The unique mapping from source to target secure hash is the primary benefit of hashing over traditional masking or column blocking. If two matching IDs from separate datasets are hashed with the same hashing algorithm, we will be able to join datasets on hashed customer IDs in downstream pipelines.

Airbyte supports multiple hashing algorithms:

• MD5 for legacy systems
• SHA-256 for current security standards (shown in example below)
• SHA-512 for enhanced security requirements

Setting up hashing is straightforward: select your target field, choose your hashing algorithm from a dropdown, and you're ready to go. This optional transformation gives you control over which data needs this level of protection.

RSA Encryption: Protecting Sensitive Data While Maintaining Access

When dealing with sensitive information like phone numbers that you'll need to access later, RSA encryption provides the perfect balance of security and usability. Unlike hashing, which is a one-way process, RSA encryption is reversible - meaning you can decrypt the data when needed. For example, a marketing analyst may need to download extracts from Databricks onto a secure machine, then decrypt phone numbers to launch a marketing campaign.

Here, we see C_PHONE being encrypted using RSA. This asymmetric encryption approach uses a public key for encryption (what you configure in Airbyte) and a private key for decryption (kept secure for authorized access).

Our example shows this in action:

• Original phone numbers get encrypted using your public key
• The encrypted values appear in your destination as secure strings
• Using the corresponding private key, authorized systems can later decrypt and access the original phone numbers

This ensures your sensitive data remains protected throughout the pipeline while remaining accessible to authorized users with the proper decryption keys.

Why These Features Work Together

This integrated approach to data security and control delivers more than just individual features. By combining these capabilities in one platform, you can:

• Meet GDPR and HIPAA requirements while maintaining data usability
• Protect sensitive information without complex security configurations
• Standardize data handling across all your connectors
• Turn complex compliance requirements into simple, repeatable processes

Making Your Data Work for You

Gone are the days of reverse engineering cryptic field names to rediscover their original meaning. Looking at Airbyte's interface, we can see field renaming in action—transforming unclear names like C_LAST into more readable formats like LAST_NAME. This helps ensure your data is more accessible and meaningful across your entire organization.

Advanced Row Filtering

Looking at our customer data stream, imagine you only want to process records from specific states. Airbyte makes this precise data selection straightforward. In the interface, we can see a real-world example: filtering customer records where C_STATE equals "NY".

More than just simple filtering, the interface reveals the true power of Airbyte's filtering capabilities:

• Filter based on string and numerical values
• Remove irrelevant data entries
• Optimize storage by filtering out unnecessary data
• Maintain compliance by excluding specific data points

Let's look at our example in detail. The customer table shows various state codes (EA, SP, AQ...), and with a simple filter condition (to exclude the state NY), we can precisely control which records flow through our pipeline. This isn't just about reducing data volume—it's about ensuring only the most relevant data makes it to your destination.

The Airbyte Advantage: From Features to Real-World Impact

What sets Airbyte's mapping implementation apart is its unified approach to data security and control. Through a single, intuitive interface, you can:

• Define all your mapping rules in one place
• Apply multiple security measures simultaneously
• Maintain data integrity throughout the process
• Configure mappings through simple dropdown menus
• Manage mappings at the stream level for precise control

This streamlined approach transforms complex data security requirements into manageable tasks. Whether you're hashing sensitive customer IDs, encrypting phone numbers, standardizing field names, or filtering state-specific data, everything happens in one cohesive workflow. No jumping between tools, no complex configurations—just straightforward, powerful data control.

The real power lies in this integration: you can secure, standardize, and filter your data all within a single mapping configuration. For organizations dealing with sensitive information or needing to maintain strict data standards, this means faster implementation, better compliance, and more reliable data pipelines. This is exactly what makes Airbyte's mapping features a game-changer in the data integration space.

Stay tuned for more cohesive product updates in our recent 1.5.0 release!

Suggested Read

Data Mapping