Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for dynamic renaming of keys #5074

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

divbok
Copy link

@divbok divbok commented Oct 16, 2024

Description

It includes support for dynamic renaming of keys in rename processor where the user can now give a regex pattern to match the keys that needs to be replaced.

Issues Resolved

Resolves #4849

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Collaborator

@oeyh oeyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! A few minor issues in addition to David's comments.
Also, not blocking this PR, have you got a chance to take a look at adding the to_key_pattern option as well? It would be great value added. We can add it in a separate PR if it's straightforward to do.

@divbok
Copy link
Author

divbok commented Oct 16, 2024

@oeyh Haven't got a chance yet to look at the to_key_pattern. I can analyse the complexity and see how it goes. Will raise a separate PR if it is trivial.

import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.*;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build is failing due to this. Imports need to be explicit.

Comment on lines 85 to 86
if (fromKeyPattern != null) {
fromKeyCompiledPattern = Pattern.compile(fromKeyPattern);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work when Data Prepper reads config from a yaml file. This constructor is not called. We probably want to move it to getFromKeyCompiledPattern() method.

        public Pattern getFromKeyCompiledPattern() {
            if (fromKeyPattern != null && fromKeyCompiledPattern == null) {
                fromKeyCompiledPattern = Pattern.compile(fromKeyPattern);
            }
            return fromKeyCompiledPattern;
        }

@divbok divbok force-pushed the rename_processor branch 2 times, most recently from 1f76593 to 5225f66 Compare October 21, 2024 19:45
@@ -161,7 +161,8 @@ private static boolean isValidKey(final String key) {
|| c == '@'
|| c == '/'
|| c == '['
|| c == ']')) {
|| c == ']'
|| c == '|')) {
Copy link
Collaborator

@oeyh oeyh Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlvenable Are you OK with this approach? This change is for it to work with the example data provided in #4849, where "|" appears in field keys. Without it, the processor fails to delete the original field due to invalid char "|" in the key with a config like this:

    - rename_keys:
        entries:
          - from_key_pattern: "delivered_at.+"
            to_key: "delivered_at"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline. We agree on skipping the validation for the delete method.

@divbok divbok requested a review from sb2k16 as a code owner October 29, 2024 17:51

public Pattern getFromKeyCompiledPattern() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to add @JsonIgnore to this method.

Signed-off-by: Divyansh Bokadia <[email protected]>

Dynamic renaming of keys

Signed-off-by: Divyansh Bokadia <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Data Prepper support for dynamic renaming of keys
3 participants