Add custom enum deserializer to improve error messaging, improve byte count error mesages #5076

graytaylor0 · 2024-10-16T17:36:15Z

Description

Currently when enums are deserialized with an invalid value the error will say that the value must not be null, because the enums we use have a @JsonCreator that looks up the options in the map and returns null.

This adds a custom enum deserializer that will give a clearer message and provide the valid options to the user.

Also, the ByteCount error messages were improved to provide valid options and show which field is invalid.

New error message for invalid enum values

entry-pipeline.processor.parse_json: caused by: Parameter "handle_failed_events" for plugin "parse_json" is invalid: Invalid value "skip_silentl". Valid options include [drop, drop_silently, skip, skip_silently].

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

san81 · 2024-10-17T16:45:53Z

...gin-framework/src/main/java/org/opensearch/dataprepper/plugin/ObjectMapperConfiguration.java

@@ -33,6 +34,7 @@ public class ObjectMapperConfiguration {
    ObjectMapper extensionPluginConfigObjectMapper() {
        final SimpleModule simpleModule = new SimpleModule();
        simpleModule.addDeserializer(Duration.class, new DataPrepperDurationDeserializer());
+        simpleModule.addDeserializer(Enum.class, new EnumDeserializer());


This will be applied to every Enum defined in the project?

Yes it will be

Any configuration file that has a variable that is an Enum

san81

Super helpful fix for the customers go understand the error message along with listing out possible values for them to fix. 👍

san81 · 2024-10-17T16:49:46Z

...sor/src/main/java/org/opensearch/dataprepper/plugins/processor/drop/DropEventsProcessor.java

@@ -36,7 +36,8 @@ public DropEventsProcessor(

        if (dropEventProcessorConfig.getDropWhen() != null &&
                (!expressionEvaluator.isValidExpressionStatement(dropEventProcessorConfig.getDropWhen()))) {
-            throw new InvalidPluginConfigurationException("drop_when {} is not a valid expression statement. See https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/ for valid expression syntax");
+            throw new InvalidPluginConfigurationException(String.format("drop_when \"%s\" is not a valid expression statement. " +
+                    "See https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/ for valid expression syntax", dropEventProcessorConfig.getDropWhen()));


Not a concern but just helpful advice for any possible future refactoring needs. What if the documentation url path changes in the future? May be better to reference these urls from a single constants file or something like that.

That's a good suggestion we should do that. We have these in other locations as well like the @JsonPropertyDescriptions

san81 · 2024-10-17T16:57:44Z

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/types/ByteCount.java

@@ -79,11 +79,13 @@ public static ByteCount parse(final String string) {
        final String unitString = matcher.group("unit");

        if(unitString == null) {
-            throw new ByteCountInvalidInputException("Byte counts must have a unit.");
+            throw new ByteCountInvalidInputException("Byte counts must have a unit. Valid byte units include: " +


I think, it is better to enhance the corresponding test case messages to assert for this updated message. Currently, they are only looking for not null.

I'll update the test case to assert for this message

san81 · 2024-10-17T17:05:40Z

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/types/ByteCount.java

        }

        final Unit unit = Unit.fromString(unitString)
-                .orElseThrow(() -> new ByteCountInvalidInputException("Invalid byte unit: '" + unitString + "'"));
+                .orElseThrow(() -> new ByteCountInvalidInputException("Invalid byte unit: '" + unitString + "'. Valid byte units include: "
+                        + Arrays.stream(Unit.values()).map(unitValue -> unitValue.unitString).collect(Collectors.toList())));


Probably a nit. This Arrays.stream statement repeated twice. It is also there in line number 83. I think you can add a method in Unit enum class to return this array of code values to keep the logic at one place.

Line 51 already does something similar, though it creates a map which will be out of order. If you make it a linked map you could use the keys() from it.

data-prepper/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/types/ByteCount.java

Lines 51 to 52 in c651aaa

private static final Map<String, Unit> UNIT_MAP = Arrays.stream(Unit.values())

.collect(Collectors.toMap(unit -> unit.unitString, Function.identity()));

dlvenable · 2024-10-18T00:16:45Z

...peline-parser/src/main/java/org/opensearch/dataprepper/pipeline/parser/EnumDeserializer.java

+        final String enumValue = node.asText();
+
+        for (Object enumConstant : enumClass.getEnumConstants()) {
+            if (enumConstant.toString().equalsIgnoreCase(enumValue)) {


I don't think this is the right approach. We should be sticking with a single case to avoid multiple possible values for the same thing.

Rather, use the core Jackson deserializer to get the value. Or, use @JsonCreator which is what it uses.

By the way, I think that is fine to make something like enumConstant.toString().toLowerCase().equals(enumValue) as a default behavior. But, we should still support @JsonValue and @JsonCreator. So, you could have logic like the following pseudo-code:

if(findAnnotedMethod(enumClass, JsonCreator.class)) { return useJsonCreator(enumClass); } else { if( enumConstant.toString().toLowerCase().equals(enumValue) { return enumConstant; } }

Is there a problem with allowing both lowercase and uppercase? I think it is fair to assume that if the user inputs either, this is the intended choice. I have felt like it is not clear which value to put into the YAML before.

We should avoid it because it ends up causing exactly this confusion. So rather than be so loose that anything goes, let's just require the correct input. We can consistently take in lowercase-values which is what our documentation tells people to do.

Also, this is so loose that we will allow aLL-soRTs-of-VALues.

Can JsonValue just be used for this too? Not sure how we would use JsonCreator for this

Right now the enums all use @JsonCreator to determine how to deserialize from JSON into POJO. The @JsonValue determines how to reverse this for the schemas. Maybe @JsonValue can do both, but I'm not sure.

This is the pattern:

data-prepper/data-prepper-plugins/mutate-event-processors/src/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/TargetType.java

Lines 52 to 61 in aff479b

@JsonCreator

static TargetType fromOptionValue(final String option) {

return OPTIONS_MAP.get(DataType.fromTypeName(option));

}

@JsonValue

public String getOptionValue() {

return dataType.getTypeName();

}

dlvenable · 2024-10-18T00:17:11Z

...peline-parser/src/main/java/org/opensearch/dataprepper/pipeline/parser/EnumDeserializer.java

+        }
+
+        throw new IllegalArgumentException(String.format(INVALID_ENUM_VALUE_ERROR_FORMAT, enumValue,
+                Arrays.stream(enumClass.getEnumConstants()).map(valueEnum -> valueEnum.toString().toLowerCase()).collect(Collectors.toList())));


Again, let's not assume these are always the same. Use @JsonValue instead.

Can we assume that enums will always have a method annotated with @JsonValue, or should we fall back to default lowercase behavior if no method is found?

No, we shouldn't assume this. Some may be missing. I think falling back is a good solution when there is no @JsonValue.

… count error mesages Signed-off-by: Taylor Gray <[email protected]>

…tion methods Signed-off-by: Taylor Gray <[email protected]>

dlvenable · 2024-10-22T21:26:14Z

...peline-parser/src/main/java/org/opensearch/dataprepper/pipeline/parser/EnumDeserializer.java

+        final String enumValue = node.asText();
+
+        final Optional<Method> jsonCreator = findJsonCreatorMethod();
+        jsonCreator.ifPresent(method -> method.setAccessible(true));


Ideally, this is in a try-finally block.

try { jsonCreator.ifPresent(method -> method.setAccessible(true)); // for loop } finally { jsonCreator.ifPresent(method -> method.setAccessible(false)); }

dlvenable · 2024-10-22T21:27:12Z

...ne-parser/src/test/java/org/opensearch/dataprepper/pipeline/parser/EnumDeserializerTest.java

+    }
+
+    private enum TestEnum {
+        TEST("test");


I think this test case needs a different name to ensure it works, right?

Suggested change

TEST("test");

TEST("test_display_value");

dlvenable · 2024-10-22T21:28:01Z

...ne-parser/src/test/java/org/opensearch/dataprepper/pipeline/parser/EnumDeserializerTest.java

+        final EnumDeserializer objectUnderTest = new EnumDeserializer();
+        JsonDeserializer<?> result = objectUnderTest.createContextual(context, property);
+
+        assertTrue(result instanceof EnumDeserializer);


Suggested change

assertTrue(result instanceof EnumDeserializer);

assertThat(result, instanceOf(EnumDeserializer.class));

This will output the class type if it fails the assertion.

dlvenable · 2024-10-22T21:29:44Z

...ne-parser/src/test/java/org/opensearch/dataprepper/pipeline/parser/EnumDeserializerTest.java

+    }
+
+    @ParameterizedTest
+    @EnumSource(HandleFailedEventsOption.class)


Let's rely on test classes to avoid changes in one class affecting others. Maybe update TestEnum with more values.

Signed-off-by: Taylor Gray <[email protected]>

dlvenable

Thank you @graytaylor0 !

…ove byte count error mesages (opensearch-project#5076)" This reverts commit e9bffee.

…ove byte count error mesages (opensearch-project#5076)" This reverts commit e9bffee. Signed-off-by: Taylor Gray <[email protected]>

graytaylor0 requested review from chenqi0805, engechas, dinujoh, kkondaka, KarstenSchnitter, dlvenable and oeyh as code owners October 16, 2024 17:36

graytaylor0 force-pushed the EnumAndMiscValidations branch from ec00227 to bf036e6 Compare October 16, 2024 18:49

oeyh previously approved these changes Oct 17, 2024

View reviewed changes

san81 reviewed Oct 17, 2024

View reviewed changes

graytaylor0 dismissed oeyh’s stale review via c651aaa October 17, 2024 20:03

graytaylor0 force-pushed the EnumAndMiscValidations branch from bf036e6 to c651aaa Compare October 17, 2024 20:03

san81 approved these changes Oct 17, 2024

View reviewed changes

dlvenable requested changes Oct 18, 2024

View reviewed changes

graytaylor0 added 2 commits October 21, 2024 13:34

Add custom enum deserializer to improve error messaging, improve byte…

0cefbd0

… count error mesages Signed-off-by: Taylor Gray <[email protected]>

Adjust enum deserializer to look for JsonCreator and JsonValue annota…

99705f9

…tion methods Signed-off-by: Taylor Gray <[email protected]>

graytaylor0 force-pushed the EnumAndMiscValidations branch from c651aaa to 99705f9 Compare October 22, 2024 17:51

graytaylor0 requested a review from dlvenable October 22, 2024 17:51

dlvenable requested changes Oct 22, 2024

View reviewed changes

Address PR comments

9c71a0d

Signed-off-by: Taylor Gray <[email protected]>

graytaylor0 requested a review from sb2k16 as a code owner October 22, 2024 22:13

graytaylor0 requested a review from dlvenable October 22, 2024 22:14

dlvenable approved these changes Oct 23, 2024

View reviewed changes

graytaylor0 merged commit e9bffee into opensearch-project:main Oct 23, 2024
46 of 47 checks passed

graytaylor0 added a commit to graytaylor0/data-prepper that referenced this pull request Oct 25, 2024

Revert "Add custom enum deserializer to improve error messaging, impr…

c410c91

…ove byte count error mesages (opensearch-project#5076)" This reverts commit e9bffee.

graytaylor0 added a commit to graytaylor0/data-prepper that referenced this pull request Oct 25, 2024

Revert "Add custom enum deserializer to improve error messaging, impr…

0d03ff8

…ove byte count error mesages (opensearch-project#5076)" This reverts commit e9bffee.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add custom enum deserializer to improve error messaging, improve byte count error mesages #5076

Add custom enum deserializer to improve error messaging, improve byte count error mesages #5076

graytaylor0 commented Oct 16, 2024 •

edited

Loading

san81 Oct 17, 2024

graytaylor0 Oct 17, 2024

graytaylor0 Oct 17, 2024

san81 Oct 17, 2024

san81 left a comment

san81 Oct 17, 2024

graytaylor0 Oct 17, 2024

san81 Oct 17, 2024

graytaylor0 Oct 17, 2024

san81 Oct 17, 2024

dlvenable Oct 18, 2024

dlvenable Oct 18, 2024

dlvenable Oct 18, 2024

graytaylor0 Oct 21, 2024

dlvenable Oct 21, 2024

graytaylor0 Oct 21, 2024

dlvenable Oct 21, 2024

dlvenable Oct 18, 2024

graytaylor0 Oct 21, 2024

dlvenable Oct 21, 2024

dlvenable Oct 22, 2024

dlvenable Oct 22, 2024

dlvenable Oct 22, 2024

dlvenable Oct 22, 2024

dlvenable left a comment

	private static final Map<String, Unit> UNIT_MAP = Arrays.stream(Unit.values())
	.collect(Collectors.toMap(unit -> unit.unitString, Function.identity()));


	@JsonCreator
	static TargetType fromOptionValue(final String option) {
	return OPTIONS_MAP.get(DataType.fromTypeName(option));
	}

	@JsonValue
	public String getOptionValue() {
	return dataType.getTypeName();
	}

	assertTrue(result instanceof EnumDeserializer);
	assertThat(result, instanceOf(EnumDeserializer.class));

Add custom enum deserializer to improve error messaging, improve byte count error mesages #5076

Add custom enum deserializer to improve error messaging, improve byte count error mesages #5076

Conversation

graytaylor0 commented Oct 16, 2024 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

san81 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment

graytaylor0 commented Oct 16, 2024 •

edited

Loading