diff --git a/website/content/en/docs/reference/vrl/_index.md b/website/content/en/docs/reference/vrl/_index.md index 8404bc1e18463..1cb2412890f05 100644 --- a/website/content/en/docs/reference/vrl/_index.md +++ b/website/content/en/docs/reference/vrl/_index.md @@ -25,14 +25,15 @@ VRL programs act on a single observability [event](#event) and can be used to: Those programs are specified as part of your Vector [configuration]. Here's an example `remap` transform that contains a VRL program in the `source` field: -```toml {title="vector.toml"} -[transforms.modify] -type = "remap" -inputs = ["logs"] -source = ''' - del(.user_info) - .timestamp = now() -''' +```YAML {title="vector.yaml"} +transforms: + modify: + type: remap + inputs: + - logs + source: | + del(.user_info) + .timestamp = now() ``` This program changes the contents of each event that passes through this @@ -50,30 +51,49 @@ HTTP log events that look like this: } ``` -You want to apply these changes to each event: +Let's assume you want to apply a set of changes to each event that arrives to your Remap transform in order to produce +an event with the following fields: -- Parse the raw string into JSON -- Attempt to convert the timestamp and checks if the conversion was successful -- If the conversion is successful, convert the time to a UNIX timestamp; otherwise, use the current time -- Remove the `username` field -- Remove the temporary timestamp (`parsed_timestamp`) field -- Convert the `message` to lowercase +- `message` (string) +- `status` (int) +- `timestamp` (int) +- `timestamp_str` (timestamp) -This VRL program would accomplish all of that: +The following VRL program demonstrates how to achieve the above: ```coffee +# Parse the raw string into a JSON object, this way we can manipulate fields. . = parse_json!(string!(.message)) -.parsed_timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S.%fZ") - -if is_timestamp(.parsed_timestamp) { - .timestamp = to_unix_timestamp(.parsed_timestamp) +# At this point `.` is the following: +#{ +# "message": "SUCCESS", +# "status": 200, +# "timestamp": "2021-03-01T19:19:24.646170Z", +# "username": "ub40fan4life" +#} + +# Attempt to parse the timestamp that was in the original message. +# Note that `.timestamp` can be `null` if it wasn't present. +parsed_timestamp, err = parse_timestamp(.timestamp, format: "%Y-%m-%dT%H:%M:%S.%fZ") + +# Check if the conversion was successful. Note here that all errors must be handled, more on that later. +if err == null { + # Note that the `to_unix_timestamp` expects a `timestamp` argument. + # The following will compile because `parse_timestamp` returns a `timestamp`. + .timestamp = to_unix_timestamp(parsed_timestamp) } else { + # Conversion failed, in this case use the current time. .timestamp = to_unix_timestamp(now()) } +# Convert back to timestamp for this tutorial. +.timestamp_str = from_unix_timestamp!(.timestamp) + +# Remove the `username` field from the final target. del(.username) -del(.parsed_timestamp) + +# Convert the `message` to lowercase. .message = downcase(string!(.message)) ``` @@ -83,7 +103,8 @@ Finally, the resulting event: { "message": "success", "status": 200, - "timestamp": 1614626364 + "timestamp": 1614644364, + "timestamp_str": "2021-03-02T00:19:24Z" } ``` @@ -94,11 +115,13 @@ event. But you can also use VRL to specify conditions, which convert events into a single Boolean expression. Here's an example [`filter`][filter] transform that filters out all messages for which the `severity` field equals `"info"`: -```toml {title="vector.toml"} -[transforms.filter_out_info] -type = "filter" -inputs = ["logs"] -condition = '.severity != "info"' +```yaml {title="vector.yaml"} +transforms: + filter_out_info: + type: filter + inputs: + - logs + condition: '.severity != "info"' ``` Conditions can also be more multifaceted. This condition would filter out all @@ -106,7 +129,7 @@ events for which the `severity` field is `"info"`, the `status_code` field is greater than or equal to 400, and the `host` field isn't set: ```coffee -condition = '.severity != "info" && .status_code < 400 && exists(.host) +condition = '.severity != "info" && .status_code < 400 && exists(.host)' ``` {{< info title="More VRL examples" >}} You can find more VRL examples further