Feature Request - Add files property to remap transform

[brittonhayes]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The proposed feature would allow users to break down Vector remaps into smaller, more manageable units of configuration. This would enable:

1. Improved organization of VRL code
2. Easier sharing of VRL snippets across components
3. Better testability of individual remap sections
4. Simplified maintenance of complex transformations

Attempted Solutions

Currently, users can only specify a single file for the remap transform using the file property. To achieve similar functionality, users might:

1. Create multiple remap components, which can be cumbersome and less efficient.
2. Use external tools like jsonnet to combine multiple VRL files, adding complexity to the configuration process.

However, these solutions do not provide the seamless experience that a native files property would offer.

Proposal

Add a new files property to the remap transform that accepts an array of file paths. When files is provided (and file or source are not), Vector would concatenate the contents of the specified VRL files with a newline separator.

Example configuration:

transforms:
  my_transform_id:
    type: remap
    inputs:
      - my-source-or-transform-id
    files:
      - ./my/remap_a.vrl
      - ./my/remap_b.vrl

This would be equivalent to:

transforms:
  my_transform_id:
    type: remap
    inputs:
      - my-source-or-transform-id
    source: |
      # Contents of ./my/remap_a.vrl
      . = parse_json!(string!(.message))
      .timestamp = to_unix_timestamp(to_timestamp!(.timestamp))

      # Contents of ./my/remap_b.vrl
      del(.username)
      .message = downcase(string!(.message))

This feature would allow users to organize their VRL code into smaller, more manageable files while still benefiting from the efficiency of a single remap transform.

References

No directly related issues found. This feature request is based on user needs for improved VRL management and configuration flexibility.

• Conversation about the feature in Discord

Version

0.41.0