What’s new with Splunk Dataflow template: Automatic log parsing, UDF support, and more!What’s new with Splunk Dataflow template: Automatic log parsing, UDF support, and more!Solutions Architect

More reliability and error handling

“The board is set, the pieces are moving. We come to it at last, the great battle of our time.”

— Gandalf

Last but not least, the latest Dataflow template improves pipeline fault tolerance and provides a simplified Dataflow operator troubleshooting experience.

In particular, Splunk Dataflow template’s retry capability (with exponential backoff) has been extended to cover transient network failures (e.g. Connection timeout), in addition to transient Splunk server errors (e.g. Server is busy). Previously, the pipeline would immediately drop these events in the dead-letter topic. While this avoids data loss, it also adds unnecessary burden for operator who are responsible to replay these undelivered messages. The new Splunk Dataflow template minimizes this overhead by attempting retries whenever possible, and only dropping messages to dead-letter topic when it’s a persistent issue like those listed in Delivery error types, or when the maximum retry elapsed time has expired (15 min).

Finally, as more customers adopt UDFs to customize the behavior of their pipelines per previous section, we’ve invested in better logging for UDF-based errors such as JavaScript syntax errors. Previously, you could only troubleshoot these errors by inspecting the undelivered messages in the dead-letter topic. You can now view these errors in the worker logs directly from the Dataflow job page in Cloud Console. Here’s an example query you can use in Logs Explorer:

Leave a Comment