Understanding Event Time
This chapter explains how the Event Time of an Event is created and can be adjusted.
Default Event Time
A Splunk Search scheduled for periodic runs gets sent to the scheduler.
The scheduler will prioritize all jobs according to Scheduler options
in savedsearches.conf and limits.conf.
When the search job runs, the job contains metadata that shows essential
information about the search.
The metadata can, e.g., be seen in the Job Inspector or through the /services/jobs/<sid> endpoint.
When creating a Splunk Alert with an AME Alert Action, the Alert Action will
use the searchTelemetry timestamp for maximum precision
(Note that the job inspector UI does not show milliseconds).
This timestamp will then become the Event Time that will be shown in the first column of the Event Summary.
Note that this timestamp will also be displayed in the Alert Time
column of the Data Tab.
Timezones
As standard in Splunk, AME also stores all timestamps in Epoch Format in UTC. The timestamps presented to the user depend on the timezone settings in the user's preferences.
If a User selects -- Default System Timezone--, the timezone will be set to UTC.
Why is the Default System Timezone in UTC when my servers are in a different timezone?
There is no safe way to convert an OS timezone setting into IANA Format in Javascript. Therefore, the safest way is to set the timezone as a user manually.
Overriding the Event Time
There are cases when the Event Time is not helpful for managing alerts.
Let's assume an Event indexed by Splunk is created by another alerting
or detection system, and instead of the Splunk Search Job time,
the indexed _time of the Splunk event should be used.
This can be easily done by adjusting the search by adding either | eval ame._time=_time or to make the field invisible | eval _ame._time=_time.
AME will use this timestamp as the event time. Note that under the data tab, the Alert Time still contains the timestamp for when
the Alert was fired.