Skip to main content
Version: 1.3.0

Troubleshooting

Users are able to see instances and queries that they should not be able to see.

Most likely ElasticSPL was upgraded from a version prior to 1.2.0 without following the upgrade path. Please rerun the following update tasks:

  • Instance Pemission Migration
  • Query Permission Migration

To do so, visit the Setup and Migration page (by navigating to /app/SA-DP-elasticspl/setup) and click on the Execute button for the corresponding tasks.

I get an error when trying to add / update / delete a configuration object.

Open your browser's developer tools and check the network tab for the request that failed. The response should contain a uuid that can be used to find the corresponding log entries. Copy the uuid and run the following command on the search head:

index=_internal source=*datapunctum* <uuid>
I get an error when trying to ping an Elasticsearch instance.

Check the error presented in the notification in the top right corner. Follow the following steps for basic troubleshooting:

  • Check the logs in the _internal index for more details
  • Check the Elasticsearch instance's logs for more details
  • If applicable, connect to the Splunk instance using SSH and run the following command to check if the Splunk instance can reach the Elasticsearch instance:
curl -XGET <elasticsearch-instance-url>
  • If applicable and using basic authentication, check if the credentials are correct by running the following command:
curl -XGET <elasticsearch-instance-url> -u <username>:<password>
  • If applicable and using API keys, check if the credentials are correct by running the following command (replace <api_id> and <api_key> or generate the base64 encoded string using your preferred method):
curl -v -H "Authorization: ApiKey `echo -n <api_id>:<api_key> | base64`" <elasticsearch-instance-url>
I get a "Results limited to 10000" warning when running a query.

If you see this warning the parameter max_results is set to a value lower than 10000. You can increase the value of max_results or remove it from the command. The value must be an integer. In case removing the parameter max_results did not solve the issue, you have to increase the parameter max_results on the instance used. See Instance Configuration for more details.

I get a "XXXX logs didn't have a timestamp field, please check your query" warning when running a query.

If you see this warning the parameter timestamp_field is set to a value that doesn't exist in the logs. You must set the parameter to a field that exists in the logs.

My query is slow.

A slow query can be caused by several factors. The following steps can be taken to improve the performance of a query:

  • Reduce the number of fields returned by the query (only available in DSL queries)
  • Consider using Aggregation DSL queries if you are using stats after fetching data from Elasticsearch (see here for a guided way to build DSL queries)
  • Reduce the number of logs returned by the query
  • Reduce the time range of the query
  • Depending on the data searched, the parameter scroll_size can be increased or decreased to improve the performance of the query. The default value is 1000. Greater values will result in fewer requests to Elasticsearch but will require more memory on both Elasticsearch and Splunk. Smaller values will result in more requests to Elasticsearch but will require less memory
  • Check the latency between the Splunk instance and the Elasticsearch instance as many requests are made between the two instances

In general, the denser the logs are, the slower the query will be. Furthermore, the more fields are returned by the query, the more data needs to be transferred between the Elasticsearch instance and the Splunk instance. This can cause the query to be slow.