Elastart indices sooo big - how housekeeping them?

[SteMarcMan]

Hi there,
a quick question: The elastalert indices are getting bigger and bigger.
How you truncate / shrink / rollover them over time?
And do you do it in SOC or in elasticsearch itself?
Any ideas please?

Thank you and kind regards!

[Zer0-cyber-web]

There are Index Lifecycle Policies in SOC and Elastic Kibana.
/kibana/app/management/data/index_lifecycle_management/policies
Refer to SO documentation for more info.

In brief - you just set up time of life for all indices and can specify for every index separetely. The most space-consuming index is Zeek, so try to reduce Delete policy for this index.

Other procedures you mentioned are for SQL databases, while Elastic (which is the base for SO) - is NOSQL database, therefore - database maintanance is performed by Elastic itself and no need for additional procedures.

[SteMarcMan]

Actually I am referring to this:
/kibana/app/management/data/index_management/indices?filter=elas
where my elastalert_error really big is!
I need that index either to shrink or start getting redone from scratch.
Could you please give me further advice how you would do this....

What I know: If I click on that index I get this

But I do not see a usable option for my purpose. Any hint?
And in SOC can you tell me where I can set up the time of life for every index separately. Sorry, I did not find it.

Thanks!

[Zer0-cyber-web]

could you give some more info on that index? What is it's size at the moment?
Besides - it is an error index. it indicates that you have some problems... I would first look inside that index and eliminated the cause. Then you can always delete or flush that index, when you will be sure that you don't need it anymore.

[SteMarcMan]

Yes sure:
The size is 13.26GB with only 1 primary shard and 0 replica.
These definition settings sound interesting to me:

{
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "1",
"provided_name": "elastalert_error",
"creation_date": "1742457030336",
"number_of_replicas": "0",
"uuid": "U9UZKYaVQlCKP9bjT_yoeQ",
"version": {
"created": "8505000"
}
}
},
"defaults": {
"index": {
"flush_after_merge": "512mb",
"time_series": {
"end_time": "9999-12-31T23:59:59.999Z",
"start_time": "-9999-01-01T00:00:00Z",
"es87tsdb_codec": {
"enabled": "true"
}
},
"final_pipeline": "_none",
"max_inner_result_window": "100",
"unassigned": {
"node_left": {
"delayed_timeout": "1m"
}
},
"max_terms_count": "65536",
"rollup": {
"source": {
"name": "",
"uuid": ""
}
},
"lifecycle": {
"prefer_ilm": "true",
"rollover_alias": "",
"origination_date": "-1",
"name": "",
"parse_origination_date": "false",
"step": {
"wait_time_threshold": "12h"
},
"indexing_complete": "false"
},
"mode": "standard",
"routing_partition_size": "1",
"force_memory_term_dictionary": "false",
"max_docvalue_fields_search": "100",
"merge": {
"scheduler": {
"max_thread_count": "2",
"auto_throttle": "true",
"max_merge_count": "7"
},
"policy": {
"merge_factor": "32",
"floor_segment": "2mb",
"max_merge_at_once_explicit": "30",
"max_merge_at_once": "10",
"max_merged_segment": "0b",
"expunge_deletes_allowed": "10.0",
"segments_per_tier": "10.0",
"type": "UNSET",
"deletes_pct_allowed": "20.0"
}
}, ...

It is Ok (for the moment!) that I have so many errors, since I am testing the alerting.
But yes, I totally agree with you, errors need to be ironed out - but this is another construction site ;-).
Cheers!

[SteMarcMan]

I decided just to delete the index.