Elasticsearch index design and lifecycle management directly impact search performance, storage costs, and cluster stability. This guide covers the essentials for database engineers running ELK stacks.
Index Settings at Creation
json
PUT /logs-2025-06
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.refresh_interval": "30s",
"index.translog.durability": "async"
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"message": { "type": "text" },
"level": { "type": "keyword" },
"service": { "type": "keyword" }
}
}
}Shard Sizing Guidelines
- Target 20-50 GB per shard
- Over-sharding degrades performance — avoid thousands of small shards
- Rule of thumb: shards = (expected index size in GB) / 30
Index Lifecycle Management (ILM)
json
PUT _ilm/policy/logs-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": { "max_size": "50gb", "max_age": "7d" },
"set_priority": { "priority": 100 }
}
},
"warm": {
"min_age": "7d",
"actions": {
"forcemerge": { "max_num_segments": 1 },
"shrink": { "number_of_shards": 1 }
}
},
"delete": {
"min_age": "30d",
"actions": { "delete": {} }
}
}
}
}Force Merge Old Indices
bash
# Merge to 1 segment for read-only historical indices
POST /logs-2025-01/_forcemerge?max_num_segments=1
# Check segment count
GET /logs-2025-01/_segmentsCluster Health Monitoring
bash
GET _cluster/health
GET _cat/indices?v&health=yellow
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reasonKey Takeaways
- Size shards to 20-50 GB — over-sharding causes more overhead than under-sharding
- Use ILM to automatically roll over hot indices, compress warm indices, and delete old ones
- Force merge historical (read-only) indices to 1 segment to improve search performance
- Use
keywordtype for fields you filter/aggregate on,textonly for full-text search
JusDB Can Help
Elasticsearch cluster management is complex. JusDB can review your index design, ILM policies, and shard allocation strategy.