OpenSearch Shard Optimization: Managing Petabyte-Scale Log Indexes Without Cluster Performance Degradation

মূল বিষয়সমূহ

• Size shards deliberately. Target 10-50 GB per shard; smaller shards waste overhead, larger ones slow recovery.
• Use time-based indices. Daily or weekly indices match log ingestion patterns and simplify retention.
• Limit shard count. Keep total shards under a few thousand per node to avoid memory pressure.
• Configure ILM policies. Automate rollover, shrink, and delete to control shard growth.
• Use forcemerge on old indices. Merging segments reduces memory and improves query speed.
• Monitor shard allocation. Uneven distribution creates hot nodes; use allocation awareness.
• Tune refresh intervals. Longer intervals reduce indexing overhead for high-throughput logs.
• Plan capacity. Calculate shard count from data volume and retention before deploying.

Architecture

Logs flow into time-based indices, each with a configured number of primary shards. ILM policies rollover when size or age thresholds are met, shrink old indices, and delete expired data. Shard allocation balances across nodes, and forcemerge compacts segments.

Production configuration

{
  "policy": {
    "description": "Log retention policy",
    "default_state": "hot",
    "states": [
      {"name": "hot", "actions": [{"rollover": {"max_size": "50gb", "max_age": "1d"}}], "transitions": [{"state_name": "warm", "conditions": {"min_age": "1d"}}]},
      {"name": "warm", "actions": [{"forcemerge": {"max_num_segments": 1}}, {"shrink": {"number_of_shards": 1}}], "transitions": [{"state_name": "delete", "conditions": {"min_age": "30d"}}]},
      {"name": "delete", "actions": [{"delete": {}}]}
    ]
  }
}

curl -X PUT 'localhost:9200/_index_template/logs' -H 'Content-Type: application/json' -d '{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "30s"
    }
  }
}'

curl -s 'localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,store' | head -20
curl -s 'localhost:9200/_cluster/health?pretty'
curl -s 'localhost:9200/_cat/nodes?v&h=name,shards'

Edge cases from production

• Rollover requires an alias; ensure the write alias points to the current index.
• Shrinking changes shard count; queries must handle the new topology.
• Forcemerge on large indices is I/O intensive; schedule during low traffic.
• Replica shards double storage; balance cost against availability needs.

সচরাচর জিজ্ঞাসা