[Ceph] Troubleshoot Ceph PG Incomplete
Investigating
You can check globally using ceph -s and you can see there is incomplete or inactive but for more detail you can see by using this command. On this command result osd 100 is the primary of pg 19.64
ceph pg dump_stuck inactive
output:
ok
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
19.64 incomplete [100,85,89] 100 [100,85,89] 100
Or you can use this command for more details
ceph pg ls incomplete
output:
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
19.64 297 0 0 0 1245708288 0 0 1328 incomplete 24m 69498'1328 204713:96240 [100,85,89]p100 [100,85,89]p100 2022-04-04T04:47:37.785708+0000 2021-10-21T06:24:54.701161+0000
You can also check where is the PG lives
root@juju-bf8473-19-lxd-0:~# ceph osd lspools
1 default.rgw.buckets.data
2 default.rgw.control
3 default.rgw.data.root
4 default.rgw.gc
5 default.rgw.log
6 default.rgw.intent-log
7 default.rgw.meta
8 default.rgw.usage
9 default.rgw.users.keys
10 default.rgw.users.email
11 default.rgw.users.swift
12 default.rgw.users.uid
13 default.rgw.buckets.extra
14 default.rgw.buckets.index
15 .rgw.root
16 gnocchi
17 cinder-ceph
18 glance
19 scbench
20 cinder-ceph-ssd
21 device_health_metrics
22 testbench
23 rbd-kubernetes
24 k8s-uat
PG start with pool id follow by other numbers, for example in this case pg 19.64 lives on sbench
First Workaround
Set flag on ceph cluster
sudo ceph osd set noout
sudo ceph osd set norebalance
sudo ceph osd set nobackfill
sudo ceph osd set norecover
Check the pg query
ceph pg {pg.id} query | grep -Ew 'state|peer'
Try restart the osd one by one
ceph osd find {osd.id}
systemctl restart ceph-osd@{osd.id}
After that check the cluster status
Second Workaround
If the first workaround still not helping try use this but you must aware that there is possibility of data will be loss and you still need to set the ceph flags
Stop the primary osd daemon
systemctl stop ceph-osd@{osd.id}
After that mark complete the pg
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-{osd.id} --op mark-complete --pgid {pg.id}
Then you can start the osd daemon again and check the ceph cluster
systemctl start ceph-osd@{osd.id}
References
- https://www.oreilly.com
- https://lists.ceph.io
- https://medium.com/opsops
Happy, Enjoy Ngoprek ~
Comments