Redis is crashlooping and with it all OpenReplay services

Issues & Bugs 🛠️ Deployment
1

Hello,
My server storage has been full, which I increased then restart the VM. But now it looks like all OpenReplay services are down. Restarting all of the; with openreplay -r all doesn’t help as redis is crashlooping therefore preventing other services from starting. It seems its due to a corrupted file(s):

Bad file format reading the append only file appendonly.aof.57.incr.aof

image
image1836Ă—1482 295 KB

2

That’s right, some files are corrupt due to the storage going full and not a proper restart of the services including Redis. Most of OpenReplay services rely on Redis for the data pipeline. If Redis fails to start, all other components would certainly complain and start crashlooping.

Here is how to fix:

  1. cd /var/lib/rancher/k3s/storage/
  2. Under it you should see a directory with pvc-*****redis-master**** in it, then cd appendonlydir and finally run rm -rf * to delete all files
  3. Finally run kubectl rollout restart sts -n db redis-master to restart redis