Rescue a Running Service
During a rushed deploy, someone ran rm -rf /opt/exporter while the metrics-exporter service was still running, deleting its program and its config, and there's no backup. The good news: the service is still up, and while it runs, Linux keeps everything about it in /proc. Recover what you need, then restart it cleanly. Your password is student.
How it works: the terminal below is a real Linux machine running in your browser. Click it and
type. Tasks tick themselves off as you complete them; questions are answered in the box under each task.
Press Reset any time for a fresh machine. Your password on this machine (for
sudo) is student.
Hints
Find the PID once with pgrep -x exporter; everything else lives in /proc/<PID>/.
The process runs as svc, so read its /proc files with sudo.
Solution explanation
Everything is read from /proc/<PID>/, which the kernel keeps for as long as the process lives. cwd is a link to its working folder, environ holds EXPORTER_CONF, and fd/255 is the script that bash keeps open while it runs it, even after the file is deleted, so cat brings it back. The process belongs to svc, so reading its private files needs sudo. Recover first, then stop it: once it's gone, so is everything in /proc.
Show solution
ps -o user= -C exporter
pgrep -x exporter
sudo readlink /proc/$(pgrep -x exporter)/cwd
sudo cat /proc/$(pgrep -x exporter)/environ | tr '\0' '\n'
sudo ls -l /proc/$(pgrep -x exporter)/fd
sudo cat /proc/$(pgrep -x exporter)/fd/255 > rescue/exporter
head -2 rescue/exporter
sudo pkill -x exporter