How to free the disk space from deleted files in which PIDs are still running?

Sometimes we have removed large size of logfiles or files from linux server as part of housekeeping. But it will not release the disk space while running lsof command

[root@testserver ~]# lsof / | grep -i deleted
sh          3716       root    4w   REG  249,0  2269000 3342481 /var/log/cellos/mcelogd-mon.log (deleted)
sh          3716       root    5w   REG  249,0  2333681 3342525 /var/log/cellos/mcelogd-mon.trc (deleted)
tgtd        4144       root    4w   REG  249,0 26684869 3342480 /var/log/cellos/qd.log (deleted)
tgtd        4144       root    5w   REG  249,0 27453138 3342483 /var/log/cellos/qd.trc (deleted)
tgtd        4145       root    4w   REG  249,0 26684869 3342480 /var/log/cellos/qd.log (deleted)
tgtd        4145       root    5w   REG  249,0 27453138 3342483 /var/log/cellos/qd.trc (deleted)
iscsiuio    4592       root    4w   REG  249,0 26684869 3342480 /var/log/cellos/qd.log (deleted)
iscsiuio    4592       root    5w   REG  249,0 27453138 3342483 /var/log/cellos/qd.trc (deleted)

Normally we need to stop the logging service (syslog) before deleting the log files or stop the service which is related to the files.
Stopping  log collection on running server is not good practice.

Then how to solve the issue?

Free the Disk Space

Find the PID of deleted process, go to /proc then nullify it.

[root@testserver ~]# cd /proc/3716/fd
[root@testserver fd]# ls -al
total 0
dr-x------ 2 root root  0 Apr  9 17:08 .
dr-xr-xr-x 9 root root  0 Apr  9 17:08 ..
lrwx------ 1 root root 64 Apr  9 17:08 0 -> /dev/null
l-wx------ 1 root root 64 Apr  9 17:08 1 -> /var/log/exadatatmp/__imglog_stdout__.tFvWrjjEZTPOJWm3racIEPF5W9YQqMI3
lrwx------ 1 root root 64 Apr  9 17:08 10 -> /dev/null
lrwx------ 1 root root 64 Apr  9 17:08 11 -> /dev/null
l-wx------ 1 root root 64 Apr  9 17:08 2 -> /var/log/exadatatmp/__imglog_stderr__.tFvWrjjEZTPOJWm3racIEPF5W9YQqMI3
lr-x------ 1 root root 64 Apr  9 17:08 255 -> pipe:[21686]
l-wx------ 1 root root 64 Apr  9 17:08 4 -> /var/log/cellos/mcelogd-mon.log (deleted)
l-wx------ 1 root root 64 Apr  9 17:08 5 -> /var/log/cellos/mcelogd-mon.trc (deleted)

The process ID 3716 running on 4w (write mode).

[root@testserver fd]# ls -al 4
l-wx------ 1 root root 64 Apr  9 17:08 4 -> /var/log/cellos/mcelogd-mon.log (deleted)

We can release the space by nullify this file.

[root@testserver ~]# >/proc/3716/fd/4
[root@testserver ~]# lsof / | grep -i deleted | grep -i 3716
sh          3716       root    5w   REG  249,0        0 3342525 /var/log/cellos/mcelogd-mon.trc (deleted)

Alternate way, we can compress the file or we can kill the PID if not needed.