NFS Clients hang during read/write due to deadlock in NFS code

System: AIX
Actor: Customer

NFS clients are hung during read/write operation. From dump analysis, it was found that this hand is due to a deadlock situation in NFS code.

From dump we saw huge number of kbiod threads that needs to flush the IO are stuck as F_RECLAIM flag is set. The n4bg thread which is trying to recover the state of file is waiting for the IOs to be flushed.

0)> f 1025 ===>>> All kbiod thread are in same stack
pvthread+040100 STACK:
[0010AB30]e_block_thread+000290 ()
[0010B6A8]e_sleep_thread+0000E8 (??, ??, ??)
[00014F50].kernel_add_gate_cstack+000030 ()
[0630FFD0]aix_cv_wait_sig+000070 (??, ??)
[063A3F00]nfs4_state_fsid_hold+000060 (??)
[063A2DEC]nfs4_state_start_io+0000CC (??, ??, ??)
[06329084]nfs4write+000304 (??, ??, ??, ??, ??, ??, ??, ??)
[0633EBC4]nfs4_bio+000764 (??, ??, ??)
[0636552C]do_async_op+00022C (??)
[06364BDC]nfs_async_start+0002FC (??)
[0029C154]threadentry+000054 (??, ??, ??, ??)
[kdb_read_mem] no real storage @ FFFFFFFFFFF8E60


Further looking at the n4bg threads:

(0)> th -n n4bg
SLOT NAME STATE TID PRI RQ CPUID CL WCHAN

pvthread+009400 148 n4bg SLEEP 9400A7 03C 0 0 F1000A0150507730
pvthread+069D00 1693 n4bg SLEEP 69D00CB 03C 1025 0 vmmary_scb+488EF488
pvthread+09A400 2468 n4bg SLEEP 1A401EF 03C 1025 0 F1000A0310290970

(0)> f 1693
pvthread+069D00 STACK:
[00074CC0]begfst+0001E4 ()
[00074CBC]begfst+0001E0 ()
[0072D6FC]vms_iowaitf+00007C (??, ??)
[06366B90]flush_vp+0004F0 (??, ??)
[063673F8]binvalfree+0001B8 (??)
[063BAF00]nfs4_remove_delegation+000160 (??, ??)
[063C2804]nfs4_recover_open_state+000464 (??, ??)
[063C1834]recover_fsid_files+0000D4 (??)
[0029C154]threadentry+000054 (??, ??, ??, ??)
[kdb_read_mem] no real storage @ FFFFFFFFFFF8E60

This deadlock is causing a hang on 4 NFS clients and it is impacting customer business

Description:

Changing the vms_iowait() as a timed call might help to fix this issue. Any code change which will overcome this deadlock situation is appreciated by customer.