TotalView Integration broken with PE1304 (linux)

Product or Service: Parallel Environment Runtime Edition for Linux on
x86 Architecture 1.2.0
Component ID: 5725G0001
.
Operating System: Linux
.
Problem title
TotalView Integration broken with PE1304
.
Problem description
With PE1209 the following command opened an interactive totalview window:

bsub -a tv -P SCSG0001 -W 00:30 -n 4 -R "span[ptile=4]" -q small -e %J.
err -o %J.out mpirun.lsf ./a.out

The same command in the just-deployed PE1304 just runs a.out without the debugger.

We have also tried to manually invoke totalview in a normal bsub job,
but that does not work either.

Additional details:


It's not clear that the implementation of totalview and POE ever
really worked as expected in the Linux evironment. The customer uses the 'mpirun.lsf' script which appears to be broken unless it
uses mpcc instead of mpicc.

The difference is that when complied with mpcc, POE (or PAM?) was linked in such that you could start

foo

and foo, realizing it was supposed to be invoked by poe, would do the right thing. However, with UCAR/NCAR deciding they should really compile with mpicc, PAM is no longer linked in, and foo will now no
longer invoke poe.

That's the background. With the mpirun.lsf script they would form a
launch string of

totalview foo poe -a

which invokes TotalView on foo, treats poe as a core file(!) and does
nothing with -a which is used to indicate anything that follows is an
argument. TotalView should be invoked as

totalview poe -a foo

But unfortunately that doesn't work these days, since we don't have the proctable (or similar mechanism) support in poe on linux. On AIX this information would tell us what nodes were being used, and which processes were started and should be attached to.

What we can do with

totalview foo

is bring up the starter screen and then the user can invoke poe - Linux as the starter method. It's really not entirely clear to me why it is failing so badly now, but that was the change that triggered it. The customer has a workaround for now, but its thought that the real solution would be to get proctable support added for Linux POE.