As you indicated, Peter, it is not as easy as it sounds to add all job template attributes
to the job info object. As mentioned before I think in most systems you can change the
submission attributes after submission (in Grid Engine we have qalter, JSV, job classes,
sge_request files, the -@ parameter, path aliasing) therefore we should be conservative and not
specify that the submission parameters in job info needs to match 100% the job template.
Andre: You don?t need to keep the job template at this point in time anyhow. There is a
get job template method specified on the job object itself.
When it comes to the Univa Grid Engine implementation, I?m storing the job template
in the library when a submission is done. When you leave the application and do a
re-connect the behavior is unspecified (meaning there is no job template available
anymore). While there is now a way in Univa Grid Engine 8.2 to get the information
out of qstat (you can see the qsub request line in qstat and qacct) I would not rely on
this to be available on all systems. Even when a 3rd party DRMAA2 implementation
relay on parsing this out, this is highly error prone (as new parameters and syntax
changes can come in, which makes an implementation unusable).
Also, we need to consider that the job can be either submitted from the drmaa2 app
(where we have the job template) and in a monitoring session the job can be
from any source (where we don?t have the job template). Having here a bunch of
attributes which finally are optional and not available makes the interface not
attractive. The job info object is extensible hence each implementation can decided
what to add and what not as the last way out to get something like a job name
into job info.
Just one example to make it more clearer:
You can submit a job with an certain output path in job template but on
the machine where it runs the output path could be translated in something
different, probably because of a different mount point. In UGE we have
something called path aliasing, which is done by the system transparently
to the user. Now which output path would be correct?
When the job template output path is right, then the question
is if it is catchable by all systems in a general way with job status calls?
When the real output path is right then the question is do we get that one
for all systems? I don?t think we can specify which one is the right on
DRMAA level hence we would need to make all of them optional or implementation
specific. Hence the implementation specific approach seems more
natural for me.
Due to the fact that we have already a method for returning the job template,
and in order to have small consistent set of attributes in job info, I
would really not go the way to do such a major change.
Why I added this *job name* request in the job info object is because
the job name (rather then other attributes) is probably the most
natural attribute of a job. In Univa Grid Engine you can do a lot of
useful things with it: defining job dependencies (or multi-job dependencies),
using is for a grouped qstat etc.
Short:
- I just want to add a job name in job info since it is one of the most
basic job info attributes
- I do not want to specify whether the job name matches the job name
of the job template or not. I would only specify that this is the job name
the system considers as the job name - and as a fall back to have it
available in all implementations it could be also just the job id.
- I do not want to add all job template attributes to the job info object
since most of them we don?t need for filtering in real live and due to
the fact that they would be optional anyhow. Hence implementation
specific extensions is the way to go IMHO.
- I can also live with the fact that job name is implementation specific
but I really see the job name very basic and really worthy for job
filtering (especially when you want to group jobs like in UGE case).
Sorry the delayed answer
Cheers
Daniel
Post by Peter TrögerI agree to Bruno and Andre that the advantages are worth it. Performance considerations should be done in the language bindings and / or the implementations, the root spec is the wrong place for that. I therefore vote for the simple addition all job templates attributes to the JobInfo structure.
An interesting side aspect are implementation-specific job template attributes. I would prefer to leave it to the implementors if they want to support them also in the JobInfo structure.
If Daniel agrees to all of this, especially from his implementation viewpoint in Grid Engine 8.2, then I would update the root spec and the C binding accordingly.
Best regards,
Peter.
Post by Andre MerzkyI think the job template information are easy to add, and make app
writing easier (I don't need to keep the old submitted job template
around...). Only drawback I see is increase in memory footprint when
used to report on many jobs -- but when used as a filtering
instruction, this should not matter.
My $0,02, Andre.
Post by Peter TrögerDear all,
http://redmine.ogf.org/issues/102
This leads to the situation that you cannot filter for it when doing monitoring.
Although we quickly decided to add it, I meanwhile figured out that this is a deeper problem. JobInfo intentionally contains nothing from the job template at the moment, which makes perfectly sense if you see it as a pure reporting data structure. You don?t need to fetch information that you handed in by yourself as the calling application.
For filtering, however, you obviously want to have all of the template attributes being usable. This is not possible at the moment.
Another case is the ?change job template after submission? magic in Grid Engine, which could lead to the fact that your job has different attributes from what you originally specified. You may also want to see that in the JobInfo structure.
All in all, it seems like that ?JobInfo? needs all of attributes from ?JobTemplate? too, not only the job name.
What do you think ?
Best regards,
Peter.
--
drmaa-wg mailing list
drmaa-wg at ogf.org
https://www.ogf.org/mailman/listinfo/drmaa-wg
--
It was a sad and disappointing day when I discovered that my Universal
Remote Control did not, in fact, control the Universe. (Not even
remotely.)