[DRMAA-WG] Meeting Minutes OGF-42, Day 1

[Peter Tröger]

Dear all,

we discussed a list of open issues for the DRMAAv2 root specification and C binding. Here is a summary of the decisions:

- Root specification and C binding get a new function ?reap()? in the Job interface:
http://redmine.ogf.org/issues/163
- The C binding gets a new function ?exit()? on global scope:
http://redmine.ogf.org/issues/103
- Root specification and C binding get a new attribute ?jobName? in the ?JobInfo? structure:
http://redmine.ogf.org/issues/102
- The C binding will define the file name for the implementation-specific header file:
http://redmine.ogf.org/issues/162
- All extensible structures in the C binding get an additional attribute, pointing to the impl.-specific data structure:
http://redmine.ogf.org/issues/160
- The C binding gets methods for list copying and dictionary copying:
http://redmine.ogf.org/issues/58
- C binding constants shall be declared as extern, so that the implementation defines them:
http://redmine.ogf.org/issues/57
- Minor text clarifications and method name fixes in root specification and C binding:
http://redmine.ogf.org/issues/61
http://redmine.ogf.org/issues/114
http://redmine.ogf.org/issues/113
http://redmine.ogf.org/issues/104
http://redmine.ogf.org/issues/59
http://redmine.ogf.org/issues/165

For the following issue, we came to no feasible conclusion, so your feedback is desperately demanded:
http://redmine.ogf.org/issues/116
http://redmine.ogf.org/issues/115

Please note that DRMAAv2 (C binding) is now shipped with Univa Grid Engine 6.2 as official part of the product, which is a huge step forward. This also makes the development and testing of language bindings much easier.

Daniel Gruber gave us some insight into the Go language binding and the according prototype implementation:

http://redmine.ogf.org/projects/drmaav2-go-binding/repository/revisions/master/changes/drmaa2-go.pdf
https://github.com/dgruber/drmaa/tree/master

Due to several requests, we are now also actively targeting the Python and Java language binding. Check for updates in the OGF Redmine pages.

Best regards,
Peter.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140910/e1531c87/attachment.pgp>

[Bruno P. Kinoshita]

Hi Peter!

Unfortunately I'm new to DRMAA and can't contribute much to the decision, but thanks a lot for summarizing it.

I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.

Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.

Thanks
Bruno

________________________________
From: Peter Tr?ger <peter at troeger.eu>
To: drmaa-wg <drmaa-wg at ogf.org>
Sent: Tuesday, September 9, 2014 8:06 PM
Subject: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear all,
http://redmine.ogf.org/issues/163
http://redmine.ogf.org/issues/103
http://redmine.ogf.org/issues/102
http://redmine.ogf.org/issues/162
http://redmine.ogf.org/issues/160
http://redmine.ogf.org/issues/58
http://redmine.ogf.org/issues/57
http://redmine.ogf.org/issues/61
http://redmine.ogf.org/issues/114
http://redmine.ogf.org/issues/113
http://redmine.ogf.org/issues/104
http://redmine.ogf.org/issues/59
http://redmine.ogf.org/issues/165
http://redmine.ogf.org/issues/116
http://redmine.ogf.org/issues/115
Please note that DRMAAv2 (C binding) is now shipped with Univa Grid Engine 6.2 as official part of the product, which is a huge step forward. This also makes the development and testing of language bindings much easier.
http://redmine.ogf.org/projects/drmaav2-go-binding/repository/revisions/master/changes/drmaa2-go.pdf
https://github.com/dgruber/drmaa/tree/master
Due to several requests, we are now also actively targeting the Python and Java language binding. Check for updates in the OGF Redmine pages.
Best regards,
Peter.
--
drmaa-wg mailing list
drmaa-wg at ogf.org
https://www.ogf.org/mailman/listinfo/drmaa-wg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140909/a06b22f7/attachment.html>

[Peter Tröger]

Dear Bruno,

thanks for your feedback, we always need encouraged people to join the DRMAA working group.

Post by Bruno P. Kinoshita
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.

We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.

Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).

Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.

Best regards,
Peter.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140910/edef5bb3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140910/edef5bb3/attachment.pgp>

[Bruno P. Kinoshita]

Hello Peter

Post by Peter Tröger
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.

That's our plan. And we also believe that using an industry-proven software, with formed community and documentation will make it simpler for contributors and new developers.

Post by Peter Tröger
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).

That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot.

Post by Peter Tröger
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.

Thanks! Are you going to use GitHub? If so I can probably bother you there with issues and maybe PR's :)

All the best,
Bruno

Post by Peter Tröger
________________________________
From: Peter Tr?ger <peter at troeger.eu>
To: Bruno P. Kinoshita <brunodepaulak at yahoo.com.br>
Cc: drmaa-wg <drmaa-wg at ogf.org>
Sent: Wednesday, September 10, 2014 4:14 AM
Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear Bruno,
thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.

Post by Bruno P. Kinoshita
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.

We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Best regards,
Peter.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140910/5b4e8008/attachment-0001.html>

[Khalid Shakir]

Hi Bruno,

I've seen your blog posts on possibly using SWIG or JNI for DRMAA. Back
with DRMAAv1, with both a C <http://redmine.ogf.org/documents/10> and a Java
<http://redmine.ogf.org/documents/11> specs available, we implemented a
generic DRMAAv1 Java-to-C binding using JNA <https://github.com/twall/jna>'s
dynamic invocation. While I never had time to support our wrapper
<https://github.com/broadgsa/gatk/tree/9d15d241af5d4a3709e3c76249cff4c2a9053d60/public/gatk-tools-public/src/main/java/org/broadinstitute/gatk/utils/jna/drmaa/v1_0>
as a proper mavenized library, I'd love to see someone publish a similar
DRMAAv2 JNA wrapper as an artifact, of course after both specs are out.

Anecdotally, after a user contributed patch mapped other parts of our
software to PBS specific variables, the above wrapper seemed to work for at
least a another
<http://gatkforums.broadinstitute.org/discussion/2150/queue-and-pbs> without
further modification to our DRMAAv1 JNA code. While I haven't measured the
actual speed of GridEngine-via-JNA vs. GridEngine-via-JNI, this dynamic
wrapper "feels" fast enough for most users, as the JNA probably adds <0.5s
per call, ever. This known cost
<http://developers.opengamma.com/blog/2012/05/25/jna-jni-and-raw-java-performance>
has been worth it since we now only include a single DRMAAv1-JNA wrapper,
versus multiple JNI wrappers for DRMAAv1-PBS, DRMAAv1-GridEngine, etc.

Best wishes,
-k


On Wed, Sep 10, 2014 at 9:10 PM, Bruno P. Kinoshita <

Post by Bruno P. Kinoshita
Hello Peter

Post by Peter Tröger
We spent several years on defining an API that fits to the majority of

cluster products today and tomorrow, so using the DRMAA2 spec may save you
some thinking time. Please note also that many features of DRMAA2 are
optional, so you can start small and extend your implementations
step-by-step.
That's our plan. And we also believe that using an industry-proven
software, with formed community and documentation will make it simpler for
contributors and new developers.

Post by Peter Tröger
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which

would allow you to implement a DRMA2 Java binding by a thin Java-to-C
conversion layer for this product. If we get DRMAA2 C libraries for the
other products, this layer would be re-usable (as we had it with DRMAAv1).
That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot.

Post by Peter Tröger
Including your request, I have now a number of serious demands for the

Java binding spec. I will start with an initial version during the next
days and keep the list informed. Testing and implementations for that would
be great anyways.
Thanks! Are you going to use GitHub? If so I can probably bother you there
with issues and maybe PR's :)
All the best,
Bruno
------------------------------
*From:* Peter Tr?ger <peter at troeger.eu>
*To:* Bruno P. Kinoshita <brunodepaulak at yahoo.com.br>
*Cc:* drmaa-wg <drmaa-wg at ogf.org>
*Sent:* Wednesday, September 10, 2014 4:14 AM
*Subject:* Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear Bruno,
thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS
Java API at the moment, but users asked for a generic grid computing
plug-in for Jenkins, supporting different vendors.
Maybe I can help with the Java binding if necessary, with testing,
reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2,
or write our own little common API in Java for PBS, SGE, etc.
We spent several years on defining an API that fits to the majority of
cluster products today and tomorrow, so using the DRMAA2 spec may save you
some thinking time. Please note also that many features of DRMAA2 are
optional, so you can start small and extend your implementations
step-by-step.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would
allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion
layer for this product. If we get DRMAA2 C libraries for the other
products, this layer would be re-usable (as we had it with DRMAAv1).
Including your request, I have now a number of serious demands for the
Java binding spec. I will start with an initial version during the next
days and keep the list informed. Testing and implementations for that would
be great anyways.
Best regards,
Peter.
--
drmaa-wg mailing list
drmaa-wg at ogf.org
https://www.ogf.org/mailman/listinfo/drmaa-wg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140911/5e6c116f/attachment.html>

[Bruno P. Kinoshita]

Hello Khalid

Very interesting links, thanks! I haven't used JNI or JNA for quite some time. Supporting Maven is a requirement for us, so if the DRMAA v2 isn't released to Maven Central we will probably fork and release it under a different groupId.

Due to Jenkins master/slave architecture, we have two models in mind. One where the user will start a slave remotely in one computer with access to submit jobs and manage the queue (here we can use JNA or JNI). And the other model, where installing a slave (some Java code communicating with the master via SSH) won't be necessary, and we will use overthere [1] to send commands and transfer files via SSH.

Our main requirements in this Jenkins plug-in are to submit Jobs but be able to mix local processing and cluster processing, and manage the queue remotely, so that researchers won't have to log in to check the status of the submitted jobs.

In any case, we will probably have to use the DRMAA v2 Java binding, but for the latter case write custom remote Java APIs (one for PBS, one for SGE, and so it goes) that don't use JNI or JNA, but sends commands via SSH and parses the output.

It is not clear which is the best way, so we will work with the researchers interested in testing the plug-in to see which model works best.

Thanks!
Bruno

[1] https://github.com/xebialabs/overthere

________________________________
From: Khalid Shakir <kshakir at broadinstitute.org>
To: Bruno P. Kinoshita <brunodepaulak at yahoo.com.br>
Cc: drmaa-wg <drmaa-wg at ogf.org>
Sent: Wednesday, September 10, 2014 3:15 PM
Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Hi Bruno,
I've seen your blog posts on possibly using SWIG or JNI for DRMAA. Back with DRMAAv1, with both a C and a Java specs available, we implemented a generic DRMAAv1 Java-to-C binding using JNA's dynamic invocation. While I never had time to support our wrapper as a proper mavenized library, I'd love to see someone publish a similar DRMAAv2 JNA wrapper as an artifact, of course after both specs are out.
Anecdotally, after a user contributed patch mapped other parts of our software to PBS specific variables, the above wrapper seemed to work for at least a another without further modification to our DRMAAv1 JNA code. While I haven't measured the actual speed of GridEngine-via-JNA vs. GridEngine-via-JNI, this dynamic wrapper "feels" fast enough for most users, as the JNA probably adds <0.5s per call, ever. This known cost has been worth it since we now only include a single DRMAAv1-JNA wrapper, versus multiple JNI wrappers for DRMAAv1-PBS, DRMAAv1-GridEngine, etc.
Best wishes,
-k
Hello Peter

Post by Bruno P. Kinoshita

Post by Peter Tröger
We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.

That's our plan. And we also believe that using an industry-proven software, with formed community and documentation will make it simpler for contributors and new developers.

Post by Peter Tröger
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).

That sounds great. I'll try to hack an ansible + vagrant/docker script to give it a shot.

Post by Peter Tröger
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.

Thanks! Are you going to use GitHub? If so I can probably bother you there with issues and maybe PR's :)
All the best,
Bruno

Post by Peter Tröger
________________________________
From: Peter Tr?ger <peter at troeger.eu>
To: Bruno P. Kinoshita <brunodepaulak at yahoo.com.br>
Cc: drmaa-wg <drmaa-wg at ogf.org>
Sent: Wednesday, September 10, 2014 4:14 AM
Subject: Re: [DRMAA-WG] Meeting Minutes OGF-42, Day 1
Dear Bruno,
thanks for your feedback, we always need encouraged people to join the DRMAA working group.
I am the maintainer of a PBS Plug-in for Jenkins. It uses a custom PBS Java API at the moment, but users asked for a generic grid computing plug-in for Jenkins, supporting different vendors.

Post by Bruno P. Kinoshita
Maybe I can help with the Java binding if necessary, with testing, reviewing, writing docs, etc. We are deciding whether to wait for DRMAA v2, or write our own little common API in Java for PBS, SGE, etc.

We spent several years on defining an API that fits to the majority of cluster products today and tomorrow, so using the DRMAA2 spec may save you some thinking time. Please note also that many features of DRMAA2 are optional, so you can start small and extend your implementations step-by-step.
Univa Grid Engine 8.2 is now supporting DRMAA2 out of the box, which would allow you to implement a DRMA2 Java binding by a thin Java-to-C conversion layer for this product. If we get DRMAA2 C libraries for the other products, this layer would be re-usable (as we had it with DRMAAv1).
Including your request, I have now a number of serious demands for the Java binding spec. I will start with an initial version during the next days and keep the list informed. Testing and implementations for that would be great anyways.
Best regards,
Peter.

--
drmaa-wg mailing list
drmaa-wg at ogf.org
https://www.ogf.org/mailman/listinfo/drmaa-wg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ogf.org/pipermail/drmaa-wg/attachments/20140911/d7e9285c/attachment-0001.html>