Java likes to allocate and/or reserve a lot of memory. Often (always?) it will allocate or reserve as much RAM as it sees available on the system. Why? Who knows, but it seems ridiculous.
To reign it in, specify a memory allocation as a command-line option when you compile java or run a java script/command.
To compile things with java on the cluster, one needs
javac -J-Xmx123M
To run,
java -Xmx123M
where mx123M is specifies a maximum heap size of 123 Mb. This needs to be less than the available RAM, so for a qlogin job default vmem=3G, you can probably go up to around 2800M at run time.
Compilation should require less RAM, we haven't found a lower limit but think 100M will suffice for many applications. You just have to give it something so it doesn't try to allocate a crazy high number by default.