This security hole effects all Mozilla-family nightlies, alphas and betas issued since 2004-04-25 (when they are used together with the MRJ Plugin), but not any of the released versions -- so, for example, Firefox 1.0.X, Mozilla 1.7.X and Camino 0.8.X aren't vulnerable.
It's a serious vulnerability -- at least as serious as the Sun Java Plugin Arbitrary Package Access Vulnerability. In fact, in vulnerable browsers it disables the fix for the Arbitrary Package Access vulnerability that has been included with the Java Embedding Plugin since version 0.8.8.
The hole is caused by an error in how the old MRJ Plugin Carbon and older (pre 0.9.2) versions of the MRJ Plugin JEP handled the security for JavaScript-to-Java LiveConnect. But it was triggered by changes that were made silently by Mozilla.org to the nsIScriptSecurityManager interface, on the "trunk" (as opposed to the "Aviary" and "Mozilla 1.7" "branches") on 2004-04-25 and again on 2005-02-02. The effect of these changes is, in the worst case, to completely turn off security for JavaScript-to-Java LiveConnect.
Because JavaScript-to-Java LiveConnect appeared to continue to function normally, I was unaware of these changes until very recently.
Similar changes were made, also silently, to the nsIScriptSecurityManager interface on the "Aviary" branch (from which Firefox releases are currently being made) on 2004-10-24, and first appeared in Firefox 1.0RC1 -- but these changes caused the MRJ Plugin to crash, so I became aware of them very quickly. Version 0.8.7 of the Java Embedding Plugin contained a workaround for them. The "Aviary" changes were ported to the "Mozilla 1.7" branch on 2004-12-03, for the Mozilla 1.7.5 release. But by then (of course) the JEP already contained a workaround, so they weren't a problem. For more information see:
http://sourceforge.net/tracker/index.php?func=detail&aid=1063222&group_id=107955&atid=649116
https://bugzilla.mozilla.org/show_bug.cgi?id=234169#c75
The nsIScriptSecurityManager interface is used by a plugin to find the
browser's JavaScript "security manager", so that it can know how to
set up security for a JavaScript-to-Java LiveConnect call. Sometimes
the security manager will (in effect) tell the browser that it's doing
JavaScript-to-Java LiveConnect "on its own account", and that no
additional security checks are needed. On these occasions it's normal
to skip these security checks.
However, if the nsIScriptSecurityManager interface has changed, the
plugin may no longer know how to find the JavaScript security manager.
In this case the plugin should prevent any JavaScript-to-Java LiveConnect
calls from taking place. This is how the MRJ Plugin JEP now behaves (as of
version 0.9.2). But previous versions allowed the call to proceed without
any security checks. This is the hole -- earlier versions of the MRJ Plugin
behaved the same way when it couldn't find a security manager as when the
security manager told it no security checks were needed.
I've said that only in the worst case does this hole completely turn off
security for a JavaScript-to-Java LiveConnect call. The reason is that Java
security has many layers, and is quite redundant. Only in the worst case
will not having any security checks on the initial call from JavaScript into
Java result in Java's security being completely disabled.
I have opened the following bug at bugzilla.mozilla.org. I hope to use it
to keep track of changes to Mozilla's browser security interface.
https://bugzilla.mozilla.org/show_bug.cgi?id=293973
(An earlier version of this document claimed that the JavaScript security
manager sometimes isn't present. But this turns out not to be true -- the
JavaScript security manager is always present and accessible, if you have
the information you need to find it.)