The following behaviour appears to be an IE quirk as the RichTextBox editor is simply an IFrame element.
In HTML mode if you clear all content and then paste in javascript it will disappear from the editor document even though it still exists in the document content on postback.
If you type some text in the editor and then copy the javascript in Html mode after the text or if you leave the <P> </P> created by the editor and paste under this in Html mode then it retains the javascript so that you're able to see it. If you clear the editor and paste the script then it's not displayed even though it does exist in the document. It exhibits the same behavior for a <style> tag in the document.
The RichTextBox property HtmlOutputStripJavaScript runs server side processing so if you set HtmlOutputStripJavaScript = true; and click save after pasting javascript (with some preceding text or <P> </P> as mentioned above) you'll see the javascript stripped on postback however if it's set HtmlOutputStripJavaScript = false; then it's not stripped on postback, so this property is applying the javascript strip method as we'd expect.
So the bottom line is you need some content or an Html element (like <P> </P>) prior to your javascript so that it displays in Html mode.