About the author

Tomasz Rybicki is a Ph.D. candidate at the Department of Electronics and Information Technology at the Warsaw University of Technology. He is a member of MEAG (the Mobile and Embedded Applications Group - http://meag.tele.pw.edu.pl). He has been a Java programmer for over five years.

What you will learn...

• how the security model of the Java virtual machine works,

• how sandbox vulnerabilities can be exploited,

• how to perform an attack to gain direct memory access,

• how to conduct a differential analysis of power absorption,

• how to secure applications through bytecode instrumentation,

• how Java VM audits are conducted.

What you should know...

• how to program Java applications.

Leaking virtual machines

Vulnerabilities in Java virtual machines have existed as long as Java itself. Here is a list of some of the more serious errors found in various Java VM implementations. Even though the errors are very well known - some of them affect very old versions - they still make valuable reading.

In 1996, an error was found which made it possible to load a non-trusted (remote) class with the same privileges as a trusted (system) class. The error was that the VM allowed class names to begin with the \ character (backslash), which made the ClassLoader treat remote (i.e. potentially dangerous) classes in the same way as trusted local classes. The error affected JDK 1.0.

In 1997, an error was found in the applet signing mechanism. The bug affected JDK 1.1 and allowed a malicious signed applet to increase its privileges. The problem was that the method which was supposed to return a copy of the array containing applet signer identifiers returned the actual array of identifiers, which potentially made it possible for an applet to add signers to the array.

In 1999, an error was discovered in the bytecode verifier of Microsoft's implementation of the Java VM. A special sequence of bytecode instructions (see Frame .class file format) made it possible to bypass type casting control in event handler instructions. The error affected Internet Explorer 4.0 and 5.0.

In the year 2000, a VM bug was found which made it possible for an applet to establish connections with any server, not just the one it was loaded from. The error was that methods responsible for opening and closing socket connections were all treated as trusted and thus not verified by the Security Manager. The error affected Netscape Navigator and Netscape Communicator 4.0-4.74.

In 2002, a verifier error was discovered which made it possible to call a superclass method from a class other than a subclass. The error affected Internet Explorer 4.0, 5.0, 6.0 and JDK 1.1, 1.2 and 1.3. Yet another vulnerability allowed untrusted code (an applet) to create a fully functional ClassLoader system object. The error affected Internet Explorer 4.0, 5.0 and 6.0.

Java language features

The virtual machine itself contains a number of safeguards for ensuring the stability and security of the code being executed. One of these is the lack of pointers - the programmer has no direct memory access, so there is no way to insert any instructions previously unchecked by the VM into memory. Java also controls references to array elements, making it impossible to gain illegitimate memory access for instance by referring to the n+1-st element of an n-element array. Executing the code below causes IndexArrayOutOfBoundsException to be thrown:

int [10] x= {0,1,2,3,4,5,6,7,8,9};

int y = x[10];

Java also controls type casting and only allows legitimate implicit casting operations. This means that the following code:

int x;

byte y=3;

x=y;

will compile correctly, unlike this:

int x;

double y=3;

x=y;

In the second case, explicit casting is necessary: x=(int) y. Explicit casts are taken over by the compiler and conducted safely.

The garbage collector also increases code security by cleaning up all unreferenced objects, so no program can leave surprises in memory for applications which run later. The same applies to null pointer references - any attempt to access a null object will result in a null pointer exception being thrown.

Memory access control is a vital matter. To start with, any application that writes unchecked data to memory is a potential threat to system stability - with luck, only the application itself risks crashing, but in the worst case it can take other processes with it. If the application is executed on a server, with mission-critical processes running alongside, the scenario is completely unacceptable.

Also, a malicious programmer could potentially overwrite memory segments containing data belonging to vital (for instance security-related) processes, such as system processes or even the virtual machine itself. This would allow the intruder to substitute the Security Manager or ClassLoader, thus leading to a critical system threat.