FULL PRODUCT VERSION :
java version "1.4.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-b92)
Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode)

FULL OPERATING SYSTEM VERSION :
 customer  Windows 2000 [Version 5.00.2195]

A DESCRIPTION OF THE PROBLEM :
Once a file has been mapped a number of operations on that
file will fail until the mapping has been released (e.g.
delete, truncating to a size less than the mapped area).
However the programmer can't control accurately the time at
which the unmapping takes place --- typically it depends on
the processing of finalization or a PhantomReference queue.

The  customer  associated with unmapping that is noted in the
JavaDoc could be avoided if when an area is unmapped, the
memory is not made available for reallocation but instead
is reserved (e.g. by using VirtualAlloc under Windows).
Then any unwanted reference to the previously mapped area
would result in an exception without incurring performance
penalties. The finalization process could release the
reservation when there were no remaining references to the
buffer.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Run the attached code.
2. A 1KB file will remain --- the File.delete call fails
unless the System.gc() is uncommented. However we can't
rely on the GC call actually doing anything.


EXPECTED VERSUS ACTUAL BEHAVIOR :
There is no way of ensuring that the delete will work even
though we are no longer interested in the mapping.
The deleteOnExit doesn't work either (with far less excuse)
and I have submitted a bug report for that.


REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------
import java.io.*;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

class TestMemoryMapping
{
	public static void main(String[] args)
	{
		try
		{
			File f = File.createTempFile("Test", null);
			f.deleteOnExit();
			RandomAccessFile raf = new RandomAccessFile(f, "rw");
			raf.setLength(1024);
			FileChannel channel = raf.getChannel();
			MappedByteBuffer buffer = channel.map
(FileChannel.MapMode.READ_WRITE, 0, 1024);
			channel.close();
			raf.close();
			buffer = null;
			// System.gc();
			if (f.delete())
				System.out.println("Temporary file
deleted: "+f);
			else
				System.out.println("Not yet deleted: "+f);
		}
		catch (IOException ex)
		{
			ex.printStackTrace();
		}
	}
}
---------- END SOURCE ----------
(Review ID: 153923) 
======================================================================
Posted Date : 2006-05-24 10:08:14.0

N/A

There is no unmap() method on mapped byte buffers because there is no known
technique for providing one without running into insurmountable security and
performance issues.

Suppose that a thread operating on behalf of Alice maps a file into memory and
then unmaps it.  A second thread operating on behalf of Bob then maps some
other file that the underlying operating system happens to assign to the same
memory address.  Now Alice's thread can read, and possibly even modify, the
contents of Bob's file.  Oops.

This problem could be avoided by defining a private volatile field in a mapped
buffer to indicate whether or not the mapping is still valid, and having every
access to the buffer's content first check this field and throw an appropriate
exception if the mapping is no longer valid.  This solution would significantly
slow down access, however, and fast access is the whole point of supporting
mapped buffers in the first place.

Another potential solution is to implement unmap() by unmapping the buffer's
memory region and then immediately remapping it from /dev/null, setting the
page protections so that any further access to the region would cause a SIGSEGV
(or the equivalent) to be raised and then translated into an appropriate
Java-level exception.

The remapped region would need to remain valid for as long as the original
buffer object is valid, eventually being released by the cleaner thread.  This
solution therefore wouldn't reduce virtual-memory footprint but it would at
least allow the underlying file to be manipulated on those operating systems
(e.g., Windows) that forbid certain file manipulations when valid mappings
exist.

To avoid the race condition between the unmap and remap operations we'd need to
globally synchronize all mapping operations within the same VM; this could,
theoretically, limit scalability on some higher-end systems.

The downfall of this second approach is that while we can avoid this race
condition within the VM we can't avoid it everywhere within the VM's process.
Memory-mapped files are pretty widely used these days, by everything from C's
stdio functions (on some systems) to dynamic linkers (on most Unix systems).
If a mapping operation were initiated by one of these subsystems, or by native
application code (perhaps unknowingly, via an intermediate library), at just
the wrong moment, and it happened to overlap our soon-to-be-remapped region,
then the remap operation would fail.

We at Sun have given this problem a lot of thought, both during the original
development of NIO and in the time since.  We have yet to come up with a way to
implement an unmap() method that's safe, efficient, and plausibly portable
across operating systems.  We've explored several other alternatives aside from
the two described above, but all of them were even more problematic.  We'd be
thrilled if someone could come up with a workable solution, so we'll leave this
bug open in the hope that it will attract attention from someone more clever
than we are.

With regard to the "workaround", described in the JDC comments, of (ab)using
reflection in order to invoke a mapped buffer's cleaner object: This is highly
inadvisable, to put it mildly.  It is exceedingly dangerous to forcibly unmap a
mapped byte buffer that's visible to Java code.  Doing so risks both the
security and stability of the system.

  xxxxx@xxxxx   2005-03-23 02:41:14 GMT
--

(2005-05-24) In mustang b86 there is a small update that may provide some interim relief for cases where the map method currently fails due to a resource issue. The bugID for this work is 6417205.
Posted Date : 2006-05-24 10:08:14.0

Add to this the bug    channel.map
(FileChannel.MapMode.READ_WRITE, 0, 2^24);
and I am trapped because I can´t use many 
MappedByteBuffers neither a big one

Sun, could you please give us another work around.

In September 19 , Mark Reinhold , said "You could certainly 
put a mapped buffer behind a soft reference; that would allow 
the GC to keep it around until there's enough memory 
pressure for it to be reclaimed".     In the last days I ´ve 
been testing this solution , but it does not work very well, at 
least in Windows 2000 Server. After some GC the whole 
system slows down. I also tried  MappedByteBuffer x=null , 
and GC but it seems that the OS still holds the map

I'm getting a related problem on HP. Multiple
MappedByteBuffers are being created in my prog, and
sometimes to the same file. If the previous one wasn't
cleaned up, the later one yields "Not enough space" IOException.

IMO, it's not a 'request for enhancement'. It's VERY serious
bug. In another words it's a stopship. How can it's possible
to design such systems??? For exapmle, you have a
constructor but you have not a destructor. You cannot
dispose the allocated memory and it's a 'request for
enhancement'... It's one system call to unmap the buffer
under Win32. And I suppose that it's also one system call
under  Linux...

How is "stop leaking memory" a request for enhancement?  Would it 
be above or below don't core dump?  I don't mean to be a jerk but 
come on.

With the latest drop of J2SDK 1.4.2 Beta System.gc
() "workaround" does not work anymore. As the matter of 
fact this getting a complete stopship for our application since 
we need truncate underlying physical file sometimes when 
application is running. It is not currently possible since the file 
is locked by mapped buffer I have no way to unmap.
The situation seems very a like to a (happily) imaginary: 
consider you have FileInputStream.open() but do not have 
FileInputStream.close().

Unfortunately it not only about memory leak but mostly about 
system resource leak (files that couldn't be deleted).

this bug should be in the Top 25 RFE's a long time ago!!!
why don't you update this and the Top 25 Bugs list??? please
fix this bug!!!

I agree with ,vladimirkondratyev, IMHO, It's a VERY SERIOUS 
_BUG_. So, Sun, could you please give us the bug FIX

why dont u fix this already?

For performing heavy access to very large files, I have tried 
to  use a bunch of MappedByteBuffers. I need more than can 
be allocated at once, so I *could* recycle the less important 
ones.  With no way to release them, this is does not work.

I agree with the other writers. This is a bug not an RFE. 

please fix it

How can Sun not acknowledge that there is a bug in their 
code? This considerably limits usages of MappedByteBuffer as 
you can't use it anymore as alayer between your program and 
the file system.

Anyway, I believe that request should rather read: make sure 
that close() unmaps the MappedByteBuffer. I do not see the 
grounds for an additional method that would put the 
MappedByteBuffer object into a "closed" state.

Well ,a YEAR has passed and the bug is there , so I had to 
change all the code I had written  and use simple file api , 
finally "NIO" and MappedByteBuffer it´s useless.

the following workaround seems to work:

public static void clean(final Object buffer) throws Exception {
        AccessController.doPrivileged(new PrivilegedAction() {
            public Object run() {
        try {
         Method getCleanerMethod = buffer.getClass
().getMethod("cleaner",
new Class[0]);
         getCleanerMethod.setAccessible(true);
         sun.misc.Cleaner cleaner =
(sun.misc.Cleaner)getCleanerMethod.invoke(buffer,new Object
[0]);
         cleaner.clean();
        } catch(Exception e) {
         e.printStackTrace();
        }
        return null;
       }
        });
  }
}

I think the API needs both an unmap() (or close()) method 
for java.nio.MappedMemoryBuffer and to automatically 
unmap (or close) all buffers when the close() method is 
called on a FileChannel object.

Without these enhancements, memory mapped files are 
severly limited in scope and can only be used in contexts 
where no control on the size of the file is necessary. This 
excludes many applications. For example, a text editor 
application using a mapped file has to allocate a buffer larger 
than the current file size in order to speed up its operations 
but it cannot truncate the file to its real (data) size when 
editing is finished and the file is closed within the application.

Alternatively, there could be a GC mechanism for mapped memory
analagous to GC for objects.  In both cases it is memory that is at
issue.  If nothing else, the VM could run System.gc() and 
runFinalizers() before giving up and throwing an exception.  That
would be more expensive than required theoretically, but at least
it would work until a more efficient solution came along.

There should also be a way to close/destroy a mapping without closing the file.

Assume that I have a database file with has a filelock on it's header. Now the database application needs to change the filesize or map another region of the file. If I need to close the file I will lose the file lock which may result in another instance accessing the file.

Because of this I'm still using RandomAccessFile.

To:  bellomi

Wow you are a god :)

I'm at a complete loss as to how mapped buffers could be used at all in a real application without this problem being fixed.  

At minimum, there should be large warnings in the API docs that indicate that they should only be used with files that don't need to be shortened or deleted.

Another example to the complete lack of care to us Sun has, would you open source already so we can have simple isues fixed in reasonable time (keep your compability I do not care, just open source the cvs tree, and accept external code).

Would it not be possible to have 2 implementations so that we can select which behaviour we need? aka keep the current implementation and create MappedByteBufferV2 or something that would be implemented as per option 1 in the evaluation?

That way, at least we as developers could choose whether we need the full-out performance of the current implementation vs. the ability to unmap...

I have a workaround that seems to be fine.   Like all workarounds its rather ugly.

What I do is I do my map in a loop.  If I get an exception due to no more mapping space available, I run System.gc(), System.runFinalizers(), and then try again.  I do this for up to 100 times (because dying really sucks, so I'd rather try really, really hard, and sometimes it has taken more than 20 or 30 iterations before the guilty garbage is finalized).

I mention this (1) so other folks could do it, and (2) because it indicates that this really is a problem related to garbage collection.  If the system knew that a mapped buffer was unreachable it could solve this by reaping it.

Obviously it can't know this a priori, but it seems to suggest a direction for thinking about the problem.

It also suggests a VM-level workaround.  If I can run GC before dying with this problem, the VM could do that internally for me so I don't have to.  And because it can interract directly with the GC subsystem it might be more efficient (for example, it might be able to directly force a full scale, complete sweep in a single operation rather than an arbitrary number of iterations).

Longer term one could think about the general intersection between GC and non-memory resources (mapped buffers, file descriptors, whatever).

If using a loop with gc and runFinalizers, it appears to help to include a short sleep as well. Nevertheless it can still take an inordinately long time before the mapping is released, and no guarantee that it will be. I've tried allocating large amounts of junk as well in an attempt to force the garbage collector into action, but this as tricky as you need to ensure that the 'junk' migrates out of the nursery area to be effective.

We've just run up against this, and I must say that I am disappointed at the lack of action.

This should definitely be a BUG rather than an RFE.

I agree this is a real hole and really limits the use of memory maps. However, with regard to some of the comments indicating sun should unmap when the file channel close() method is called is not correct. You don't need to have the channel open when using a memory mapped byte buffer... by design I might add. You could have thousands of mappings and you certainly don't want to keep each mapping file open while you are using it.

bellomi's solution of calling his clean() method, say just before setting the buffer to null, seems to work just great. Maybe Sun could just provide us an unmap method that must be used within a doPrivileged() runnable.

Let's brainstorm rather than make demands, shall we?

Since I haven't got the source of sun's MappedByteBuffer implementation, I would assume there is a pointer to the external memory somewhere, for the native code. Wouldn't it be possible to release the buffer then reset the pointer (to null, for example) so any subsequent access will be a violation/exception (which has to be checked anyway)? 

Let's brainstorm rather than make demands, shall we?

Since I haven't got the source of sun's MappedByteBuffer implementation, I would assume there is a pointer to the external memory somewhere, for the native code. Wouldn't it be possible to release the buffer then reset the pointer (to null, for example) so any subsequent access will be a violation/exception (which has to be checked anyway)? 

This is a very serious limitation to the usability of the mapped buffer. Having no way to release an allocated memory resource is a bug, not a "request for enhancement".

Are you sure that your current implementation works?

If it was implemented using Object.finalize() method, there was a simple way to bypass it. All the attacker had to do is to create another finalizable object, which reference the MappedByteBuffer, and then re-register it.

However, since it is implemented using PhantomReference, it seems safe. Is it?

Why does this have anything to do with security -- why does security even come into the argument?  If the underlying platform doesn't provide security between mapped files, then should Java really be trying to code around it?  And what if I happen to be programming something where I don't care about security (a desktop app)?  What's wrong with providing an unmap() method and just telling people that it may not be secure -- just outline the issues surrounding its use.  Are theoretical security problems unrelated to Java really the only thing holding this back?

I am using the workaround shown bellow to unmap buffers.

Everything works fine with multiple applications reading from the same growing local file, but there is a problem if the file is accessed using a Windows UNC path like "//machine/c$/dir/file": Buffer.get() returns 0 for some bytes near EOF in such files. 

This bug is reproductible even with 127.0.0.1 as the machine.

Should I open a new bug? I was not sure considering the workaround used to unmap has not been officially approved.

Note: adding a 1 second sleep after the buffer has been GC'd fixes the problem - but that's of course not always acceptable.

===

Workaround used:

private MappedByteBuffer buffer;
public void close() {
  WeakReference<MappedByteBuffer> bufferWeakRef = new WeakReference<MappedByteBuffer>(buffer);

  buffer = null;

  long start = System.currentTimeMillis();
  while (bufferWeakRef.get() != null) {
  if (System.currentTimeMillis() - start > GC_TIMEOUT_MS) {
    throw new RuntimeException("Timeout (" + GC_TIMEOUT_MS + " ms) reached while trying to GC mapped buffer");
  }
  System.gc();
  Thread.yield();
}

I don't understand Sun's position on this. Basically this makes Memory Mapped Files usable only in a limited set of applications, as a whole bunch of File related APIs fails if used on files there were memory mapped. As far as I'm concerned this will make me stay away from memory mapped files, and I start to have doubts on NIO itself: given that this is a serious design flaw, and not considered worth fixing, I don't know how many more there are.

I think it is clear that there must be some action on this issue. How about this idea: Implement a static method somewhere that is called "cleanUnreferencedMappings()". I have one case where I use a MappedByteBuffer only within a method. At the end ot the method I could just call that static method to make sure that the buffer has been cleaned. This would not be a 100% solution, but for many case a good compromise. I guess all this would take is tmove part of the sun.misc.Cleaner class to the MappedByteBuffer class.

Reading again through the bug description, I think there is some misunderstanding on this RFE. I think it actually should be split into two RFEs, like this:

1. Add an unmap() method to MappedByteBuffer
There seems to be no easy way of doing this as we can see from the evaluation of this bug, and I think this is what Sun's engineers have been focusing on.

2. Add a way to clean unreferenced MappedByteBuffers
This is what is asked for in the bud description and it seems to be what many developers are asking for. If a MappedByteBuffer is no longer referenced, there should be a way to make sure it gets cleaned up. This means that if you create a MappedByteBuffer on a file and then set the MappedbyteBuffer to null, there should be a way to guarantee that the MappedByteBuffer gets garbage collected and deleting the file is possible. This does not mean implementing an unmap() method (the buffer will be unreferenced, so it cannot be called anyway), but implementing a static method that cleans up all unreferenced MappedByteBuffers.

Each garbage collector must have a way of updating all the references to an object. Either by using a level of indirection or by searching the heap or something else. So it must be possible for the VM to find all the references to a MappedByteBuffer. 

Can the proposed unmap method cause the VM to go and set all the references to null? No one would be able to access that address space afterwards. I understand it's kind of an unusual behavior, but it doesn't seem to have the performance and security problems mentioned above. 

Users can be warned that calling unmap may cause NPEs if they try to use the buffer afterwards. It seems like a reasonable behavior to me.

This is a long post, so I'm going to break it up to get around the 4k character limit... whew.

Here's my workaround for this bug.  I'll try to outline what I was doing previously and then my workaround (NOTE:  This is not intended to be compiled).

Basically, I was creating an array of very large objects that was causing me to run out of heap space (which was set to a gigabyte using –Xmx1024m).  So I wanted some way to dump this data directly to the disk with the hopes that I would eventually be able to use this technique to serialize the object on the fly.

So, here is what I was doing (and hence the bug):
String[] fileNames; //an array of filenames
MappedByteBuffer[] buffers; //an array of buffers for the files

<some other code>
FileChannel tempChannel;
<loop through the filenames>
	fileName[i] = <some unique file name>;
	tempChannel = new RandomAccessFile(fileName[i], "rw").getChannel();
	//rowWidth*bufferRows is the size of my buffer
	buffers[i] = tempChannel.map(FileChannel.MapMode.READ_WRITE,0,rowWidth*bufferRows);

<more code>
<eventually, somewhere else in the code>
//here, the intent was to force a dump to the disk everytime the buffer was full
//note that the above variables were class variables
private void someWriteMethod(<stuff to write>) {
	MappedByteBuffer outBuff = <whatever buffer I need to write to>;

     if(outBuff.remaining() < rowWidth) { //no more room to write another row
		outBuff.force();	//force the disk write
		outBuff.clear();	//get the buffer ready
     }
	<outBuff.put calls>
}

Then to read stuff in I would just map the entire file to an input buffer:

<code stuffs>
FileChannel tempChannel;
MappedByteBuffer rBuff;
<loop through the file names>
	tempChannel = new RandomAccessFile(fileName[i], "r").getChannel();
	rBuff = tempChannel.map(FileChannel.MapMode.READ_ONLY,0,(int)tempChannel.size());

	while(rBuff.remaining() >= rowWidth) {
		<use gets to read in data>
	}

After I read in a particular file, I then wanted to delete it.  I was pretty aggravated I couldn't delete the file.  I was sure I had some dangling reference and then I found this bug report that confirmed it.

Refer to my previous post:

I just used ByteBuffer directly to get around this problem:
String[] fileNames;		//filenames again
FileChannel[] channels;	//now we have to keep all of our FileChannels
ByteBuffer[] buffers;	//buffers to write to

<some code>
<loop through all the file names>
	fileNames[i] = <some unique file name>;
	channels[i] = new RandomAccessFile(fileNames[i], "rw");	//get a file
     //probably could just use allocate() here
	buffers[i] = ByteBuffer.allocateDirect(rowWidth*bufferRows);

<somewhere else, later in the code>
private void someWriteMethod(<stuff to write>) {
	ByteBuffer outBuff = <whatever buffer I need to write to>;
	FileChannel outCh = <the associated FileChannel>;

	//here is where things get a little different
	if(outBuff.remaining() < rowWidth) {
		//we have to flip the buffer before the write
		outBuff.flip();
		outCh.write(outBuff);	//write the entire buffer
		outBuff.clear();	//get the buffer ready for data
	}
	<outBuff.put whatever's going in the buffer>
}

Later on to read and then delete the file:
FileChannel tempChannel;
ByteBuffer rBuff;

<loop through the files>
tempChannel = new RandomAccessFile(fileNames[i],"r").getChannel();
//allocate a buffer in memory
rBuff = ByteBuffer.allocate((int)tempChannel.size());
tempChannel.read(rBuff);	//load the data into the buffer
rBuff.rewind();	//set up the buffer for reading

while(rBuff.remaining() >= rowWidth) {
	<use gets to read data out of the buffer>
}

//cleanup and delete the file we just read
tempChannel.close();
rBuff = null;	//not sure this is necessary
File file = new File(fileName[i]);
file.delete();


This is a pretty good solution for working with large objects and getting around the map problem.  I'm pretty sure this is not a generic workaround, but it should be flexible enough to be a base for most workarounds.  Also, I could probably just keep an array of RandomAccessFiles instead of the strings and I'll probably convert it to that eventually.  I hope this helps with the workarounds.

Your evaluation of the bug seems like nonsense! Security issue? This code is only accessible from trusted code anyway! There is no way a user can get a MAPPED byte buffer without priviliages. 
Synchronization? 
Many API's in Java are not thread safe!
Get over it, there are valid use cases that need something like this! Stop inventing excuses for niech cases that don't happen and just implement this really simple request. Or at the very least implement the common use cases with security/synchronization for them only (such as resize(), delete() etc... and there when synchronized you can map to null just as others suggested).

Can the 6417205  bug fix be back ported to 1.4

Ok, any progress on this? I'm using latest Java 1.5 update as of this date and still having all these issues users are describing. 

This bug is now #5 on the RFE list and why no progress. Unmap method in the API - now please!! Cleanup unused mapped buffers, now would be good!!! Document the security risk and lets move on.

I've had to enhance the API on our SDK to allow user's to choose workarounds this bug. My goodness this is childish.

I like the reset any ByteBuffer references to mapped area to NULL solution during an unmap. It makes a lot of sense to me. 

Another one would be to do a call back to a user provided listener with the buffer references being unmapped allowing an application to prepare. Do a copy of the needed data to a normal buffer or something of that sort. My application already keeps track of mapped buffer in SoftReferences and it can update its reference accordingly without any special notifications.

My SDK is sofisticated enough where it can very cleanly handle such unmappings, if it has a reasonable prior notification or not. It will know how to reaquire the data after its been unreferenced by unmap.

My main concern is that my file "editor" needs to be synchronized with the physical file and in most cases where changes are in the middle of the file, it has to copy the editor contents with the original file to a copy, then unmapped the original file and rename the copy to the name of the original. 

In other cases it overrides only sections of the file or simplly appends the editor content if its only been modified by appends.

My project: http://jnetstream.sf.net

I would like to comment on some of the work-arounds submitted by users. None of them work reliably and in my case are not usable.

I use hundreds of jUnit test cases that create temporary files and do operations on those temp files to test my API. When files are mapped and need to be modified in any way, none of the work arounds work. 

They do work in some tests, but then there are always plenty that fail because they can not either truncate the file or remove it when changes are synched with the file by file to file COPY.

I can see that when you are dealing with a single file and after some grusome trials it is possible to manipulate a previously mapped file, but when dealing with lots of files, forget it.

Dear Sun programmists, why do you not want to implement the simplest solution? I suggested it in http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521677

You create two versions of MappedByteBuffer, returned by two versions of FileChannel.map method. 

The first version is your current implementation of MappedByteBuffer. It works as fast as possible, but does not support unmapping. So, some files, mapped by this way, maybe, will not deleted when the application will finish. For many applications, it is not too serious problem: usually it is possible to support a list of temporaty files and remove them at the next application start. But for applications that really need removing mapped files, you'll offer the second version.

The second version of MappedByteBuffer is modified: it contains "unmap" method (or can be passed to an equivalent FileChannel.unmap(ByteBuffer) method). This version is based on volatile internal flag, disabling any access from Java after unmap() call, and - really - it works slower. But this slowing down WILL NOT BE CRICICAL FOR MOST APPLICATIONS! Block reading/writing bytes (ByteBuffer.get/put(byte[], int, int) will work with almost same speed. Very low number of algorithms really require super-quick access to individual buffer elements. In all cases, known to me, either the calculations can be performed by blocks (for example, per 32768 bytes), or algorithm is complex enough and saving 30-40 CPU tacts per element is not important.

In any case, the users will have an ability to choose, what is more important for them: maximal performance of low-level random operations with separate buffer elements, or stable deletion of all temporaty files that were used for mapping.

I also hope you'll fix the bug with "Cleaner terminated abnormally" ASAP. It makes using your mapping practically impossible in serious applications, even if we don't need to remove files: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4938372 This bug is also in you top list.

Can you at least make sure deleteOnExit works for these files so we can avoid leaking temp files? Is there a reason for deleteOnExit to fail on deleting files with mapped areas?

The issue with deleteOnExit not working with mapped files on Windows is covered here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6359560

In short, the main conflict, that should be resolved, is the following. If a MappedByteBuffer is unmapped, it must be impossible to access its content by any ways, including parallel threads and some views of ByteBuffer. But additional synchronization in basic ByteBuffer methods - in particular, checking a volatile flag - is unallowed due to decreasing performance.

Please view the following solution. If I understand all correctly, it seems to be good enough: no risk and no decreasing performance.

The idea of my solution is based on the following proposals.

A) The problem exists in a multi-thread application. If we are sure, that only one thread works with byte buffers, then quick implementations, based on checking non-synchronized boolean flag "unmapped" (set in the public "unmap" or similar method and checked by all other methods), become possible.

B) If some thread B never addressed to any class from java.nio package BEFORE this moment, it cannot access to MappedByteBuffer content now. The fact of first addressing a class by the thread may be fixed in Sun's ClassLoader.

C) If some thread B will access to some synchronized (for example, volatile) variable, set by another thread A at some moment, then all other non-synchronized variables, set by A after that moment, will be also visible in B. So, if "unmap()" and ClassLoader will access to such a variable, then we can be sure that any thread B, that will address java.nio package AFTER "unmap()" call in the thread A, will see non-synchronized flag "unmapped" set by the thread "A".

D) We really need unmapping not too often, usually before application exit. It is not a problem to arrange the application in so way, that "unmap()" or similar methods will be called only in situations, when only the current thread (maybe the last application thread) really used java.nio package before this moment.

So, let's suppose there is some global synchronized set "threadsUsingNIO". Every time when some thread first access  any class from java.nio package, the standard ClassLoader adds this thread to this set. Every time when a thread finishied, it removes itself from this set.

It allows to implement many possible solutions of the discussed problem.

The simplest example. Let every view of MappedByteBuffer contains a reference to the original MappedByteBuffer created by FileChannel.map method: "private MappedByteBuffer top = this;" (For original buffer, it refers to itself.) Then, let the MappedByteBuffer contains a private boolean flag "unmapped": "private boolean unmapped = false;". Every ByteBuffer method, as get/set, must check "top.unmapped" and throw an exception if it is true. (Such a check is very quick and will probably be removed from long loops by HotSpot compiler.) At least, let's create MappedByteBuffer.unmap() method:

    public final boolean unmap() {
        synchronized(threadsUsingNIO) {
            // All other threads that accesses threadsUsingNIO with the same synchronization
            // will "see" non-synchronized "unmapped" value after that access
            if (threadsUsingNIO.size() > 1) 
                return false; // failure: cannot unmap, because some other threads work with java.nio            
            top.unmapped = true;
            // Now only this thread works with java.nio.
            // If any other thread will try to access this buffer,
            // it will have to execute a code in ClassLoader for some java.nio.* class before,
            // and this code will also access to threadsUsingNIO with the same synchronization:
            // so, "top.unmapped" flags will be visible.
            top.cleaner().clean(); // or some another code that performs actual low-level unmapping
            return true;
        }
    }


Another possible solution, probably more convenient, is a special "dispose()" method in RandomAccessFile or FileChannel. This method should unmap all regions mapped by "map" method and, so, allow deletion and resizing the file. (As my "unmap()", this method returns false if there are some another threads working with java.nio). In this case, the "unmapped" flag should be a field of RandomAccessFile, and MappedByteBuffer.top should refer to this object.

At least, the suggested idea should allow to create stable deleteOnExit() method, that performs unmapping if there are no active threads working with java.nio package while performing shutdown hook.

Please comment what do you think about this. 

I've posted a separate RFE, describing this idea, yesterday (internal review ID: 935212), but, unfortunately, it was not published yet.

WHY DON'T YOU JUST SET THE CAPACITY OF THE BYTEBUFFER TO 0 IN THE UNMAP METHOD. SURELY JAVA IS DOING BOUNDS CHECKING ON EVERY BLOODY ACCESS ANYWAY.

In some cases Java will optimise the bound check away (hoist it outside a loop). It is able to do this because the bound is currntly a constant. So allowing it to change would make these optimisations impossible. Even then you might release the buffer between the bounds check and the attempt to access the buffer. So setting the capacity to zero does not provide a solution.

java.util.HashMap is not thread-safe. If you want it thread-safe you have to use a synchronized wrapper, or in 1.5+, one of the java.util.concurrent maps.

Somebody needs to get Doug Lea looking at this.

the source for directbytebuffer seems to indicate that every access is run through the checkIndex function which checks the index against limit. limit is not a constant. so doesn't directbytebuffer suffer the performance hit sun's trying to avoid??

Why not provide an unmap function with all kinds of warnings about how it can be unsafe, and let us decide if our applications need safety or need to release virtual memory and file locks.

You could even introduce it as deprecated so in most IDEs it will stick out as something that you should think twice about.

No unmap means I have to seriously consider moving to C++.

Forgive a terribly newbie question, but with regards to making the buffer thread-safe without including a volatile flag, why not make the first step of an unmap() operation to invalidate (zero out?) the address in the buffer?

The object is rendered useless. It can be easily determined that the object is useless. Moreover, the OS will raise some kind of exception that could be passed back as an NPE whenever the buffer was accessed after being unmapped.

On how many levels is this wrong? ;^)

It's coming up on FIVE years since this was first reported. I'm astounded that a usuable solution for this BUG (and it is a bug) is not yet available.  There is still not even a warning in the Javadocs that this problem exists!

I imagine that a high percentage of top end applications that actually need the kind of memory management offered by mapped files are running into difficulties because of this bug.

Add a delegater class for mmap file access, force gc on unmap ? 

RE: zeroing the buffer address (from newbie).
There are at least two problems. First, the zeroing may not be immediately visible to other threads (use of the buffer does not currently involve synchronization), so those threads will continue to use the previous value.
Secondly, the duplicate and slice methods create copies of the MappedByteBuffer which also contain the address. That is the address may appear in many objects which can't easily be found. Fixing this would require indirection and synchronization with associated performance implications.

So, I've been working on this program for a couple weeks, almost a month expecting to use byte buffers. Writing my own data structures (like the red-black tree) and so on because I didn't want to use JNI and C++ STL allocators to make my life easy.

Now I find out that, well, that's not really going to work because I can't unmap anything?  What the hell!?

Why not just make unmap() check security flags the same way an other function would, so that only the highest level code can run it?  

I mean if I can map and overwrite any file on the hard drive, why do I need to worry about this. 

INSANE!!!

Here's my wonderful workaround:

        data = null;
        System.gc();
        Thread.sleep(100);



Of course, java is open source now, right? So couldn't one of us just add an unwrap function to mappedbytebuffer ourselves? 

It failed once out of a few thousand tries.  Actually for a truely "reliable" work around you could do something like the following psudo-code:

boolean finished = false;
myBuffer = null;
System.gc();
while(!finished){
  try{
     //do whatever you are trying to do
    finished = true;
  }
  catch(java.io.IOException e){;}
}

You could add a time limit, etc. But the basic idea is, whenever you want try doing something that requires unmapping, you simply keep trying until it works. 

It's not a 'general' fix, because it depends on the filename originally mapped, as well as what you actually want to do with it (i.e. truncate, delete, whatever)

That's probably what I'm going to do from now on.

still not fun.

Here's a little hack I worked out to continue to try over and over again:

    public static void workaround(FileOp f) throws java.io.IOException{
        long sleep = 1; 
        long maxsleep = 1024*64;
        boolean success = false;
        while(!success){
            try{
                f.op();
                success = true;                
            }
            catch(java.io.IOException e){
                try{
                    Thread.sleep(sleep);
                }
                catch(Exception ex){
                    if(sleep > maxsleep){
                        throw e;
                    }
                    System.gc();
                    System.runFinalization();                    
                }
            }
        }
    }
    
    public static abstract class FileOp{
        public abstract void op() throws java.io.IOException;
    }
}

//and then in your code you do something like this:

                Bug4724038.workaround(new Bug4724038.FileOp(){
                    public void op()throws java.io.IOException{
                        resizeReloadFile(200000L);
                    }
                });

--Chad Okere

God I suck, the code should look like this:

public class Bug4724038 {
     
    public static void workaround(FileOp f) throws java.io.IOException{
        long sleep = 1; 
        long maxsleep = 1024*64;
        boolean success = false;
        while(!success){
            try{
                f.op();
                success = true;                
            }
            catch(java.io.IOException e){
                try{
                    Thread.sleep(sleep);
                }
                catch(Exception ex){}
                
                sleep *= 2;
                if(sleep > maxsleep){
                    throw e;
                }
                
                System.gc();
                System.runFinalization();                
            }
        }
    }
    
    public static abstract class FileOp{
        public abstract void op() throws java.io.IOException;
    }
}

-- Chad Okere

A few ways this can be done.  Admittedly, they're kind of ugly, but so is not being able to unmap your byte buffer until gc gets around to it...

1. FileChannel.map produces a new subclass of MappedByteBuffer for each individual instance.  There won't be a whole lot of instances in the system in most cases.  Then, the dispose function changes all the methods for that particular subclass to throw an exception when called, those methods get re-jit'd, and no one can use it.

2. A variant of System.gc() that guarantees that all unreachable instances of certain specially marked classes will be collected before it returns.  The set of specially marked classes is much smaller than the set of all garbage, so it'll be quicker, although the overhead of determining whether a particular object is reachable remains the same.

3. Probably the ugliest of all: reference count only instances of specially marked classes, allow them to be collected immediately if their reference count goes to zero.  They'll be collected eventually anyway no matter what, but if you detach all *direct* references to them, they'll get picked up right away.

Or maybe:

java.lang.ref.Reference gets a new method called "destroy()".  First it checks to make sure the referent is really not strongly reachable.  If the referent is strongly reachable, return false or throw an exception.  If the referent is strongly reachable, then finalize and collect it, then return true.

It's slow, its use is strongly discouraged, but if you have an object that needs to be disposed and it's a security risk to let any code access it in any way after it's been disposed and you don't want the overhead of checking a flag within the object while it's being used, then it's exactly what you need.

I know how to fix this w/o using any GC hacking. You will need a *native* unmap call for this to work.

0. Allocate a work buffer (you won't be able to use any allocators after 1).
1. Stop (suspend) all runnable threads but this one.
2. Unmap the memory.
3. Map a /dev/null or NUL memory map on top of it.
4. Restart all threads you stopped.