Possibly crazy idea to deal with memory layout problems once and for all

[Troy Rollo]

I have been looking at a problem that has arisen in recent versions, 
particularly when using some D3D games, in which the virtual address space 
above TASK_UNMAPPED_BASE becomes fragmented to the extent that eventually you 
get an out of memory condition, even though you still have well over a 
gigabyte free (even in a contiguous block!). This has led to some people 
using the attached hack (first attachment) to get their D3D games to work.

After considering a lot of approaches to this problem that did not seem to get 
anywhere productive, it seemed that the only way to resolve it would be to 
get in between the application and any libraries, and the kernel. The 
solution I came up with is somewhat rude, but if we assume that any libraries 
that Wine uses only call mmap() and associated calls via the C library, then 
it may well do the trick, plus make it possible to implement other virtual 
memory APIs more effectively (and in a couple of cases eliminate calls to 
wineserver).

If wine_preloader were extended to have its own implementation of all the 
friends of mmap(), and to have its own implementation of the dynamic linker, 
then in principle it could make sure only its mmap (and not the C library's) 
is called. An even more aggressive approach might be to load the C library 
and stick jumps into its mmap that redirect to the preloader's versions. The 
preloader's mmap would keep track of mappings on its own and when it receives 
an mmap with a start address of "NULL", decides on its own what base address 
to use.

Any thoughts?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.patch
Type: text/x-diff
Size: 994 bytes
Desc: not available
Url : http://www.winehq.org/pipermail/wine-devel/attachments/20060218/1b0f68bc/mmap.patch
-------------- next part --------------
7c000000-7c001000 r-xp 00001000 08:11 21                                 /home/troy/staticthing
7c001000-7c002000 rwxp 00001000 08:11 21                                 /home/troy/staticthing
ffffd000-ffffe000 rwxp ffffd000 00:00 0
ffffe000-fffff000 ---p 00000000 00:00 0