Re: Buffer Overrun Prevention in GPC

Mirsad Todorovac wrote:

In fact, this came to my mind in few considerations too: I was thinking of new breed of pointers that would allow for memory map defragmentation. As said somewhere in theoretical books, after millions of seemingly random allocations/deallocations - memory looks completely fragmented, and wtih equally distributed holes and used spaces. This as you must have noted means that actual system memory used is at least twice the malloc() allocated memory, since the holes cannot be defragmented or compacted.

This is why I would like to propose "registered pointers" (maybe it came subconsciously from Java?) and memory allocation library that would have a better heap use and corruption detection.

Registered pointers are close in semantics to pointers with ranges already intorduced on the list, but they should also allow defragmentation of the memory on-the-fly, compacting of free space and deallocating and returning ampped pages back to system (unlike current memory allocators that behave like my country's budget: they only grow and become less efficiently used with time ...).

Is that too difficult to implement?

(I am sorry if this has been discussed already.)

To explain the idea more visibly, as we are all probably tired at 4 PM:

Every pointer would be registered in a pointer table or list
*depending on implementation*

Pointers would be indirect, used through a mechanism that would  

allow transparent moving of memory fragment pointed to.

A garbage collection mechanism would browse through all pointers  

and defragment memory, much like a file system, moving used memory towards beginning, free memory towards end of memory mapping, so it could be later munmap()-ed and returned to OS when certain LOW_WATERMARK or threshold is reached

Unused fragments of memory that are not explicitly free()-ed could  

be deallocated automatically and an optional warning could be generated

Safe garbage collection memory allocator could and probably should  

be mapped in separate address range from "normal" heap memory mapping that permits direct use and hacking of pointers

Better heap use would allow for introducing "heap canary" technique
that would eliminate another source of exploits

All could be done transparent, without changing existing code

Better heap integirty checking would be introduced to existing
programs, detecting memory leaks as warnings

That's the general idea, but I haven't tried implementation yet.

Compatible programs could simply be linked with new library and have the advantage of better memory use, better heap integrity and possibly longer uptime without need to reboot system because of memory leaks because of fragmentation.

I went far from original question, but this idea is tempting me for a long time, and GPC development team seems open enough :-) that I thought of requesting implementation or putting it on the wish list.

See if <http://developer.apple.com/documentation/Performance/Conceptual/ ManagingMemory/Articles/FindingLeaks.html> and <http://developer.apple.com/documentation/Performance/Conceptual/ ManagingMemory/Articles/MallocDebug.html> already do part of what you have in mind (assuming you are running gpc on Mac OS X).

Regards,

Adriaan van Os