...

Your Win~1 (or the video driver, doesn't matter) does not fully support graphics programs running in a window.

I tried VESA and I tried VGA, all the same. But that Windows generally does not support graphics [...] isn't true.

I have that very problem from the time when vpic (remember?) was the best picture viewer for any MSDOS based systems. The VGA-mode GIF-s were OK, but any JPEG-s terminated vpic with an "This application can not be run in windowed mode" error.

But there is something more wrong. If I use GRX two times within RHide, I cannot link anymore. The linker says "memory exhausted" and I have to end and restart RHide. [...] So I think somehow GRX does not clean up it's resources very good.

That may well be. I'm using a command-line "make" (bcc, djgpp and mingw) under Win~1, and everything works fine. Can you please debug GRX? BTW, keep in mind that Win~1 looses DPMI selectors at a _very_ high rate (see djgpp docs) and cleanups them on application exit only - could that be your problem?