Prof. A Olowofoyeku (The African Chief) wrote:

On 21 Jul 2005 at 18:43, Silvio a Beccara wrote:

...

Hello,

is it possible to increase heap dimensions? I am running a GPC program which uses "big" dynamic arrays of double precision floats (up to 200 x 500) and, while it runs fine with smaller dimensions, from 128 x 256 up I get the following error:

/u/dms/abeccara/pimc/parpi/pimb: out of heap when allocating 629145608 bytes (error #853 at 8071a32)

Sounds like a unix quota thing. Can't remember the command for changing default stack and heap allocations (there is a command line tool for it) - perhaps unix gurus here will remember?

Depends on the shell, unfortunately:

csh, tcsh: limit

most other shells (sh, bash, etc.): ulimit

There are "soft" and "hard" quotas. Normal users can increase the soft quota up to the hard quota. Only root can increase the hard quota.

Frank

On Fri, 22 Jul 2005, Silvio a Beccara wrote:

Thanks everybody for the advice,

I'd just like to point out the fact that I wasn't getting the error with GPC 2.1, but I must also say that the system I was running on was slightly different (I can talk to the administrator about this). I'll first see if I can reproduce the error on a different system, to see whether it depends on quota.

As for the number of arrays used, there are quite a few, about a hundred. As for possible bugs in my code, I can reproduce here the part where I declare and initialise the arrays: maybe something's wrong, or something I don't know changed from GPC 2.1.

...

GPC should not limit your memory use -- at the bottom layer it just uses `malloc'. Note that seemingly small change to the system can significantly change memory allocation. So first check that all other factors are equal.

If you are allocating _one_ array of 630Mb then the failure is very strange, even if something else has changed.

Box here is 2 cpu's with 2gb *shared* memory. I got exactly what I expected:

ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 16380 virtual memory (kbytes, -v) unlimited

free total used free shared buffers cached Mem: 2074748 228500 1846248 0 28988 170524 -/+ buffers/cache: 28988 2045760 Swap: 0 0 0

Next, ran this simple program:

program heaptest; var i: integer; buf: pointer; siz: cardinal;

begin for i := 1 to 255 do begin siz := i * 255 * 255 * 255; getmem( buf, siz); writeln("allocated ", siz," bytes"); freemem( buf ); end; end.

The output was: [..] allocated 1989765000 bytes allocated 2006346375 bytes allocated 2022927750 bytes heaptest: out of heap when allocating 2039509125 bytes (error #853 at 8049e61)

Hope this helps Russ

Hi Russell,

thanks for the hints. I tried to compile and run your program, and I get a very similar result:

[...] allocated 2105834625 bytes allocated 2122416000 bytes allocated 2138997375 bytes ./heaptest: value out of range (error #300 at 8049dbc)

so I still don't understand why I was getting the error at a lower limit. Here are the results for my box. The memory is the same as yours, and it is shared, since this is a two-cpu box:

[abeccara@baciuco 4-128]>free total used free shared buffers cached Mem: 2055276 2015180 40096 0 288120 1410364 -/+ buffers/cache: 316696 1738580 Swap: 2048216 0 2048216

(now none is free since it's running a job), and also the limits are quite similar:

[abeccara@baciuco 4-128]>ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) 4 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 7168 virtual memory (kbytes, -v) unlimited

Best regards

Silvio a Beccara

Box here is 2 cpu's with 2gb *shared* memory. I got exactly what I expected:

ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 16380 virtual memory (kbytes, -v) unlimited

free total used free shared buffers cached Mem: 2074748 228500 1846248 0 28988 170524 -/+ buffers/cache: 28988 2045760 Swap: 0 0 0

Next, ran this simple program:

program heaptest; var i: integer; buf: pointer; siz: cardinal;

begin for i := 1 to 255 do begin siz := i * 255 * 255 * 255; getmem( buf, siz); writeln("allocated ", siz," bytes"); freemem( buf ); end; end.

The output was: [..] allocated 1989765000 bytes allocated 2006346375 bytes allocated 2022927750 bytes heaptest: out of heap when allocating 2039509125 bytes (error #853 at 8049e61)

Hope this helps Russ

--

Email.it, the professional e-mail, gratis per te: http://www.email.it/f

Sponsor:

Telefona con TELE2. Risparmia sulle tue telefonate urbane, interurbane, internazionali e verso i cellulari

Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=3756&d=25-7

On Mon, 25 Jul 2005, Frank Heckenbach wrote:

Silvio a Beccara wrote:

...

thanks for the hints. I tried to compile and run your program, and I get a very similar result:

[...] allocated 2105834625 bytes allocated 2122416000 bytes allocated 2138997375 bytes ./heaptest: value out of range (error #300 at 8049dbc)

That's no memory allocation error, but a range-check error. The computed size (next would be 130 * 255 * 255 * 255) doesn't fit in a 32 bit signed integer anymore. If you want to allocate more than 2 GB in a single allocation (which was not your original request), you could try to use an unsigned type (Cardinal or perhaps better SizeType). However, I've never tested this, so there may be RTS or libc issues.

It's an RTS issue. From error.pas:

procedure RuntimeErrorInteger (n: Integer; i: MedInt); begin SetReturnAddress (ReturnAddress (0)); ErrorMessageString := FormatStr (GetErrorMessage (n), Integer2String (i)); EndRuntimeError (n) end;

On my box "heaptest" quit at less then max MedInt so I got the expected error message:

...

...

heaptest: out of heap when allocating 2039509125 bytes (error #853 at 8049e61)

Russ

That's no memory allocation error, but a range-check error. The computed size (next would be 130 * 255 * 255 * 255) doesn't fit in a 32 bit signed integer anymore. If you want to allocate more than 2 GB in a single allocation (which was not your original request), you could try to use an unsigned type (Cardinal or perhaps better SizeType). However, I've never tested this, so there may be RTS or libc issues.

what does in this context mean "range check error"? Sometimes I get the same error when using dynamic arrays, usually when bounds are trespassed. Is there a way to know at which code line the error generates, and does the error have consequences on the results of the code?

Silvio

--

Email.it, the professional e-mail, gratis per te: http://www.email.it/f

Sponsor:

Solo 10 Euro per chiamare in tutto il mondo!! Scopri il vantaggio di Email Phone Card, clicca subito

Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=2685&d=26-7

Hi Frank,

thanks for the hints: another question, is there a way of using run-gpc in a parallel environment, i.e. with mpi? In this case one runs programs by means of the script mpirun, but I see no way of using both at the same time... so maybe there's a compiler directive to give more verbose error messages, in order for one to be able to spot the errors without manually dissecting the code?

Silvio

...

...

That's no memory allocation error, but a range-check error. The computed size (next would be 130 * 255 * 255 * 255) doesn't fit in a 32 bit signed integer anymore. If you want to allocate more than 2 GB in a single allocation (which was not your original request), you could try to use an unsigned type (Cardinal or perhaps better SizeType). However, I've never tested this, so there may be RTS or libc issues.

what does in this context mean "range check error"?

The same as in other contexts: A value is out of range. 130 * 255 * 255 * 255 is bigger than what a 32 bit signed integer can hold (2^31 - 1).

...

Sometimes I get the same error when using dynamic arrays, usually when bounds are trespassed. Is there a way to know at which code line the error generates,

addr2line and run-gpc (you can search the archives for details).

...

and does the error have consequences on the results of the code?

Sure, it aborts. ;-)

If you disable range-checking, you get undefined behaviour. This can be anything from apparently correct operation, to minor or not so minor errors (wrap-around), to serious errors (memory trashing), to segfaults ...

Frank

--

Email.it, the professional e-mail, gratis per te: http://www.email.it/f

Sponsor:

Migliaia di manuali e guide scaricabili su: Hacker, Reti, Webmaster, Windows e non solo...clicca qui

Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=3412&d=26-7

Hi Frank,

I'm going to try, it looks feasible. Vielen Dank Silvio

However, the main thing that gpc-run is to run the program with the option --gpc-rts=-E... (you might also consider --gpc-rts=-F...; run a GPC compiled program with --gpc-rts=-h for details). This gives more information, in particular a backtrace. Afterwards, it uses addr2line to translate the error and backtrace locations into routine names and source code positions.

You can probably do the same things differently to integrate them with mpi. First, I suggest to play around with those options and addr2line manually, to get a feeling for the things and how to combine them with mpirun.

Frank

--

Email.it, the professional e-mail, gratis per te: http://www.email.it/f

Sponsor:

Natsabe.it significa Natura, Salute e Bellezza ... ma non solo!

* Vasta scelta di articoli da regalo. Belli e originali.

Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=1306&d=27-7

On 1 Aug 2005 at 15:26, Peter N Lewis wrote:

At 5:48 +0200 27/7/05, Frank Heckenbach wrote:

...

However, the main thing that gpc-run is to run the program with the option --gpc-rts=-E... (you might also consider --gpc-rts=-F...; run a GPC compiled program with --gpc-rts=-h for details). This gives more information, in particular a backtrace. Afterwards, it uses addr2line to translate the error and backtrace locations into routine names and source code positions.

Hmm, it looks like addr2line is part of binutils, and my feeble attempt to compile binutils on Mac OS X comes up short when ./configure says:

*** This configuration is not supported in the following subdirectories: bfd binutils ld gas opcodes gprof (Any other directories should still work fine.)

Unfortunately binutils is mentioned a lot so I haven't been able to track down if it is possible to get addr2line working under Mac OS X.

I don't support there are any suggestions on how to get the line based on the address other than addr2line?

gdb and trap on _p_RuntimeError works, and then a backtrace shows the stack with line numbers, but that means recreating the error a second time under gdb so it's not as quick for simple errors.

At the gdb command line, run "info line *<address>", where "address" is what you would have passed to addr2line

e.g.

info line *0x303033

Best regards, The Chief -------- Prof. Abimbola A. Olowofoyeku (The African Chief) web: http://www.greatchief.plus.com/

Peter N Lewis wrote:

...

At 5:48 +0200 27/7/05, Frank Heckenbach wrote:

...

However, the main thing that gpc-run is to run the program with the option --gpc-rts=-E... (you might also consider --gpc-rts=-F...; run a GPC compiled program with --gpc-rts=-h for details). This gives more information, in particular a backtrace. Afterwards, it uses addr2line to translate the error and backtrace locations into routine names and source code positions.

Hmm, it looks like addr2line is part of binutils, and my feeble attempt to compile binutils on Mac OS X comes up short when ./configure says:

*** This configuration is not supported in the following subdirectories: bfd binutils ld gas opcodes gprof (Any other directories should still work fine.)

Unfortunately, Apple still has its own hacked cctools for Mac OS X (and I didn't find a binutils package). We will have to ask on a Darwin mailing list.

Found it here http://binutils.darwinports.com/.

Regards,

Adriaan van Os