L.S.
I was happy to notice that the memory leak in the file-io is fixed, after I succesfully compiled and installed gpc version 20020910, based on gcc-2.95.2 19991024 (release).
However, currently I am compiling a FE program using the automake option, and it turns out that the build is recursively. Creating a log file (using gpc -v and editing a bit) yields the following result:
1 /tmp/ccAHxoUT.i 2 ch_const.p 3 general.p 4 NODES/nodes.p 5 MATERIAL/constitution.p 6 NODES/nodes.p 7 MATERIAL/constitution.p 8 ELEMENTS/elements.p 9 NODES/nodes.p 10 MATERIAL/constitution.p 11 NODES/nodes.p 12 MATERIAL/constitution.p 13 ELEMENTS/elements.p 14 global.p 15 NODES/nodes.p 16 NODES/SCAL/scalnode.p ...
The main program uses the units ch_const, general, initnode, ... where initnode uses nodes, constitution, elements, ... whele elements uses nodes, constitution, ... Looking at the log file, NODES/nodes.p is compiled multiple times due to the recursive nature of automake.
That this really is a problem can be seen from the fact that, according to top, the size of gpc1 is 611M, and running for 2920min, whereas a previous version of gpc only took minutes. However, that previous version creates a memory leak. More figures on the size of the problem: gpcpp has been invoked more than 250000 times, resulting in 213 units that are compiled.
The source of the problem is the recursion. Take the following worst case scenario: unit 1. unit 2 uses unit 1 unit 3 uses unit 1,2 unit 4 uses unit 1,2,3 unit 5 uses unit 1,2,3,4 .... the program uses unit 1....n
Compilation takes 1 step for unit 1, 2 for unit 2 (namely 1 for unit 1 and 1 for unit 2), 4 for unit 3 ( namely 1 for unit 1, 2 for unit 2,1 for unit 3), 8 for unit 4 (namly 1 for unit 1, 2 for unit 2, 4 for unit 3 and 1 for unit 4)
By induction, the program requires 2^n steps. Now 250000 is negligible compared to 2^(more than 213), so my whining on the compilation effort should be taken with a bit of salt. The situation could be far worse ;-)
This recursion scenario makes sure that the object files of the units are always based on the latest versions of the source.
Anyway, I would like to suggest the following method (which I hope is easy to implement, and I hope I did not overlook some problem scenario) Basically, it boiles down to pruning the recursive branches that are done earlier.
Create an empty list of up-to-date object files for each unit that is required do: procedure create_object_file * verify that the units it requires are on the list, if not, call create_object_file for that unit * compile the unit.
In this case, compilation of unit 5 takes considerable less steps: 1 for unit 1, 2 for unit 2 (1 to see unit 1 is up to date, 1 for unit 2), 3 for unit 3 (2 to see unit 1 and 2 are up to date, 1 for unit 3), 4 for unit 4 (3 to see unit 1,2 and 3 are up to date, 1 for unit 4), and 1 for unit 5. This requires 5 unit compilations, and 6 `up to date'-checks in the list. Thus n(n+1)/2 + 1 steps are required to compile the program, of which n+1 compilations and n(n-1)/2 `up to date' checks. The savings are in the fact that an `up to date' check of, e.g., unit 4 does not require the `up to date' checks of its dependencies, since it is already in the `up to date' list after its compilation.
Based on these considerations, only 22792 steps would be required, whereas now more than 250000 gpcpp invocations occurred.
Regretfully, I do not have the time to start hacking around in gpc myself, so I hope you have some use for these comments. For the time being, I just compile the units in exactly the sequence as they occur in the logfile, and build without automake ;-)
Regards,
Marten Jan de Ruiter Structural Optmimzation and Computational Mechanics group, Delft University of Technology
Marten Jan de Ruiter wrote:
However, currently I am compiling a FE program using the automake option, and it turns out that the build is recursively. Creating a log file (using gpc -v and editing a bit) yields the following result:
1 /tmp/ccAHxoUT.i 2 ch_const.p 3 general.p 4 NODES/nodes.p 5 MATERIAL/constitution.p 6 NODES/nodes.p 7 MATERIAL/constitution.p 8 ELEMENTS/elements.p 9 NODES/nodes.p 10 MATERIAL/constitution.p 11 NODES/nodes.p 12 MATERIAL/constitution.p 13 ELEMENTS/elements.p 14 global.p 15 NODES/nodes.p 16 NODES/SCAL/scalnode.p ...
The main program uses the units ch_const, general, initnode, ... where initnode uses nodes, constitution, elements, ... whele elements uses nodes, constitution, ... Looking at the log file, NODES/nodes.p is compiled multiple times due to the recursive nature of automake.
I can't reproduce this. If I create the attached files according to this description, and look for `gpc1', each of the 7 files is compiled exactly once.
Invocations of `gpcpp' do not always mean compilation. Some of them are needed to find possible include files which must be taken into account when deciding whether to recompile. It's possible that this happens more often (here: 13 times in the worst case I found), though I'd be surprised if it's more than quadratic since indirect imports shouldn't be considered. If you can show me such an example (maybe just the import structure of your project without the actual code), I'll try to fix it.
In the future, the internal automake mechanism will be replaced by an external tool which can "oversee" things better and save unnecessary gpcpp invocations, much like you suggested.
That this really is a problem can be seen from the fact that, according to top, the size of gpc1 is 611M, and running for 2920min, whereas a previous version of gpc only took minutes.
This looke like another kind of problem since the number of compiler invocations should not have much influence on the size of a single one.
Is one of the generated gpi files very large? If so, you might want to try building the gpidump utility (in p/utils in the GPC sources) and apply it to that file to see if there's a problem there (such as the same imports listed again and again).
Frank