In message 199805230034.CAA00911@esmeralda.gerwinski.deyou write:

According to Mike Stump:

...

1.3 is horribly old, and I don't recommend you use it.

About the same holds for the test suite that comes with the GPC source.

:-)

I haven't looked at the gpc testsuite yet in any detail I'll go ahead and describe how the "torture test" harness works that we currently use for gcc & g77 tests.

The first thing you do is divide your tests into a few classes:

* Tests which should compile & execute. These tests should call abort when they fail or exit (0) when they pass.

* Tests which should merely compile.

* Test for syntax errors and such which should not compile.

These tests are placed in different directories (execute, compile and noncompile respectively). In each directory there's a test driver which is pretty generic.

There's a few language issues in the test drivers, but I can help deal with them if someone wants to take a stab at making the gpc testsuite "dejagnu aware".

There are more complicated testing harnesses, but for the majority of tests the "torture" harness is appropriate and easy to set up and use.

jeff

In message 199805272159.XAA00553@esmeralda.gerwinski.deyou write:

Hello,

I think that the `gpc-test.tar.gz' test suite should become "the" test suite for GPC. It is much more complete than the existing DejaGNU test suite.

What I'm talking about is converting that testsuite to the dejagnu format. The other possibility is to write a dejagnu driver to run that testsuite. However, that's generally more work than sorting out the existing tests into a "torture" framework.

...

• Tests which should compile & execute. These tests should

call abort when they fail or exit (0) when they pass.

For the current test suite it is important that the test programs produce the correct output.

We could probably enhance the testsuite to check for specific output of a test -- we already do that for stuff like libio/libstdc++.

Or you tweak the tests to put their output into a string, then have the test check the contents of the string for correctness and do the right thing re: exit/abort. Several of the C torture tests do this kind of thing.

Let's hope that it is able to give some special flags to the compiler which are needed for that specific test (marked with "FLAG" in the test source).

What kind of flags do you need to give to trigger the test? THe torture framework is design to run through a set of common flags (usually optimization options) for all tests.

There are ways to pass specific flags to specific tests, but it's frowned upon in the torture framework.

jeff

In message 199805300919.LAA00481@esmeralda.gerwinski.deyou write:

According to Jeffrey A Law:

...

Or you tweak the tests to put their output into a string, then have the test check the contents of the string for correctness and do the right thing re: exit/abort. [...]

Some tests are specifically testing whether the output to a string is really equivalent to output to a file. One would, in this example, have to read the contents of the file into another string and compare, but it is a lot of work to re-write all this.

I certainly understand. One possibility to avoid rewriting all that stuff would be a relatively simple extension to the torture framework that had it compare an output file with an expected output file.

That's how some of the libio/libstdc++ tests work, we'd just basically have to pick out the pieces we need. Not a major problem (IMHO).

Think of C programs that behave differently with `--traditional', `--ansi', or `--strict-ansi'. GNU Pascal has to comply to three major dialects (ISO-7185, ISO-10206, BP 7.0) with subdialects (with/without "extended syntax" or Boolean short-circuit operations or whatever), and they are becoming even more (ANSI Object Pascal, Delphi).

OK. Is there any way to annotate the testcase itself for the extra options that it needs (like a comment in the source)?

We could handle this similar to how the torture framework handles loops:

The framework looks at the source file for loop constructs, and if they're found it arranges for additional tests with loop unrolling options.

A similar mechanism could be used to annote the source to note what dialect of Pascal is used (and thus indicate special flags for the test).

One possible solution would be to have a subdirectory for each important combination of flags. Multiply this with the number of directories we must have anyway (run, compile, noncompile) and we are left with ~100 directories.

Yea, this gets out of hand! :-)

Move the existing GPC test suite

ftp://agnes.dida.physik.uni-essen.de/home/peter/gpc-test.tar.gz

into the GPC source distribution, replacing the obsolete tests in `test'. Take the stuff in `testsuite' as a starting point for a DejaGNU-based testsuite for GPC. Someone can move the tests from `test' into `testsuite' step by step. And we should rename those directories to reduce the confusion.

This is quite reasonable. And if someone wants some help on extending the torture framework to handle the stuff mentiond above I can help.

jeff