Commit graph

800 commits

Author SHA1 Message Date
Dan Fandrich
3d75029859 runtests: fix -c option when run with valgrind
The curl binary argument wasn't being quoted properly. This seems to
have broken at some point after quoting was added in commit 606b29fe.

Reported-by: Daniel Stenberg
Ref: #11073
Fixes #11074
Closes #11076
2023-05-05 09:52:38 -07:00
Dan Fandrich
faebcee349 runtests: support creating more than one runner process
The controller currently only creates and uses one, but more are now
possible.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
38465f9a55 runtests: spawn a new process for the test runner
When the -j option is given, a new process is spawned in which the test
programs are run and from which test servers are started. Only one
process can be started at once, but this is sufficient to test that the
infrastructure can isolate those functions in a new task. There should
be no visible difference between the two modes at the moment.

Ref: #10818
Closes #11064
2023-05-05 00:45:43 -07:00
Dan Fandrich
d4a1b5b60c runtests: turn singletest() into a state machine
This allows it to run in a non-blocking manner.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
a98277fcc7 runtests: change runner interface to be asynchronous
Program arguments are marshalled and then written to the end of a pipe
which is later read from and the arguments unmarshalled before the
desired function is called normally.  The function return values are
then marshalled and written into another pipe when is later read from
and unmarshalled before being returned to the caller.

The implementation is currently blocking but can be made non-blocking
without any changes to the API.  This allows calling multiple runners
without blocking in the future.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
0754de758a runtests: call citest_finishtest in singletest
This is where citest_starttest is called.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
d4d22adf40 runtests: add a runner initialization function
This sets up the runner environment to start running tests.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
82fa5ca836 runtests: remove directory from server filename variables
There will soon be multiple log directories so the paths will no longer
be static in runtests.pl. Also, get rid of $SERVER2IN which was not
used.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
b71a0c3ec9 runtests: reduce package exports after refactoring
Some recent refactoring made these export no longer necessary. Also,
stop displaying the Unix socket paths at startup since there will soon
be many of them and they're not that interesting.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
9fdad9dea1 runtests: use a function to obtain $LOGDIR for a test
This will no longer be static soon.

Ref: #10818
2023-05-05 00:45:43 -07:00
Dan Fandrich
189f9e23b6 runtests: support buffering log messages in runner & servers
Log messages generated with logmsg can now be buffered and returned from
the runner as a return value.  This will be needed with parallel testing
to allow all messages for one test to be displayed together instead of
interspersed with messages of multiple tests. Buffering can be disabled
by setting a logging callback function with setlogfunc, which is
currently being done to preserve existing logging behaviour for now.

Some additional output is generated in verbose and debugprotocol modes,
which don't always use logmsg. These modes also impact some servers
which generate extra messages. No attempt is made to buffer everything
if these modes are enabled.

Ref: #10818
Closes #11016
2023-04-24 16:28:23 -07:00
Dan Fandrich
b88ea272c1 runtests: create runner functions for clearlocks and stopservers
runtests.pl now uses runner for all server actions beyond the initial
variable configuration.

Ref: #10818
2023-04-24 16:04:24 -07:00
Dan Fandrich
8da49c9e38 runtests: display logs on server failure in singletest()
This is closer to the place where logs are displayed on test failure.
Also, only display these logs if -p is given, which is the same flag
that controls display of test failure logs. Some server log files
need to be deleted later so that they stay around long enough to be
displayed on failure.

Ref: #10818
2023-04-24 16:03:44 -07:00
Dan Fandrich
a8706fd8d0 runtests: move showdiff into runtests.pl
It's not used anywhere else.
2023-04-22 13:07:35 -07:00
Dan Fandrich
a549e046b1 runtests: refactor the main test loop into two
The test loop now has an initial loop that first runs through all
possible tests to build a set of those to attempt on this run based on
features and keywords and only then goes through that new list to run
them.  This actually makes it three loops through all tests cases, as
there is an existing loop that gathers possible test numbers from the
test files on disk.

This has two minor effects on the output: all the tests that will be
skipped are displayed at the start (instead of being interspersed with
other tests) and the -l option no longer shows a count of tests at the
end or a (misleading) statement that tests have run successfully. The
skipped tests are also omitted from the test results sent to AppVeyor
and Azure in CI builds.

Another effect is a reduction in the amount of work considered part of
the "Test definition reading and preparation time" reported with -r
making those figures slightly lower than before.

Ref: #10818
2023-04-22 13:07:35 -07:00
Dan Fandrich
020cf1c117 runtests: track only the current test timings in runner.pm
This avoids passing these data through through global variables, which
soon won't be possible.

Ref: #10818
2023-04-22 12:58:04 -07:00
Dan Fandrich
c6e7f6c61f runtests: skip test preprocessing when doing -l
This speeds up the output tremendously by avoiding unnecessary work.
2023-04-22 12:58:03 -07:00
Dan Fandrich
6210bc0111 runtests: simplify value returned regarding use of valgrind
As a side effect this will now also show in verbose mode that valgrind
is being skipped on tests that explicitly disable it, such as 600.

Ref: #10818
2023-04-22 12:58:03 -07:00
Dan Fandrich
707f74c04a runtests: refactor into more packages
testutil.pm now contains a few miscellaneous functions that are used in
several places but have no better place to live.  subvariables moves to
servers.pm since most variables that it substitutes relate to servers,
so this is the most appropriate place. Rename a few functions for better
naming consistency.

Ref: #10818
Closes #10995
2023-04-18 13:18:17 -07:00
Dan Fandrich
ba51b39732 runtests: call timestampskippedevents() in singletest
..rather than by the runner
2023-04-18 13:18:17 -07:00
Dan Fandrich
43b876d7df runtests: assume a newer Valgrind by default
The tests for an older Valgrind version should probably just be deleted,
given that they're testing for an 18-year-old version.
2023-04-18 13:18:17 -07:00
Dan Fandrich
390af1ed5e runtests: refactor test runner code into runner.pm
This is code that is directly responsible for running a single test.
This will eventually run in a separate process as part of the parallel
testing project.

Ref: #10818
2023-04-18 13:18:17 -07:00
Dan Fandrich
bfa554b207 runtests: skip unneeded work if test won't be running
This speeds up tests by avoiding unnecessary processing.

Ref: #10818
2023-04-18 13:18:17 -07:00
Dan Fandrich
a3bccb2893 runtests: factor out singletest_postcheck
This will eventually need to be part of the test runner.

Ref: #10818
2023-04-18 13:18:17 -07:00
Dan Fandrich
def8dc8071 runtests: eliminate unneeded variable 2023-04-11 14:55:32 -07:00
Dan Fandrich
44256902f3 runtests: on startup failure call displaylogs only in serverfortest
This reduces the number of calls spread throughout the code.

Ref: #10818
Closes #10919
2023-04-11 14:55:32 -07:00
Dan Fandrich
f82a38e0f3 runtests: return an error code with startservers()
The code indicates the kind of failure encountered in starting a server,
which can be used by the caller to tailor the user experience.

Ref: #10818
2023-04-11 14:55:32 -07:00
Dan Fandrich
d39db81192 tests: factor out the test server management code
This now lives in servers.pm with some configuration variables moved to
globalconfig.pm

Ref: #10818
2023-04-11 14:55:32 -07:00
Dan Fandrich
a2ce734e28 runtests: remove an inappropriate use of runclientoutput
This function is intended for running client code, not servers.
2023-04-11 14:55:32 -07:00
Dan Fandrich
fdb24b3916 runtests: only add $LIBDIR to the path for checktestcmd
Since checkcmd is for finding servers, there will never be anything in
this directory of interest to them.

Ref: #10818
2023-04-11 14:55:32 -07:00
Dan Fandrich
25aba1683a runtests: also show DISABLED tests with -l
Other reasons for skipping tests are ignored for -l, so being explicitly
disabled should be too.
2023-04-11 14:55:32 -07:00
Dan Fandrich
89ede3953b runtests: move the UNIX sockets into $PIDDIR
These were missed when the other server files were moved there.

Follow-up to 70d2fca2

Ref: #10818
2023-04-11 14:55:31 -07:00
Dan Fandrich
44677b6565 tests: tighten up perl exports
This reduces namespace pollution a little.

Ref: #10818
2023-04-11 14:55:31 -07:00
Dan Fandrich
efbaa612f7 tests: turn perl modules into full packages
This helps enforce more modularization and encapsulation. Enable and fix
warnings on a few packages.  Also, rename ftp.pm to processhelp.pm since
there's really nothing ftp-specific in it.

Ref: #10818
2023-04-11 14:55:31 -07:00
Dan Fandrich
a72a548774 runtests: fix an incorrect comment about the ld_preload feature
Follow-up to 1f631864

Ref: #10818
2023-04-10 10:51:07 -07:00
Stefan Eissing
356dd0b73a
tests: increase sws timeout for more robust testing
- for https CONNECT forwarding, this was fixed at 5 seconds
  which led to spurious CI test failures
- add --keepalive parameter to sws to control this
- let httpserver use 30 seconds

Closes #10898
2023-04-06 16:28:34 +02:00
Dan Fandrich
419a745da6 runtests: strip EOL on precheck output on Windows, too
Precheck failures would show on two lines in the test summary output
otherwise.
2023-03-31 23:08:00 -07:00
Dan Fandrich
70d2fca2f6 tests: move server config files under the pid dir
These files are generated by the test servers and must therefore be
found in the log directory to make them available to only those servers
once multiple test runners are executing in parallel. They must also not
be deleted with the log files, so they are stored in the pidfile
directory.

Ref: #10818
Closes #10875
2023-03-31 23:08:00 -07:00
Dan Fandrich
1cffced9c5 runtests: use the ssh key filenames from the sshhelp package 2023-03-31 23:08:00 -07:00
Dan Fandrich
3cc41a2d36 tests: move pidfiles and portfiles under the log directory
This is to segregate all files written by a test process into a single
root to allow for future parallel testing.

Ref: #10818
Closes #10874
2023-03-31 23:08:00 -07:00
Dan Fandrich
ec2a5bc5fc runtests: minor code cleanups 2023-03-31 23:08:00 -07:00
Dan Fandrich
62ba1d6bd9 runtests: call processexists() and pidfromfile()
rather than duplicating the logic in several places.
2023-03-31 23:08:00 -07:00
Dan Fandrich
4ced75b7ce runtests: show error message if file can't be written 2023-03-30 09:53:57 -07:00
Dan Fandrich
58e85d6d1d tests: fix remaining servers to run with a dynamic log directory
This final commit in the series is sufficient to allow the tests succeed
if $LOGDIR is changed in runtests.pl.

Ref: #10818
Closes #10866
2023-03-30 09:53:57 -07:00
Dan Fandrich
8a298119f1 tests: fix http servers to run with a dynamic log directory
Ref: #10818
2023-03-30 09:53:57 -07:00
Dan Fandrich
c753418d8b tests: fix ftpserver to run with a dynamic log directory
Ref: #10818
2023-03-30 09:53:57 -07:00
Dan Fandrich
e7a021e135 tests: use %LOGDIR to refer to the log directory
This will allow it be set dynamically.

Ref: #10818
2023-03-30 09:53:57 -07:00
Dan Fandrich
0c82fcde79 runtests: track verification time even if no files to compare 2023-03-30 09:53:57 -07:00
Dan Fandrich
0e3ae25337 tests: switch to 3-argument open in test suite
The perl 2-argument open has been considered not-quite-deprecated since
the 3-argument form was introduced almost a quarter century ago.
2023-03-30 09:53:57 -07:00
Dan Fandrich
b133f70a52 tests: silence some Perl::Critic warnings in test suite
Not all warnings are fixed; many are as much stylistic suggestions than
anything and IMHO don't do much to actually improve the code.

Ref: #10818
Closes #10861
2023-03-30 09:12:52 -07:00