mirror of
https://github.com/curl/curl.git
synced 2026-08-26 09:23:31 +03:00
tests: rename tests/tests-httpd to tests/http
- httpd is only one server we test with - the suite coveres the HTTP protocol in general where the default test cases need a more beefy environment Closes #10654
This commit is contained in:
Stefan Eissing
Daniel Stenberg
parent
9fd2d5aa72
commit
e497a96a0e
30 changed files with 58 additions and 22 deletions
6
tests/http/.gitignore
vendored
Normal file
6
tests/http/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
# Copyright (C) 2000 - 2022 Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
|
||||
config.ini
|
||||
gen
|
||||
27
tests/http/Makefile.am
Normal file
27
tests/http/Makefile.am
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
|
||||
clean-local:
|
||||
rm -rf *.pyc __pycache__
|
||||
rm -rf gen
|
||||
135
tests/http/README.md
Normal file
135
tests/http/README.md
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
<!--
|
||||
Copyright (C) 1998 - 2022 Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
|
||||
SPDX-License-Identifier: curl
|
||||
-->
|
||||
|
||||
# The curl HTTPD Test Suite
|
||||
|
||||
This is an additional test suite using a combination of Apache httpd and nghttpx servers to perform various tests beyond the capabilities of the standard curl test suite.
|
||||
|
||||
# Usage
|
||||
|
||||
The test cases and necessary files are in `tests/httpd`. You can invoke `pytest` from there or from the top level curl checkout and it will find all tests.
|
||||
|
||||
```
|
||||
curl> pytest
|
||||
platform darwin -- Python 3.9.15, pytest-6.2.0, py-1.10.0, pluggy-0.13.1
|
||||
rootdir: /Users/sei/projects/curl
|
||||
collected 5 items
|
||||
|
||||
tests/httpd/test_01_basic.py .....
|
||||
```
|
||||
|
||||
Pytest takes arguments. `-v` increases its verbosity and can be used several times. `-k <expr>` can be used to run only matching test cases. The `expr` can be something resembling a python test or just a string that needs to match test cases in their names.
|
||||
|
||||
```
|
||||
curl> pytest -vv -k test_01_02
|
||||
```
|
||||
|
||||
runs all test cases that have `test_01_02` in their name. This does not have to be the start of the name.
|
||||
|
||||
Depending on your setup, some test cases may be skipped and appear as `s` in the output. If you run pytest verbose, it will also give you the reason for skipping.
|
||||
|
||||
|
||||
# Prerequisites
|
||||
|
||||
You will need:
|
||||
|
||||
1. a recent Python, the `cryptography` module and, of course, `pytest`
|
||||
2. a apache httpd development version. On Debian/Ubuntu, the package `apache2-dev` has this.
|
||||
3. a local `curl` project build
|
||||
3. optionally, a `nghttpx` with HTTP/3 enabled or h3 test cases will be skipped.
|
||||
|
||||
### Configuration
|
||||
|
||||
Via curl's `configure` script you may specify:
|
||||
|
||||
* `--with-test-nghttpx=<path-of-nghttpx>` if you have nghttpx to use somewhere outside your `$PATH`.
|
||||
* `--with-test-httpd=<httpd-install-path>` if you have an Apache httpd installed somewhere else. On Debian/Ubuntu it will otherwise look into `/usr/bin` and `/usr/sbin` to find those.
|
||||
|
||||
## Usage Tips
|
||||
|
||||
Several test cases are parameterized, for example with the HTTP version to use. If you want to run a test with a particular protocol only, use a command line like:
|
||||
|
||||
```
|
||||
curl> pytest -k "test_02_06 and h2"
|
||||
```
|
||||
|
||||
Several test cases can be repeated, they all have the `repeat` parameter. To make this work, you have to start `pytest` in the test directory itself (for some unknown reason). Like in:
|
||||
|
||||
```
|
||||
curl/tests/http> pytest -k "test_02_06 and h2" --repeat=100
|
||||
```
|
||||
|
||||
which then runs this test case a hundred times. In case of flaky tests, you can make pytest stop on the first one with:
|
||||
|
||||
```
|
||||
curl/tests/http> pytest -k "test_02_06 and h2" --repeat=100 --maxfail=1
|
||||
```
|
||||
|
||||
which allow you to inspect output and log files for the failed run. Speaking of log files, the verbosity of pytest is also used to collect curl trace output. If you specify `-v` three times, the `curl` command is started with `--trace`:
|
||||
|
||||
```
|
||||
curl/tests/http> pytest -vvv -k "test_02_06 and h2" --repeat=100 --maxfail=1
|
||||
```
|
||||
|
||||
all of curl's output and trace file are found in `tests/http/gen/curl`.
|
||||
|
||||
## Writing Tests
|
||||
|
||||
There is a lot of [`pytest` documentation](https://docs.pytest.org/) with examples. No use in repeating that here. Assuming you are somewhat familiar with it, it is useful how *this* general test suite is setup. Especially if you want to add test cases.
|
||||
|
||||
### Servers
|
||||
|
||||
In `conftest.py` 3 "fixtures" are defined that are used by all test cases:
|
||||
|
||||
1. `env`: the test environment. It is an instance of class `testenv/env.py:Env`. It holds all information about paths, availability of features (HTTP/3!), port numbers to use, domains and SSL certificates for those.
|
||||
2. `httpd`: the Apache httpd instance, configured and started, then stopped at the end of the test suite. It has sites configured for the domains from `env`. It also loads a local module `mod_curltest?` and makes it available in certain locations. (more on mod_curltest below).
|
||||
3. `nghttpx`: an instance of nghttpx that provides HTTP/3 support. `nghttpx` proxies those requests to the `httpd` server. In a direct mapping, so you may access all the resources under the same path as with HTTP/2. Only the port number used for HTTP/3 requests will be different.
|
||||
|
||||
`pytest` manages these fixture so that they are created once and terminated before exit. This means you can `Ctrl-C` a running pytest and the server will shutdown. Only when you brutally chop its head off, might there be servers left
|
||||
behind.
|
||||
|
||||
### Test Cases
|
||||
|
||||
Tests making use of these fixtures have them in their parameter list. This tells pytest that a particular test needs them, so it has to create them. Since one can invoke pytest for just a single test, it is important that a test references the ones it needs.
|
||||
|
||||
All test cases start with `test_` in their name. We use a double number scheme to group them. This makes it ease to run only specific tests and also give a short mnemonic to communicate trouble with others in the project. Otherwise you are free to name test cases as you think fitting.
|
||||
|
||||
Tests are grouped thematically in a file with a single Python test class. This is convenient if you need a special "fixture" for several tests. "fixtures" can have "class" scope.
|
||||
|
||||
There is a curl helper class that knows how to invoke curl and interpret its output. Among other things, it does add the local CA to the command line, so that SSL connections to the test servers are verified. Nothing prevents anyone from running curl directly, for specific uses not covered by the `CurlClient` class.
|
||||
|
||||
### mod_curltest
|
||||
|
||||
The module source code is found in `testenv/mod_curltest`. It is compiled using the `apxs` command, commonly provided via the `apache2-dev` package. Compilation is quick and done once at the start of a test run.
|
||||
|
||||
The module adds 2 "handlers" to the Apache server (right now). Handler are pieces of code that receive HTTP requests and generate the response. Those handlers are:
|
||||
|
||||
* `curltest-echo`: hooked up on the path `/curltest/echo`. This one echoes a request and copies all data from the request body to the response body. Useful for simulating upload and checking that the data arrived as intended.
|
||||
* `curltest-tweak`: hooked up on the path `/curltest/tweak`. This handler is more of a Swiss army knife. It interprets parameters from the URL query string to drive its behavior.
|
||||
* `status=nnn`: generate a response with HTTP status code `nnn`.
|
||||
* `chunks=n`: generate `n` chunks of data in the response body, defaults to 3.
|
||||
* `chunk_size=nnn`: each chunk should contain `nnn` bytes of data. Maximum is 16KB right now.
|
||||
* `chunkd_delay=duration`: wait `duration` time between writing chunks
|
||||
* `delay=duration`: wait `duration` time to send the response headers
|
||||
* `body_error=(timeout|reset)`: produce an error after the first chunk in the response body
|
||||
* `id=str`: add `str` in the response header `request-id`
|
||||
|
||||
`duration` values are integers, optionally followed by a unit. Units are:
|
||||
|
||||
* `d`: days (probably not useful here)
|
||||
* `h`: hours
|
||||
* `mi`: minutes
|
||||
* `s`: seconds (the default)
|
||||
* `ms`: milliseconds
|
||||
|
||||
As you can see, `mod_curltest`'s tweak handler allow to simulate many kinds of responses. An example of its use is `test_03_01` where responses are delayed using `chunk_delay`. This gives the response a defined duration and the test uses that to reload `httpd` in the middle of the first request. A graceful reload in httpd lets ongoing requests finish, but will close the connection afterwards and tear down the serving process. The following request need then to open a new connection. This is verified by the test case.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
46
tests/http/config.ini.in
Normal file
46
tests/http/config.ini.in
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
[global]
|
||||
|
||||
[httpd]
|
||||
apxs = @APXS@
|
||||
httpd = @HTTPD@
|
||||
apachectl = @APACHECTL@
|
||||
|
||||
[test]
|
||||
http_port = 5001
|
||||
https_port = 5002
|
||||
h3_port = 5002
|
||||
proxy_port = 5004
|
||||
proxys_port = 5005
|
||||
|
||||
[nghttpx]
|
||||
nghttpx = @HTTPD_NGHTTPX@
|
||||
|
||||
[caddy]
|
||||
caddy = @CADDY@
|
||||
http_port = 5010
|
||||
https_port = 5011
|
||||
69
tests/http/conftest.py
Normal file
69
tests/http/conftest.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.append(os.path.join(os.path.dirname(__file__), '.'))
|
||||
|
||||
from testenv import Env, Nghttpx, Httpd
|
||||
|
||||
|
||||
def pytest_report_header(config, startdir):
|
||||
return f"curl http tests"
|
||||
|
||||
|
||||
@pytest.fixture(scope="package")
|
||||
def env(pytestconfig) -> Env:
|
||||
env = Env(pytestconfig=pytestconfig)
|
||||
level = logging.DEBUG if env.verbose > 0 else logging.INFO
|
||||
logging.getLogger('').setLevel(level=level)
|
||||
env.setup()
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope='package')
|
||||
def httpd(env) -> Httpd:
|
||||
httpd = Httpd(env=env)
|
||||
assert httpd.exists(), f'httpd not found: {env.httpd}'
|
||||
httpd.clear_logs()
|
||||
assert httpd.start()
|
||||
yield httpd
|
||||
httpd.stop()
|
||||
|
||||
|
||||
@pytest.fixture(scope='package')
|
||||
def nghttpx(env, httpd) -> Optional[Nghttpx]:
|
||||
if env.have_h3_server():
|
||||
nghttpx = Nghttpx(env=env)
|
||||
nghttpx.clear_logs()
|
||||
assert nghttpx.start()
|
||||
yield nghttpx
|
||||
nghttpx.stop()
|
||||
return None
|
||||
|
||||
423
tests/http/scorecard.py
Normal file
423
tests/http/scorecard.py
Normal file
|
|
@ -0,0 +1,423 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from statistics import mean
|
||||
from typing import Dict, Any
|
||||
|
||||
from testenv import Env, Httpd, Nghttpx, CurlClient, Caddy, ExecResult
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ScoreCardException(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ScoreCard:
|
||||
|
||||
def __init__(self):
|
||||
self.verbose = 0
|
||||
self.env = None
|
||||
self.httpd = None
|
||||
self.nghttpx = None
|
||||
self.caddy = None
|
||||
|
||||
def info(self, msg):
|
||||
if self.verbose > 0:
|
||||
sys.stderr.write(msg)
|
||||
sys.stderr.flush()
|
||||
|
||||
def handshakes(self, proto: str) -> Dict[str, Any]:
|
||||
props = {}
|
||||
sample_size = 10
|
||||
self.info(f'handshaking ')
|
||||
for authority in [
|
||||
f'{self.env.authority_for(self.env.domain1, proto)}'
|
||||
]:
|
||||
self.info('localhost')
|
||||
c_samples = []
|
||||
hs_samples = []
|
||||
errors = []
|
||||
for i in range(sample_size):
|
||||
self.info('.')
|
||||
curl = CurlClient(env=self.env)
|
||||
url = f'https://{authority}/'
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto)
|
||||
if r.exit_code == 0 and len(r.stats) == 1:
|
||||
c_samples.append(r.stats[0]['time_connect'])
|
||||
hs_samples.append(r.stats[0]['time_appconnect'])
|
||||
else:
|
||||
errors.append(f'exit={r.exit_code}')
|
||||
props['localhost'] = {
|
||||
'connect': mean(c_samples),
|
||||
'handshake': mean(hs_samples),
|
||||
'errors': errors
|
||||
}
|
||||
for authority in [
|
||||
'curl.se', 'google.com', 'cloudflare.com', 'nghttp2.org',
|
||||
]:
|
||||
for ipv in ['ipv4', 'ipv6']:
|
||||
self.info(f'{authority}-{ipv}')
|
||||
c_samples = []
|
||||
hs_samples = []
|
||||
errors = []
|
||||
for i in range(sample_size):
|
||||
self.info('.')
|
||||
curl = CurlClient(env=self.env)
|
||||
args = [
|
||||
'--http3-only' if proto == 'h3' else '--http2',
|
||||
f'--{ipv}', f'https://{authority}/'
|
||||
]
|
||||
r = curl.run_direct(args=args, with_stats=True)
|
||||
if r.exit_code == 0 and len(r.stats) == 1:
|
||||
c_samples.append(r.stats[0]['time_connect'])
|
||||
hs_samples.append(r.stats[0]['time_appconnect'])
|
||||
else:
|
||||
errors.append(f'exit={r.exit_code}')
|
||||
props[f'{authority}-{ipv}'] = {
|
||||
'connect': mean(c_samples) if len(c_samples) else -1,
|
||||
'handshake': mean(hs_samples) if len(hs_samples) else -1,
|
||||
'errors': errors
|
||||
}
|
||||
self.info('\n')
|
||||
return props
|
||||
|
||||
def _make_docs_file(self, docs_dir: str, fname: str, fsize: int):
|
||||
fpath = os.path.join(docs_dir, fname)
|
||||
data1k = 1024*'x'
|
||||
flen = 0
|
||||
with open(fpath, 'w') as fd:
|
||||
while flen < fsize:
|
||||
fd.write(data1k)
|
||||
flen += len(data1k)
|
||||
return flen
|
||||
|
||||
def _check_downloads(self, r: ExecResult, count: int):
|
||||
error = ''
|
||||
if r.exit_code != 0:
|
||||
error += f'exit={r.exit_code} '
|
||||
if r.exit_code != 0 or len(r.stats) != count:
|
||||
error += f'stats={len(r.stats)}/{count} '
|
||||
fails = [s for s in r.stats if s['response_code'] != 200]
|
||||
if len(fails) > 0:
|
||||
error += f'{len(fails)} failed'
|
||||
return error if len(error) > 0 else None
|
||||
|
||||
def transfer_single(self, url: str, proto: str, count: int):
|
||||
sample_size = count
|
||||
count = 1
|
||||
samples = []
|
||||
errors = []
|
||||
self.info(f'{sample_size}x single')
|
||||
for i in range(sample_size):
|
||||
curl = CurlClient(env=self.env)
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto)
|
||||
err = self._check_downloads(r, count)
|
||||
if err:
|
||||
errors.append(err)
|
||||
else:
|
||||
samples.append(r.stats[0]['speed_download'])
|
||||
self.info(f'.')
|
||||
return {
|
||||
'count': count,
|
||||
'samples': sample_size,
|
||||
'speed': mean(samples) if len(samples) else -1,
|
||||
'errors': errors
|
||||
}
|
||||
|
||||
def transfer_serial(self, url: str, proto: str, count: int):
|
||||
sample_size = 1
|
||||
samples = []
|
||||
errors = []
|
||||
url = f'{url}?[0-{count - 1}]'
|
||||
self.info(f'{sample_size}x{count} serial')
|
||||
for i in range(sample_size):
|
||||
curl = CurlClient(env=self.env)
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto)
|
||||
self.info(f'.')
|
||||
err = self._check_downloads(r, count)
|
||||
if err:
|
||||
errors.append(err)
|
||||
else:
|
||||
for s in r.stats:
|
||||
samples.append(s['speed_download'])
|
||||
return {
|
||||
'count': count,
|
||||
'samples': sample_size,
|
||||
'speed': mean(samples) if len(samples) else -1,
|
||||
'errors': errors
|
||||
}
|
||||
|
||||
def transfer_parallel(self, url: str, proto: str, count: int):
|
||||
sample_size = 1
|
||||
samples = []
|
||||
errors = []
|
||||
url = f'{url}?[0-{count - 1}]'
|
||||
self.info(f'{sample_size}x{count} parallel')
|
||||
for i in range(sample_size):
|
||||
curl = CurlClient(env=self.env)
|
||||
start = datetime.now()
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
err = self._check_downloads(r, count)
|
||||
if err:
|
||||
errors.append(err)
|
||||
else:
|
||||
duration = datetime.now() - start
|
||||
total_size = sum([s['size_download'] for s in r.stats])
|
||||
samples.append(total_size / duration.total_seconds())
|
||||
return {
|
||||
'count': count,
|
||||
'samples': sample_size,
|
||||
'speed': mean(samples) if len(samples) else -1,
|
||||
'errors': errors
|
||||
}
|
||||
|
||||
def download_url(self, url: str, proto: str, count: int):
|
||||
self.info(f' {url}: ')
|
||||
props = {
|
||||
'single': self.transfer_single(url=url, proto=proto, count=10),
|
||||
'serial': self.transfer_serial(url=url, proto=proto, count=count),
|
||||
'parallel': self.transfer_parallel(url=url, proto=proto, count=count),
|
||||
}
|
||||
self.info(f'\n')
|
||||
return props
|
||||
|
||||
def downloads(self, proto: str, test_httpd: bool = True,
|
||||
test_caddy: bool = True) -> Dict[str, Any]:
|
||||
scores = {}
|
||||
if test_httpd:
|
||||
if proto == 'h3':
|
||||
port = self.env.h3_port
|
||||
via = 'nghttpx'
|
||||
descr = f'port {port}, proxying httpd'
|
||||
else:
|
||||
port = self.env.https_port
|
||||
via = 'httpd'
|
||||
descr = f'port {port}'
|
||||
self.info(f'{via} downloads\n')
|
||||
self._make_docs_file(docs_dir=self.httpd.docs_dir, fname='score1.data', fsize=1024*1024)
|
||||
url1 = f'https://{self.env.domain1}:{port}/score1.data'
|
||||
self._make_docs_file(docs_dir=self.httpd.docs_dir, fname='score10.data', fsize=10*1024*1024)
|
||||
url10 = f'https://{self.env.domain1}:{port}/score10.data'
|
||||
self._make_docs_file(docs_dir=self.httpd.docs_dir, fname='score100.data', fsize=100*1024*1024)
|
||||
url100 = f'https://{self.env.domain1}:{port}/score100.data'
|
||||
scores[via] = {
|
||||
'description': descr,
|
||||
'1MB-local': self.download_url(url=url1, proto=proto, count=50),
|
||||
'10MB-local': self.download_url(url=url10, proto=proto, count=50),
|
||||
'100MB-local': self.download_url(url=url100, proto=proto, count=50),
|
||||
}
|
||||
if test_caddy and self.caddy:
|
||||
port = self.caddy.port
|
||||
via = 'caddy'
|
||||
descr = f'port {port}'
|
||||
self.info('caddy downloads\n')
|
||||
self._make_docs_file(docs_dir=self.caddy.docs_dir, fname='score1.data', fsize=1024 * 1024)
|
||||
url1 = f'https://{self.env.domain1}:{port}/score1.data'
|
||||
self._make_docs_file(docs_dir=self.caddy.docs_dir, fname='score10.data', fsize=10 * 1024 * 1024)
|
||||
url10 = f'https://{self.env.domain1}:{port}/score10.data'
|
||||
self._make_docs_file(docs_dir=self.caddy.docs_dir, fname='score100.data', fsize=100 * 1024 * 1024)
|
||||
url100 = f'https://{self.env.domain1}:{port}/score100.data'
|
||||
scores[via] = {
|
||||
'description': descr,
|
||||
'1MB-local': self.download_url(url=url1, proto=proto, count=50),
|
||||
'10MB-local': self.download_url(url=url10, proto=proto, count=50),
|
||||
'100MB-local': self.download_url(url=url100, proto=proto, count=50),
|
||||
}
|
||||
return scores
|
||||
|
||||
def score_proto(self, proto: str,
|
||||
handshakes: bool = True,
|
||||
downloads: bool = True,
|
||||
test_httpd: bool = True,
|
||||
test_caddy: bool = True):
|
||||
self.info(f"scoring {proto}\n")
|
||||
p = {}
|
||||
if proto == 'h3':
|
||||
p['name'] = 'h3'
|
||||
if not self.env.have_h3_curl():
|
||||
raise ScoreCardException('curl does not support HTTP/3')
|
||||
for lib in ['ngtcp2', 'quiche', 'msh3']:
|
||||
if self.env.curl_uses_lib(lib):
|
||||
p['implementation'] = lib
|
||||
break
|
||||
elif proto == 'h2':
|
||||
p['name'] = 'h2'
|
||||
if not self.env.have_h2_curl():
|
||||
raise ScoreCardException('curl does not support HTTP/2')
|
||||
for lib in ['nghttp2', 'hyper']:
|
||||
if self.env.curl_uses_lib(lib):
|
||||
p['implementation'] = lib
|
||||
break
|
||||
else:
|
||||
raise ScoreCardException(f"unknown protocol: {proto}")
|
||||
|
||||
if 'implementation' not in p:
|
||||
raise ScoreCardException(f'did not recognized {p} lib')
|
||||
p['version'] = Env.curl_lib_version(p['implementation'])
|
||||
|
||||
score = {
|
||||
'curl': self.env.curl_version(),
|
||||
'os': self.env.curl_os(),
|
||||
'protocol': p,
|
||||
}
|
||||
if handshakes:
|
||||
score['handshakes'] = self.handshakes(proto=proto)
|
||||
if downloads:
|
||||
score['downloads'] = self.downloads(proto=proto,
|
||||
test_httpd=test_httpd,
|
||||
test_caddy=test_caddy)
|
||||
self.info("\n")
|
||||
return score
|
||||
|
||||
def fmt_ms(self, tval):
|
||||
return f'{int(tval*1000)} ms' if tval >= 0 else '--'
|
||||
|
||||
def fmt_mb(self, val):
|
||||
return f'{val/(1024*1024):0.000f} MB' if val >= 0 else '--'
|
||||
|
||||
def fmt_mbs(self, val):
|
||||
return f'{val/(1024*1024):0.000f} MB/s' if val >= 0 else '--'
|
||||
|
||||
def print_score(self, score):
|
||||
print(f'{score["protocol"]["name"].upper()} in curl {score["curl"]} ({score["os"]}) via '
|
||||
f'{score["protocol"]["implementation"]}/{score["protocol"]["version"]} ')
|
||||
if 'handshakes' in score:
|
||||
print('Handshakes')
|
||||
print(f' {"Host":<25} {"Connect":>12} {"Handshake":>12} {"Errors":<20}')
|
||||
for key, val in score["handshakes"].items():
|
||||
print(f' {key:<25} {self.fmt_ms(val["connect"]):>12} '''
|
||||
f'{self.fmt_ms(val["handshake"]):>12} {"/".join(val["errors"]):<20}')
|
||||
if 'downloads' in score:
|
||||
print('Downloads')
|
||||
for dkey, dval in score["downloads"].items():
|
||||
print(f' {dkey}: {dval["description"]}')
|
||||
for skey, sval in dval.items():
|
||||
if isinstance(sval, str):
|
||||
continue
|
||||
print(f' {skey:<13} {"Samples":>10} {"Count":>10} {"Speed":>17} {"Errors":<20}')
|
||||
for key, val in sval.items():
|
||||
print(f' {key:<11} {val["samples"]:>10} '''
|
||||
f'{val["count"]:>10} {self.fmt_mbs(val["speed"]):>17} '
|
||||
f'{"/".join(val["errors"]):<20}')
|
||||
|
||||
def main(self):
|
||||
parser = argparse.ArgumentParser(prog='scorecard', description="""
|
||||
Run a range of tests to give a scorecard for a HTTP protocol
|
||||
'h3' or 'h2' implementation in curl.
|
||||
""")
|
||||
parser.add_argument("-v", "--verbose", action='count', default=0,
|
||||
help="log more output on stderr")
|
||||
parser.add_argument("-t", "--text", action='store_true', default=False,
|
||||
help="print text instead of json")
|
||||
parser.add_argument("-d", "--downloads", action='store_true', default=False,
|
||||
help="evaluate downloads only")
|
||||
parser.add_argument("--httpd", action='store_true', default=False,
|
||||
help="evaluate httpd server only")
|
||||
parser.add_argument("--caddy", action='store_true', default=False,
|
||||
help="evaluate caddy server only")
|
||||
parser.add_argument("protocols", nargs='*', help="Name(s) of protocol to score")
|
||||
args = parser.parse_args()
|
||||
|
||||
self.verbose = args.verbose
|
||||
if args.verbose > 0:
|
||||
console = logging.StreamHandler()
|
||||
console.setLevel(logging.INFO)
|
||||
console.setFormatter(logging.Formatter(logging.BASIC_FORMAT))
|
||||
logging.getLogger('').addHandler(console)
|
||||
|
||||
protocols = args.protocols if len(args.protocols) else ['h2', 'h3']
|
||||
handshakes = True
|
||||
downloads = True
|
||||
test_httpd = True
|
||||
test_caddy = True
|
||||
if args.downloads:
|
||||
handshakes = False
|
||||
if args.caddy:
|
||||
test_caddy = True
|
||||
test_httpd = False
|
||||
if args.httpd:
|
||||
test_caddy = False
|
||||
test_httpd = True
|
||||
|
||||
rv = 0
|
||||
self.env = Env()
|
||||
self.env.setup()
|
||||
self.httpd = None
|
||||
self.nghttpx = None
|
||||
self.caddy = None
|
||||
try:
|
||||
self.httpd = Httpd(env=self.env)
|
||||
assert self.httpd.exists(), f'httpd not found: {self.env.httpd}'
|
||||
self.httpd.clear_logs()
|
||||
assert self.httpd.start()
|
||||
if 'h3' in protocols:
|
||||
self.nghttpx = Nghttpx(env=self.env)
|
||||
self.nghttpx.clear_logs()
|
||||
assert self.nghttpx.start()
|
||||
if self.env.caddy:
|
||||
self.caddy = Caddy(env=self.env)
|
||||
self.caddy.clear_logs()
|
||||
assert self.caddy.start()
|
||||
|
||||
for p in protocols:
|
||||
score = self.score_proto(proto=p, handshakes=handshakes,
|
||||
downloads=downloads,
|
||||
test_caddy=test_caddy,
|
||||
test_httpd=test_httpd)
|
||||
if args.text:
|
||||
self.print_score(score)
|
||||
else:
|
||||
print(json.JSONEncoder(indent=2).encode(score))
|
||||
|
||||
except ScoreCardException as ex:
|
||||
sys.stderr.write(f"ERROR: {str(ex)}\n")
|
||||
rv = 1
|
||||
except KeyboardInterrupt:
|
||||
log.warning("aborted")
|
||||
rv = 1
|
||||
finally:
|
||||
if self.caddy:
|
||||
self.caddy.stop()
|
||||
self.caddy = None
|
||||
if self.nghttpx:
|
||||
self.nghttpx.stop(wait_dead=False)
|
||||
if self.httpd:
|
||||
self.httpd.stop()
|
||||
self.httpd = None
|
||||
sys.exit(rv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ScoreCard().main()
|
||||
94
tests/http/test_01_basic.py
Normal file
94
tests/http/test_01_basic.py
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import pytest
|
||||
|
||||
from testenv import Env
|
||||
from testenv import CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestBasic:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
|
||||
# simple http: GET
|
||||
def test_01_01_http_get(self, env: Env, httpd):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'http://{env.domain1}:{env.http_port}/data.json'
|
||||
r = curl.http_get(url=url)
|
||||
assert r.exit_code == 0
|
||||
assert r.response['status'] == 200
|
||||
assert r.json['server'] == env.domain1
|
||||
|
||||
# simple https: GET, any http version
|
||||
def test_01_02_https_get(self, env: Env, httpd):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{env.https_port}/data.json'
|
||||
r = curl.http_get(url=url)
|
||||
assert r.exit_code == 0
|
||||
assert r.response['status'] == 200
|
||||
assert r.json['server'] == env.domain1
|
||||
|
||||
# simple https: GET, h2 wanted and got
|
||||
def test_01_02_h2_get(self, env: Env, httpd):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{env.https_port}/data.json'
|
||||
r = curl.http_get(url=url, extra_args=['--http2'])
|
||||
assert r.exit_code == 0
|
||||
assert r.response['status'] == 200
|
||||
assert r.response['protocol'] == 'HTTP/2'
|
||||
assert r.json['server'] == env.domain1
|
||||
|
||||
# simple https: GET, h2 unsupported, fallback to h1
|
||||
def test_01_02_h2_unsupported(self, env: Env, httpd):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain2}:{env.https_port}/data.json'
|
||||
r = curl.http_get(url=url, extra_args=['--http2'])
|
||||
assert r.exit_code == 0
|
||||
assert r.response['status'] == 200
|
||||
assert r.response['protocol'] == 'HTTP/1.1'
|
||||
assert r.json['server'] == env.domain2
|
||||
|
||||
# simple h3: GET, want h3 and get it
|
||||
@pytest.mark.skipif(condition=not Env.have_h3_curl(), reason="no h3 curl")
|
||||
@pytest.mark.skipif(condition=not Env.have_h3_server(), reason="no h3 server")
|
||||
def test_01_03_h3_get(self, env: Env, httpd, nghttpx):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{env.h3_port}/data.json'
|
||||
r = curl.http_get(url=url, extra_args=['--http3'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
assert r.response['status'] == 200
|
||||
assert r.response['protocol'] == 'HTTP/3'
|
||||
assert r.json['server'] == env.domain1
|
||||
289
tests/http/test_02_download.py
Normal file
289
tests/http/test_02_download.py
Normal file
|
|
@ -0,0 +1,289 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import difflib
|
||||
import filecmp
|
||||
import logging
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestDownload:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, httpd, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, httpd):
|
||||
env.make_data_file(indir=httpd.docs_dir, fname="data-100k", fsize=100*1024)
|
||||
env.make_data_file(indir=httpd.docs_dir, fname="data-1m", fsize=1024*1024)
|
||||
env.make_data_file(indir=httpd.docs_dir, fname="data-10m", fsize=10*1024*1024)
|
||||
|
||||
# download 1 file
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_01_download_1(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download 2 files
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_02_download_2(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-1]'
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=2, exp_status=200)
|
||||
|
||||
# download 100 files sequentially
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_03_download_100_sequential(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-99]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=100, exp_status=200)
|
||||
# http/1.1 sequential transfers will open 1 connection
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download 100 files parallel
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_04_download_100_parallel(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-99]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=100, exp_status=200)
|
||||
if proto == 'http/1.1':
|
||||
# http/1.1 parallel transfers will open multiple connections
|
||||
assert r.total_connects > 1
|
||||
else:
|
||||
# http2 parallel transfers will use one connection (common limit is 100)
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download 500 files sequential
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_05_download_500_sequential(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-499]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=500, exp_status=200)
|
||||
if proto == 'http/1.1':
|
||||
# http/1.1 parallel transfers will open multiple connections
|
||||
assert r.total_connects > 1
|
||||
else:
|
||||
# http2 parallel transfers will use one connection (common limit is 100)
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download 500 files parallel (default max of 100)
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_06_download_500_parallel(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data.json?[000-499]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=500, exp_status=200)
|
||||
if proto == 'http/1.1':
|
||||
# http/1.1 parallel transfers will open multiple connections
|
||||
assert r.total_connects > 1
|
||||
else:
|
||||
# http2 parallel transfers will use one connection (common limit is 100)
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download files parallel, check connection reuse/multiplex
|
||||
@pytest.mark.parametrize("proto", ['h2', 'h3'])
|
||||
def test_02_07_download_reuse(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count=200
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-{count-1}]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto,
|
||||
with_stats=True, extra_args=[
|
||||
'--parallel', '--parallel-max', '200'
|
||||
])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
# should have used 2 connections only (test servers allow 100 req/conn)
|
||||
assert r.total_connects == 2, "h2 should use fewer connections here"
|
||||
|
||||
# download files parallel with http/1.1, check connection not reused
|
||||
@pytest.mark.parametrize("proto", ['http/1.1'])
|
||||
def test_02_07b_download_reuse(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count=20
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-{count-1}]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto,
|
||||
with_stats=True, extra_args=[
|
||||
'--parallel', '--parallel-max', '200'
|
||||
])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
# http/1.1 should have used count connections
|
||||
assert r.total_connects == count, "http/1.1 should use this many connections"
|
||||
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_08_1MB_serial(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count = 20
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data-1m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_09_1MB_parallel(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count = 20
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data-1m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--parallel'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_10_10MB_serial(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count = 20
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data-10m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_11_10MB_parallel(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count = 20
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data-10m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--parallel'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_02_12_head_serial_https(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count = 100
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}/data-10m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--head'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
|
||||
@pytest.mark.parametrize("proto", ['h2'])
|
||||
def test_02_13_head_serial_h2c(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
count = 100
|
||||
urln = f'http://{env.domain1}:{env.http_port}/data-10m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--head', '--http2-prior-knowledge', '--fail-early'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
|
||||
def test_02_20_h2_small_frames(self, env: Env, httpd, repeat):
|
||||
# Test case to reproduce content corruption as observed in
|
||||
# https://github.com/curl/curl/issues/10525
|
||||
# To reliably reproduce, we need an Apache httpd that supports
|
||||
# setting smaller frame sizes. This is not released yet, we
|
||||
# test if it works and back out if not.
|
||||
httpd.set_extra_config(env.domain1, lines=[
|
||||
f'H2MaxDataFrameLen 1024',
|
||||
])
|
||||
assert httpd.stop()
|
||||
if not httpd.start():
|
||||
# no, not supported, bail out
|
||||
httpd.set_extra_config(env.domain1, lines=None)
|
||||
assert httpd.start()
|
||||
pytest.skip(f'H2MaxDataFrameLen not supported')
|
||||
# ok, make 100 downloads with 2 parallel running and they
|
||||
# are expected to stumble into the issue when using `lib/http2.c`
|
||||
# from curl 7.88.0
|
||||
count = 100
|
||||
urln = f'https://{env.authority_for(env.domain1, "h2")}/data-1m?[0-{count-1}]'
|
||||
curl = CurlClient(env=env)
|
||||
r = curl.http_download(urls=[urln], alpn_proto="h2", extra_args=[
|
||||
'--parallel', '--parallel-max', '2'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
srcfile = os.path.join(httpd.docs_dir, 'data-1m')
|
||||
for i in range(count):
|
||||
dfile = curl.download_file(i)
|
||||
assert os.path.exists(dfile)
|
||||
if not filecmp.cmp(srcfile, dfile, shallow=False):
|
||||
diff = "".join(difflib.unified_diff(a=open(srcfile).readlines(),
|
||||
b=open(dfile).readlines(),
|
||||
fromfile=srcfile,
|
||||
tofile=dfile,
|
||||
n=1))
|
||||
assert False, f'download {dfile} differs:\n{diff}'
|
||||
# restore httpd defaults
|
||||
httpd.set_extra_config(env.domain1, lines=None)
|
||||
assert httpd.stop()
|
||||
assert httpd.start()
|
||||
|
||||
113
tests/http/test_03_goaway.py
Normal file
113
tests/http/test_03_goaway.py
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import time
|
||||
from datetime import timedelta
|
||||
from threading import Thread
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient, ExecResult
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestGoAway:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
|
||||
# download files sequentially with delay, reload server for GOAWAY
|
||||
def test_03_01_h2_goaway(self, env: Env, httpd, nghttpx, repeat):
|
||||
proto = 'h2'
|
||||
count = 3
|
||||
self.r = None
|
||||
def long_run():
|
||||
curl = CurlClient(env=env)
|
||||
# send 10 chunks of 1024 bytes in a response body with 100ms delay in between
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=10&chunk_size=1024&chunk_delay=100ms'
|
||||
self.r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
|
||||
t = Thread(target=long_run)
|
||||
t.start()
|
||||
# each request will take a second, reload the server in the middle
|
||||
# of the first one.
|
||||
time.sleep(1.5)
|
||||
assert httpd.reload()
|
||||
t.join()
|
||||
r: ExecResult = self.r
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
# reload will shut down the connection gracefully with GOAWAY
|
||||
# we expect to see a second connection opened afterwards
|
||||
assert r.total_connects == 2
|
||||
for idx, s in enumerate(r.stats):
|
||||
if s['num_connects'] > 0:
|
||||
log.debug(f'request {idx} connected')
|
||||
# this should take `count` seconds to retrieve
|
||||
assert r.duration >= timedelta(seconds=count)
|
||||
|
||||
# download files sequentially with delay, reload server for GOAWAY
|
||||
@pytest.mark.skipif(condition=not Env.have_h3_server(), reason="no h3 server")
|
||||
def test_03_02_h3_goaway(self, env: Env, httpd, nghttpx, repeat):
|
||||
proto = 'h3'
|
||||
count = 3
|
||||
self.r = None
|
||||
def long_run():
|
||||
curl = CurlClient(env=env)
|
||||
# send 10 chunks of 1024 bytes in a response body with 100ms delay in between
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=10&chunk_size=1024&chunk_delay=100ms'
|
||||
self.r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
|
||||
t = Thread(target=long_run)
|
||||
t.start()
|
||||
# each request will take a second, reload the server in the middle
|
||||
# of the first one.
|
||||
time.sleep(1.5)
|
||||
assert nghttpx.reload(timeout=timedelta(seconds=2))
|
||||
t.join()
|
||||
r: ExecResult = self.r
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
# reload will shut down the connection gracefully with GOAWAY
|
||||
# we expect to see a second connection opened afterwards
|
||||
assert r.total_connects == 2
|
||||
for idx, s in enumerate(r.stats):
|
||||
if s['num_connects'] > 0:
|
||||
log.debug(f'request {idx} connected')
|
||||
# this should take `count` seconds to retrieve
|
||||
assert r.duration >= timedelta(seconds=count)
|
||||
r.check_stats(count=count, exp_status=200, exp_exitcode=0)
|
||||
|
||||
|
||||
146
tests/http/test_04_stuttered.py
Normal file
146
tests/http/test_04_stuttered.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
from typing import Tuple, List, Dict
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestStuttered:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
|
||||
# download 1 file, check that delayed response works in general
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_04_01_download_1(self, env: Env, httpd, nghttpx, repeat,
|
||||
proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 1
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=100&chunk_size=100&chunk_delay=10ms'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download 50 files in 100 chunks a 100 bytes with 10ms delay between
|
||||
# prepend 100 file requests to warm up connection processing limits
|
||||
# (Apache2 increases # of parallel processed requests after successes)
|
||||
@pytest.mark.parametrize("proto", ['h2', 'h3'])
|
||||
def test_04_02_100_100_10(self, env: Env,
|
||||
httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 50
|
||||
warmups = 100
|
||||
curl = CurlClient(env=env)
|
||||
url1 = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-{warmups-1}]'
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count-1}]'\
|
||||
'&chunks=100&chunk_size=100&chunk_delay=10ms'
|
||||
r = curl.http_download(urls=[url1, urln], alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=warmups+count, exp_status=200)
|
||||
assert r.total_connects == 1
|
||||
t_avg, i_min, t_min, i_max, t_max = self.stats_spread(r.stats[warmups:], 'time_total')
|
||||
if t_max < (5 * t_min) and t_min < 2:
|
||||
log.warning(f'avg time of transfer: {t_avg} [{i_min}={t_min}, {i_max}={t_max}]')
|
||||
|
||||
# download 50 files in 1000 chunks a 10 bytes with 1ms delay between
|
||||
# prepend 100 file requests to warm up connection processing limits
|
||||
# (Apache2 increases # of parallel processed requests after successes)
|
||||
@pytest.mark.parametrize("proto", ['h2', 'h3'])
|
||||
def test_04_03_1000_10_1(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 50
|
||||
warmups = 100
|
||||
curl = CurlClient(env=env)
|
||||
url1 = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-{warmups-1}]'
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=1000&chunk_size=10&chunk_delay=100us'
|
||||
r = curl.http_download(urls=[url1, urln], alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=warmups+count, exp_status=200)
|
||||
assert r.total_connects == 1
|
||||
t_avg, i_min, t_min, i_max, t_max = self.stats_spread(r.stats[warmups:], 'time_total')
|
||||
if t_max < (5 * t_min):
|
||||
log.warning(f'avg time of transfer: {t_avg} [{i_min}={t_min}, {i_max}={t_max}]')
|
||||
|
||||
# download 50 files in 10000 chunks a 1 byte with 10us delay between
|
||||
# prepend 100 file requests to warm up connection processing limits
|
||||
# (Apache2 increases # of parallel processed requests after successes)
|
||||
@pytest.mark.parametrize("proto", ['h2', 'h3'])
|
||||
def test_04_04_1000_10_1(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 50
|
||||
warmups = 100
|
||||
curl = CurlClient(env=env)
|
||||
url1 = f'https://{env.authority_for(env.domain1, proto)}/data.json?[0-{warmups-1}]'
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=10000&chunk_size=1&chunk_delay=50us'
|
||||
r = curl.http_download(urls=[url1, urln], alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=warmups+count, exp_status=200)
|
||||
assert r.total_connects == 1
|
||||
t_avg, i_min, t_min, i_max, t_max = self.stats_spread(r.stats[warmups:], 'time_total')
|
||||
if t_max < (5 * t_min):
|
||||
log.warning(f'avg time of transfer: {t_avg} [{i_min}={t_min}, {i_max}={t_max}]')
|
||||
|
||||
def stats_spread(self, stats: List[Dict], key: str) -> Tuple[float, int, float, int, float]:
|
||||
stotals = 0.0
|
||||
s_min = 100.0
|
||||
i_min = -1
|
||||
s_max = 0.0
|
||||
i_max = -1
|
||||
for idx, s in enumerate(stats):
|
||||
val = float(s[key])
|
||||
stotals += val
|
||||
if val > s_max:
|
||||
s_max = val
|
||||
i_max = idx
|
||||
if val < s_min:
|
||||
s_min = val
|
||||
i_min = idx
|
||||
return stotals/len(stats), i_min, s_min, i_max, s_max
|
||||
92
tests/http/test_05_errors.py
Normal file
92
tests/http/test_05_errors.py
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional, Tuple, List, Dict
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient, ExecResult
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
@pytest.mark.skipif(condition=not Env.httpd_is_at_least('2.4.55'),
|
||||
reason=f"httpd version too old for this: {Env.httpd_version()}")
|
||||
class TestErrors:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
|
||||
# download 1 file, check that we get CURLE_PARTIAL_FILE
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_05_01_partial_1(self, env: Env, httpd, nghttpx, repeat,
|
||||
proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 1
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=3&chunk_size=16000&body_error=reset'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--retry', '0'
|
||||
])
|
||||
assert r.exit_code != 0, f'{r}'
|
||||
invalid_stats = []
|
||||
for idx, s in enumerate(r.stats):
|
||||
if 'exitcode' not in s or s['exitcode'] not in [18, 56, 92]:
|
||||
invalid_stats.append(f'request {idx} exit with {s["exitcode"]}')
|
||||
assert len(invalid_stats) == 0, f'failed: {invalid_stats}'
|
||||
|
||||
# download files, check that we get CURLE_PARTIAL_FILE for all
|
||||
@pytest.mark.parametrize("proto", ['h2', 'h3'])
|
||||
def test_05_02_partial_20(self, env: Env, httpd, nghttpx, repeat,
|
||||
proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
if proto == 'h3' and env.curl_uses_lib('quiche'):
|
||||
pytest.skip("quiche not reliable, sometimes reports success")
|
||||
count = 5
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, proto)}' \
|
||||
f'/curltest/tweak?id=[0-{count - 1}]'\
|
||||
'&chunks=3&chunk_size=16000&body_error=reset'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--retry', '0', '--parallel',
|
||||
])
|
||||
assert r.exit_code != 0, f'{r}'
|
||||
assert len(r.stats) == count, f'did not get all stats: {r}'
|
||||
invalid_stats = []
|
||||
for idx, s in enumerate(r.stats):
|
||||
if 'exitcode' not in s or s['exitcode'] not in [18, 56, 92]:
|
||||
invalid_stats.append(f'request {idx} exit with {s["exitcode"]}\n{s}')
|
||||
assert len(invalid_stats) == 0, f'failed: {invalid_stats}'
|
||||
86
tests/http/test_06_eyeballs.py
Normal file
86
tests/http/test_06_eyeballs.py
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional, Tuple, List, Dict
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient, ExecResult
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
@pytest.mark.skipif(condition=not Env.have_h3_server(),
|
||||
reason=f"missing HTTP/3 server")
|
||||
@pytest.mark.skipif(condition=not Env.have_h3_curl(),
|
||||
reason=f"curl built without HTTP/3")
|
||||
class TestEyeballs:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
|
||||
# download using only HTTP/3 on working server
|
||||
def test_06_01_h3_only(self, env: Env, httpd, nghttpx, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, "h3")}/data.json'
|
||||
r = curl.http_download(urls=[urln], extra_args=['--http3-only'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
assert r.stats[0]['http_version'] == '3'
|
||||
|
||||
# download using only HTTP/3 on missing server
|
||||
def test_06_02_h3_only(self, env: Env, httpd, nghttpx, repeat):
|
||||
nghttpx.stop_if_running()
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, "h3")}/data.json'
|
||||
r = curl.http_download(urls=[urln], extra_args=['--http3-only'])
|
||||
assert r.exit_code == 7, f'{r}' # could not connect
|
||||
|
||||
# download using HTTP/3 on missing server with fallback on h2
|
||||
def test_06_03_h3_fallback_h2(self, env: Env, httpd, nghttpx, repeat):
|
||||
nghttpx.stop_if_running()
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain1, "h3")}/data.json'
|
||||
r = curl.http_download(urls=[urln], extra_args=['--http3'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
assert r.stats[0]['http_version'] == '2'
|
||||
|
||||
# download using HTTP/3 on missing server with fallback on http/1.1
|
||||
def test_06_04_h3_fallback_h1(self, env: Env, httpd, nghttpx, repeat):
|
||||
nghttpx.stop_if_running()
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.authority_for(env.domain2, "h3")}/data.json'
|
||||
r = curl.http_download(urls=[urln], extra_args=['--http3'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
assert r.stats[0]['http_version'] == '1.1'
|
||||
221
tests/http/test_07_upload.py
Normal file
221
tests/http/test_07_upload.py
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestUpload:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, httpd, nghttpx):
|
||||
if env.have_h3():
|
||||
nghttpx.start_if_needed()
|
||||
env.make_data_file(indir=env.gen_dir, fname="data-100k", fsize=100*1024)
|
||||
env.make_data_file(indir=env.gen_dir, fname="data-10m", fsize=10*1024*1024)
|
||||
|
||||
# upload small data, check that this is what was echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_01_upload_1_small(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
data = '0123456789'
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-0]'
|
||||
r = curl.http_upload(urls=[url], data=data, alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
respdata = open(curl.response_file(0)).readlines()
|
||||
assert respdata == [data]
|
||||
|
||||
# upload large data, check that this is what was echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_02_upload_1_large(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
fdata = os.path.join(env.gen_dir, 'data-100k')
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-0]'
|
||||
r = curl.http_upload(urls=[url], data=f'@{fdata}', alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
indata = open(fdata).readlines()
|
||||
respdata = open(curl.response_file(0)).readlines()
|
||||
assert respdata == indata
|
||||
|
||||
# upload data sequentially, check that they were echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_10_upload_sequential(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 50
|
||||
data = '0123456789'
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-{count-1}]'
|
||||
r = curl.http_upload(urls=[url], data=data, alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == [data]
|
||||
|
||||
# upload data parallel, check that they were echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_11_upload_parallel(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 50
|
||||
data = '0123456789'
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-{count-1}]'
|
||||
r = curl.http_upload(urls=[url], data=data, alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == [data]
|
||||
|
||||
# upload large data sequentially, check that this is what was echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_20_upload_seq_large(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
fdata = os.path.join(env.gen_dir, 'data-100k')
|
||||
count = 50
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-{count-1}]'
|
||||
r = curl.http_upload(urls=[url], data=f'@{fdata}', alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
indata = open(fdata).readlines()
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == indata
|
||||
|
||||
# upload very large data sequentially, check that this is what was echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_12_upload_seq_large(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
fdata = os.path.join(env.gen_dir, 'data-10m')
|
||||
count = 2
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-{count-1}]'
|
||||
r = curl.http_upload(urls=[url], data=f'@{fdata}', alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
indata = open(fdata).readlines()
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == indata
|
||||
|
||||
# upload data parallel, check that they were echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_20_upload_parallel(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
count = 50
|
||||
data = '0123456789'
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-{count-1}]'
|
||||
r = curl.http_upload(urls=[url], data=data, alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == [data]
|
||||
|
||||
# upload large data parallel, check that this is what was echoed
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_21_upload_parallel_large(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
if proto == 'h3' and env.curl_uses_lib('quiche'):
|
||||
pytest.skip("quiche stalls on parallel, large uploads, unless --trace is used???")
|
||||
fdata = os.path.join(env.gen_dir, 'data-100k')
|
||||
count = 50
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/echo?id=[0-{count-1}]'
|
||||
r = curl.http_upload(urls=[url], data=f'@{fdata}', alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
indata = open(fdata).readlines()
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == indata
|
||||
|
||||
# PUT 100k
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_30_put_100k(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
fdata = os.path.join(env.gen_dir, 'data-100k')
|
||||
count = 1
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/put?id=[0-{count-1}]'
|
||||
r = curl.http_put(urls=[url], fdata=fdata, alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
exp_data = [f'{os.path.getsize(fdata)}']
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == exp_data
|
||||
|
||||
# PUT 10m
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_07_31_put_10m(self, env: Env, httpd, nghttpx, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3():
|
||||
pytest.skip("h3 not supported")
|
||||
fdata = os.path.join(env.gen_dir, 'data-10m')
|
||||
count = 1
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.authority_for(env.domain1, proto)}/curltest/put?id=[0-{count-1}]&chunk_delay=10ms'
|
||||
r = curl.http_put(urls=[url], fdata=fdata, alpn_proto=proto,
|
||||
extra_args=['--parallel'])
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
exp_data = [f'{os.path.getsize(fdata)}']
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
for i in range(count):
|
||||
respdata = open(curl.response_file(i)).readlines()
|
||||
assert respdata == exp_data
|
||||
|
||||
147
tests/http/test_08_caddy.py
Normal file
147
tests/http/test_08_caddy.py
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient, Caddy
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=not Env.has_caddy(), reason=f"missing caddy")
|
||||
class TestCaddy:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def caddy(self, env):
|
||||
caddy = Caddy(env=env)
|
||||
assert caddy.start()
|
||||
yield caddy
|
||||
caddy.stop()
|
||||
|
||||
def _make_docs_file(self, docs_dir: str, fname: str, fsize: int):
|
||||
fpath = os.path.join(docs_dir, fname)
|
||||
data1k = 1024*'x'
|
||||
flen = 0
|
||||
with open(fpath, 'w') as fd:
|
||||
while flen < fsize:
|
||||
fd.write(data1k)
|
||||
flen += len(data1k)
|
||||
return flen
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, caddy):
|
||||
self._make_docs_file(docs_dir=caddy.docs_dir, fname='data1.data', fsize=1024*1024)
|
||||
self._make_docs_file(docs_dir=caddy.docs_dir, fname='data10.data', fsize=10*1024*1024)
|
||||
self._make_docs_file(docs_dir=caddy.docs_dir, fname='data100.data', fsize=100*1024*1024)
|
||||
|
||||
# download 1 file
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_08_01_download_1(self, env: Env, caddy: Caddy, repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3_curl():
|
||||
pytest.skip("h3 not supported in curl")
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{caddy.port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download 1MB files sequentially
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_08_02_download_1mb_sequential(self, env: Env, caddy: Caddy,
|
||||
repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3_curl():
|
||||
pytest.skip("h3 not supported in curl")
|
||||
count = 50
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.domain1}:{caddy.port}/data1.data?[0-{count-1}]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
# sequential transfers will open 1 connection
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download 1MB files parallel
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_08_03_download_1mb_parallel(self, env: Env, caddy: Caddy,
|
||||
repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3_curl():
|
||||
pytest.skip("h3 not supported in curl")
|
||||
count = 50
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.domain1}:{caddy.port}/data1.data?[0-{count-1}]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--parallel'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
if proto == 'http/1.1':
|
||||
# http/1.1 parallel transfers will open multiple connections
|
||||
assert r.total_connects > 1
|
||||
else:
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download 10MB files sequentially
|
||||
@pytest.mark.parametrize("proto", ['h2', 'h3'])
|
||||
def test_08_04_download_10mb_sequential(self, env: Env, caddy: Caddy,
|
||||
repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3_curl():
|
||||
pytest.skip("h3 not supported in curl")
|
||||
if proto == 'h3' and env.curl_uses_lib('quiche'):
|
||||
pytest.skip("quiche stalls after a certain amount of data")
|
||||
count = 20
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.domain1}:{caddy.port}/data10.data?[0-{count-1}]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto)
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
# sequential transfers will open 1 connection
|
||||
assert r.total_connects == 1
|
||||
|
||||
# download 10MB files parallel
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
|
||||
def test_08_05_download_1mb_parallel(self, env: Env, caddy: Caddy,
|
||||
repeat, proto):
|
||||
if proto == 'h3' and not env.have_h3_curl():
|
||||
pytest.skip("h3 not supported in curl")
|
||||
if proto == 'h3' and env.curl_uses_lib('quiche'):
|
||||
pytest.skip("quiche stalls after a certain amount of data")
|
||||
count = 50
|
||||
curl = CurlClient(env=env)
|
||||
urln = f'https://{env.domain1}:{caddy.port}/data10.data?[0-{count-1}]'
|
||||
r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[
|
||||
'--parallel'
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=count, exp_status=200)
|
||||
if proto == 'http/1.1':
|
||||
# http/1.1 parallel transfers will open multiple connections
|
||||
assert r.total_connects > 1
|
||||
else:
|
||||
assert r.total_connects == 1
|
||||
|
||||
72
tests/http/test_09_push.py
Normal file
72
tests/http/test_09_push.py
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestPush:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, httpd):
|
||||
push_dir = os.path.join(httpd.docs_dir, 'push')
|
||||
if not os.path.exists(push_dir):
|
||||
os.makedirs(push_dir)
|
||||
env.make_data_file(indir=push_dir, fname="data1", fsize=100*1024)
|
||||
env.make_data_file(indir=push_dir, fname="data2", fsize=100*1024)
|
||||
env.make_data_file(indir=push_dir, fname="data3", fsize=100*1024)
|
||||
httpd.set_extra_config(env.domain1, [
|
||||
f'H2EarlyHints on',
|
||||
f'<Location /push/data1>',
|
||||
f' H2PushResource /push/data2',
|
||||
f'</Location>',
|
||||
f'<Location /push/data2>',
|
||||
f' H2PushResource /push/data1',
|
||||
f' H2PushResource /push/data3',
|
||||
f'</Location>',
|
||||
])
|
||||
# activate the new config
|
||||
httpd.reload()
|
||||
|
||||
# download a file that triggers a "103 Early Hints" response
|
||||
def test_09_01_early_hints(self, env: Env, httpd, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{env.https_port}/push/data1'
|
||||
r = curl.http_download(urls=[url], alpn_proto='h2', with_stats=False,
|
||||
with_headers=True)
|
||||
assert r.exit_code == 0, f'{r}'
|
||||
assert len(r.responses) == 2, f'{r.responses}'
|
||||
assert r.responses[0]['status'] == 103, f'{r.responses}'
|
||||
assert 'link' in r.responses[0]['header'], f'{r.responses[0]}'
|
||||
assert r.responses[0]['header']['link'] == '</push/data2>; rel=preload', f'{r.responses[0]}'
|
||||
133
tests/http/test_10_proxy.py
Normal file
133
tests/http/test_10_proxy.py
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestProxy:
|
||||
|
||||
@pytest.fixture(autouse=True, scope='class')
|
||||
def _class_scope(self, env, httpd):
|
||||
push_dir = os.path.join(httpd.docs_dir, 'push')
|
||||
if not os.path.exists(push_dir):
|
||||
os.makedirs(push_dir)
|
||||
|
||||
# download via http: proxy (no tunnel)
|
||||
def test_10_01_proxy_http(self, env: Env, httpd, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'http://localhost:{env.http_port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto='http/1.1', with_stats=True,
|
||||
extra_args=[
|
||||
'--proxy', f'http://{env.proxy_domain}:{env.proxy_port}/',
|
||||
'--resolve', f'{env.proxy_domain}:{env.proxy_port}:127.0.0.1',
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download via https: proxy (no tunnel)
|
||||
def test_10_02_proxy_https(self, env: Env, httpd, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'http://localhost:{env.http_port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto='http/1.1', with_stats=True,
|
||||
extra_args=[
|
||||
'--proxy', f'https://{env.proxy_domain}:{env.proxys_port}/',
|
||||
'--resolve', f'{env.proxy_domain}:{env.proxys_port}:127.0.0.1',
|
||||
'--proxy-cacert', env.ca.cert_file,
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download http: via http: proxytunnel
|
||||
def test_10_03_proxytunnel_http(self, env: Env, httpd, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'http://localhost:{env.http_port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto='http/1.1', with_stats=True,
|
||||
extra_args=[
|
||||
'--proxytunnel',
|
||||
'--proxy', f'http://{env.proxy_domain}:{env.proxy_port}/',
|
||||
'--resolve', f'{env.proxy_domain}:{env.proxy_port}:127.0.0.1',
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download http: via https: proxytunnel
|
||||
def test_10_04_proxy_https(self, env: Env, httpd, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'http://localhost:{env.http_port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto='http/1.1', with_stats=True,
|
||||
extra_args=[
|
||||
'--proxytunnel',
|
||||
'--proxy', f'https://{env.proxy_domain}:{env.proxys_port}/',
|
||||
'--resolve', f'{env.proxy_domain}:{env.proxys_port}:127.0.0.1',
|
||||
'--proxy-cacert', env.ca.cert_file,
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download https: with proto via http: proxytunnel
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2'])
|
||||
def test_10_05_proxytunnel_http(self, env: Env, httpd, proto, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://localhost:{env.https_port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto, with_stats=True,
|
||||
with_headers=True,
|
||||
extra_args=[
|
||||
'--proxytunnel',
|
||||
'--proxy', f'http://{env.proxy_domain}:{env.proxy_port}/',
|
||||
'--resolve', f'{env.proxy_domain}:{env.proxy_port}:127.0.0.1',
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
exp_proto = 'HTTP/2' if proto == 'h2' else 'HTTP/1.1'
|
||||
assert r.response['protocol'] == exp_proto
|
||||
|
||||
# download https: with proto via https: proxytunnel
|
||||
@pytest.mark.parametrize("proto", ['http/1.1', 'h2'])
|
||||
def test_10_06_proxy_https(self, env: Env, httpd, proto, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://localhost:{env.https_port}/data.json'
|
||||
r = curl.http_download(urls=[url], alpn_proto=proto, with_stats=True,
|
||||
with_headers=True,
|
||||
extra_args=[
|
||||
'--proxytunnel',
|
||||
'--proxy', f'https://{env.proxy_domain}:{env.proxys_port}/',
|
||||
'--resolve', f'{env.proxy_domain}:{env.proxys_port}:127.0.0.1',
|
||||
'--proxy-cacert', env.ca.cert_file,
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
exp_proto = 'HTTP/2' if proto == 'h2' else 'HTTP/1.1'
|
||||
assert r.response['protocol'] == exp_proto
|
||||
|
||||
129
tests/http/test_11_unix.py
Normal file
129
tests/http/test_11_unix.py
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
from threading import Thread
|
||||
import pytest
|
||||
|
||||
from testenv import Env, CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
class UDSFaker:
|
||||
|
||||
def __init__(self, path):
|
||||
self._uds_path = path
|
||||
self._done = False
|
||||
|
||||
@property
|
||||
def path(self):
|
||||
return self._uds_path
|
||||
|
||||
def start(self):
|
||||
def process(self):
|
||||
self._socket.listen(1)
|
||||
self._process()
|
||||
|
||||
try:
|
||||
os.unlink(self._uds_path)
|
||||
except OSError:
|
||||
if os.path.exists(self._uds_path):
|
||||
raise
|
||||
self._socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
self._socket.bind(self._uds_path)
|
||||
self._thread = Thread(target=process, daemon=True, args=[self])
|
||||
self._thread.start()
|
||||
|
||||
def stop(self):
|
||||
self._done = True
|
||||
self._socket.close()
|
||||
|
||||
def _process(self):
|
||||
while self._done is False:
|
||||
try:
|
||||
c, client_address = self._socket.accept()
|
||||
try:
|
||||
data = c.recv(16)
|
||||
c.sendall("""HTTP/1.1 200 Ok
|
||||
Server: UdsFaker
|
||||
Content-Type: application/json
|
||||
Content-Length: 19
|
||||
|
||||
{ "host": "faked" }""".encode())
|
||||
finally:
|
||||
c.close()
|
||||
|
||||
except ConnectionAbortedError:
|
||||
self._done = True
|
||||
|
||||
|
||||
|
||||
@pytest.mark.skipif(condition=Env.setup_incomplete(),
|
||||
reason=f"missing: {Env.incomplete_reason()}")
|
||||
class TestUnix:
|
||||
|
||||
@pytest.fixture(scope="class")
|
||||
def uds_faker(self, env: Env) -> UDSFaker:
|
||||
uds_path = os.path.join(env.gen_dir, 'uds_11.sock')
|
||||
faker = UDSFaker(path=uds_path)
|
||||
faker.start()
|
||||
yield faker
|
||||
faker.stop()
|
||||
|
||||
# download http: via unix socket
|
||||
def test_11_01_unix_connect_http(self, env: Env, httpd, uds_faker, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'http://{env.domain1}:{env.http_port}/data.json'
|
||||
r = curl.http_download(urls=[url], with_stats=True,
|
||||
extra_args=[
|
||||
'--unix-socket', uds_faker.path,
|
||||
])
|
||||
assert r.exit_code == 0
|
||||
r.check_stats(count=1, exp_status=200)
|
||||
|
||||
# download https: via unix socket
|
||||
def test_11_02_unix_connect_http(self, env: Env, httpd, uds_faker, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{env.https_port}/data.json'
|
||||
r = curl.http_download(urls=[url], with_stats=True,
|
||||
extra_args=[
|
||||
'--unix-socket', uds_faker.path,
|
||||
])
|
||||
assert r.exit_code == 35 # CONNECT_ERROR (as faker is not TLS)
|
||||
|
||||
# download HTTP/3 via unix socket
|
||||
def test_11_03_unix_connect_quic(self, env: Env, httpd, uds_faker, repeat):
|
||||
curl = CurlClient(env=env)
|
||||
url = f'https://{env.domain1}:{env.https_port}/data.json'
|
||||
r = curl.http_download(urls=[url], with_stats=True,
|
||||
alpn_proto='h3',
|
||||
extra_args=[
|
||||
'--unix-socket', uds_faker.path,
|
||||
])
|
||||
assert r.exit_code == 96 # QUIC CONNECT ERROR
|
||||
32
tests/http/testenv/__init__.py
Normal file
32
tests/http/testenv/__init__.py
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
from .env import Env
|
||||
from .certs import TestCA, Credentials
|
||||
from .caddy import Caddy
|
||||
from .httpd import Httpd
|
||||
from .curl import CurlClient, ExecResult
|
||||
from .nghttpx import Nghttpx
|
||||
169
tests/http/testenv/caddy.py
Normal file
169
tests/http/testenv/caddy.py
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
from datetime import timedelta, datetime
|
||||
from json import JSONEncoder
|
||||
|
||||
from .curl import CurlClient
|
||||
from .env import Env
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Caddy:
|
||||
|
||||
def __init__(self, env: Env):
|
||||
self.env = env
|
||||
self._caddy = os.environ['CADDY'] if 'CADDY' in os.environ else env.caddy
|
||||
self._caddy_dir = os.path.join(env.gen_dir, 'caddy')
|
||||
self._docs_dir = os.path.join(self._caddy_dir, 'docs')
|
||||
self._conf_file = os.path.join(self._caddy_dir, 'Caddyfile')
|
||||
self._error_log = os.path.join(self._caddy_dir, 'caddy.log')
|
||||
self._tmp_dir = os.path.join(self._caddy_dir, 'tmp')
|
||||
self._process = None
|
||||
self._rmf(self._error_log)
|
||||
|
||||
@property
|
||||
def docs_dir(self):
|
||||
return self._docs_dir
|
||||
|
||||
@property
|
||||
def port(self) -> str:
|
||||
return self.env.caddy_https_port
|
||||
|
||||
def clear_logs(self):
|
||||
self._rmf(self._error_log)
|
||||
|
||||
def is_running(self):
|
||||
if self._process:
|
||||
self._process.poll()
|
||||
return self._process.returncode is None
|
||||
return False
|
||||
|
||||
def start_if_needed(self):
|
||||
if not self.is_running():
|
||||
return self.start()
|
||||
return True
|
||||
|
||||
def start(self, wait_live=True):
|
||||
self._mkpath(self._tmp_dir)
|
||||
if self._process:
|
||||
self.stop()
|
||||
self._write_config()
|
||||
args = [
|
||||
self._caddy, 'run'
|
||||
]
|
||||
caddyerr = open(self._error_log, 'a')
|
||||
self._process = subprocess.Popen(args=args, cwd=self._caddy_dir, stderr=caddyerr)
|
||||
if self._process.returncode is not None:
|
||||
return False
|
||||
return not wait_live or self.wait_live(timeout=timedelta(seconds=5))
|
||||
|
||||
def stop_if_running(self):
|
||||
if self.is_running():
|
||||
return self.stop()
|
||||
return True
|
||||
|
||||
def stop(self, wait_dead=True):
|
||||
self._mkpath(self._tmp_dir)
|
||||
if self._process:
|
||||
self._process.terminate()
|
||||
self._process.wait(timeout=2)
|
||||
self._process = None
|
||||
return not wait_dead or self.wait_dead(timeout=timedelta(seconds=5))
|
||||
return True
|
||||
|
||||
def restart(self):
|
||||
self.stop()
|
||||
return self.start()
|
||||
|
||||
def wait_dead(self, timeout: timedelta):
|
||||
curl = CurlClient(env=self.env, run_dir=self._tmp_dir)
|
||||
try_until = datetime.now() + timeout
|
||||
while datetime.now() < try_until:
|
||||
check_url = f'https://{self.env.domain1}:{self.port}/'
|
||||
r = curl.http_get(url=check_url)
|
||||
if r.exit_code != 0:
|
||||
return True
|
||||
log.debug(f'waiting for caddy to stop responding: {r}')
|
||||
time.sleep(.1)
|
||||
log.debug(f"Server still responding after {timeout}")
|
||||
return False
|
||||
|
||||
def wait_live(self, timeout: timedelta):
|
||||
curl = CurlClient(env=self.env, run_dir=self._tmp_dir)
|
||||
try_until = datetime.now() + timeout
|
||||
while datetime.now() < try_until:
|
||||
check_url = f'https://{self.env.domain1}:{self.port}/'
|
||||
r = curl.http_get(url=check_url)
|
||||
if r.exit_code == 0:
|
||||
return True
|
||||
log.error(f'curl: {r}')
|
||||
log.debug(f'waiting for caddy to become responsive: {r}')
|
||||
time.sleep(.1)
|
||||
log.error(f"Server still not responding after {timeout}")
|
||||
return False
|
||||
|
||||
def _rmf(self, path):
|
||||
if os.path.exists(path):
|
||||
return os.remove(path)
|
||||
|
||||
def _mkpath(self, path):
|
||||
if not os.path.exists(path):
|
||||
return os.makedirs(path)
|
||||
|
||||
def _write_config(self):
|
||||
domain1 = self.env.domain1
|
||||
creds1 = self.env.get_credentials(domain1)
|
||||
self._mkpath(self._docs_dir)
|
||||
self._mkpath(self._tmp_dir)
|
||||
with open(os.path.join(self._docs_dir, 'data.json'), 'w') as fd:
|
||||
data = {
|
||||
'server': f'{domain1}',
|
||||
}
|
||||
fd.write(JSONEncoder().encode(data))
|
||||
with open(self._conf_file, 'w') as fd:
|
||||
conf = [ # base server config
|
||||
f'{{',
|
||||
f' http_port {self.env.caddy_http_port}',
|
||||
f' https_port {self.env.caddy_https_port}',
|
||||
f' servers :{self.env.caddy_https_port} {{',
|
||||
f' protocols h3 h2 h1',
|
||||
f' }}',
|
||||
f'}}',
|
||||
f'{domain1}:{self.env.caddy_https_port} {{',
|
||||
f' file_server * {{',
|
||||
f' root {self._docs_dir}',
|
||||
f' }}',
|
||||
f' tls {creds1.cert_file} {creds1.pkey_file}',
|
||||
f'}}',
|
||||
]
|
||||
fd.write("\n".join(conf))
|
||||
528
tests/http/testenv/certs.py
Normal file
528
tests/http/testenv/certs.py
Normal file
|
|
@ -0,0 +1,528 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import os
|
||||
import re
|
||||
from datetime import timedelta, datetime
|
||||
from typing import List, Any, Optional
|
||||
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.backends import default_backend
|
||||
from cryptography.hazmat.primitives import hashes
|
||||
from cryptography.hazmat.primitives.asymmetric import ec, rsa
|
||||
from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurvePrivateKey
|
||||
from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
|
||||
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, NoEncryption, load_pem_private_key
|
||||
from cryptography.x509 import ExtendedKeyUsageOID, NameOID
|
||||
|
||||
|
||||
EC_SUPPORTED = {}
|
||||
EC_SUPPORTED.update([(curve.name.upper(), curve) for curve in [
|
||||
ec.SECP192R1,
|
||||
ec.SECP224R1,
|
||||
ec.SECP256R1,
|
||||
ec.SECP384R1,
|
||||
]])
|
||||
|
||||
|
||||
def _private_key(key_type):
|
||||
if isinstance(key_type, str):
|
||||
key_type = key_type.upper()
|
||||
m = re.match(r'^(RSA)?(\d+)$', key_type)
|
||||
if m:
|
||||
key_type = int(m.group(2))
|
||||
|
||||
if isinstance(key_type, int):
|
||||
return rsa.generate_private_key(
|
||||
public_exponent=65537,
|
||||
key_size=key_type,
|
||||
backend=default_backend()
|
||||
)
|
||||
if not isinstance(key_type, ec.EllipticCurve) and key_type in EC_SUPPORTED:
|
||||
key_type = EC_SUPPORTED[key_type]
|
||||
return ec.generate_private_key(
|
||||
curve=key_type,
|
||||
backend=default_backend()
|
||||
)
|
||||
|
||||
|
||||
class CertificateSpec:
|
||||
|
||||
def __init__(self, name: Optional[str] = None,
|
||||
domains: Optional[List[str]] = None,
|
||||
email: Optional[str] = None,
|
||||
key_type: Optional[str] = None,
|
||||
single_file: bool = False,
|
||||
valid_from: timedelta = timedelta(days=-1),
|
||||
valid_to: timedelta = timedelta(days=89),
|
||||
client: bool = False,
|
||||
sub_specs: Optional[List['CertificateSpec']] = None):
|
||||
self._name = name
|
||||
self.domains = domains
|
||||
self.client = client
|
||||
self.email = email
|
||||
self.key_type = key_type
|
||||
self.single_file = single_file
|
||||
self.valid_from = valid_from
|
||||
self.valid_to = valid_to
|
||||
self.sub_specs = sub_specs
|
||||
|
||||
@property
|
||||
def name(self) -> Optional[str]:
|
||||
if self._name:
|
||||
return self._name
|
||||
elif self.domains:
|
||||
return self.domains[0]
|
||||
return None
|
||||
|
||||
@property
|
||||
def type(self) -> Optional[str]:
|
||||
if self.domains and len(self.domains):
|
||||
return "server"
|
||||
elif self.client:
|
||||
return "client"
|
||||
elif self.name:
|
||||
return "ca"
|
||||
return None
|
||||
|
||||
|
||||
class Credentials:
|
||||
|
||||
def __init__(self,
|
||||
name: str,
|
||||
cert: Any,
|
||||
pkey: Any,
|
||||
issuer: Optional['Credentials'] = None):
|
||||
self._name = name
|
||||
self._cert = cert
|
||||
self._pkey = pkey
|
||||
self._issuer = issuer
|
||||
self._cert_file = None
|
||||
self._pkey_file = None
|
||||
self._store = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return self._name
|
||||
|
||||
@property
|
||||
def subject(self) -> x509.Name:
|
||||
return self._cert.subject
|
||||
|
||||
@property
|
||||
def key_type(self):
|
||||
if isinstance(self._pkey, RSAPrivateKey):
|
||||
return f"rsa{self._pkey.key_size}"
|
||||
elif isinstance(self._pkey, EllipticCurvePrivateKey):
|
||||
return f"{self._pkey.curve.name}"
|
||||
else:
|
||||
raise Exception(f"unknown key type: {self._pkey}")
|
||||
|
||||
@property
|
||||
def private_key(self) -> Any:
|
||||
return self._pkey
|
||||
|
||||
@property
|
||||
def certificate(self) -> Any:
|
||||
return self._cert
|
||||
|
||||
@property
|
||||
def cert_pem(self) -> bytes:
|
||||
return self._cert.public_bytes(Encoding.PEM)
|
||||
|
||||
@property
|
||||
def pkey_pem(self) -> bytes:
|
||||
return self._pkey.private_bytes(
|
||||
Encoding.PEM,
|
||||
PrivateFormat.TraditionalOpenSSL if self.key_type.startswith('rsa') else PrivateFormat.PKCS8,
|
||||
NoEncryption())
|
||||
|
||||
@property
|
||||
def issuer(self) -> Optional['Credentials']:
|
||||
return self._issuer
|
||||
|
||||
def set_store(self, store: 'CertStore'):
|
||||
self._store = store
|
||||
|
||||
def set_files(self, cert_file: str, pkey_file: Optional[str] = None,
|
||||
combined_file: Optional[str] = None):
|
||||
self._cert_file = cert_file
|
||||
self._pkey_file = pkey_file
|
||||
self._combined_file = combined_file
|
||||
|
||||
@property
|
||||
def cert_file(self) -> str:
|
||||
return self._cert_file
|
||||
|
||||
@property
|
||||
def pkey_file(self) -> Optional[str]:
|
||||
return self._pkey_file
|
||||
|
||||
@property
|
||||
def combined_file(self) -> Optional[str]:
|
||||
return self._combined_file
|
||||
|
||||
def get_first(self, name) -> Optional['Credentials']:
|
||||
creds = self._store.get_credentials_for_name(name) if self._store else []
|
||||
return creds[0] if len(creds) else None
|
||||
|
||||
def get_credentials_for_name(self, name) -> List['Credentials']:
|
||||
return self._store.get_credentials_for_name(name) if self._store else []
|
||||
|
||||
def issue_certs(self, specs: List[CertificateSpec],
|
||||
chain: Optional[List['Credentials']] = None) -> List['Credentials']:
|
||||
return [self.issue_cert(spec=spec, chain=chain) for spec in specs]
|
||||
|
||||
def issue_cert(self, spec: CertificateSpec,
|
||||
chain: Optional[List['Credentials']] = None) -> 'Credentials':
|
||||
key_type = spec.key_type if spec.key_type else self.key_type
|
||||
creds = None
|
||||
if self._store:
|
||||
creds = self._store.load_credentials(
|
||||
name=spec.name, key_type=key_type, single_file=spec.single_file, issuer=self)
|
||||
if creds is None:
|
||||
creds = TestCA.create_credentials(spec=spec, issuer=self, key_type=key_type,
|
||||
valid_from=spec.valid_from, valid_to=spec.valid_to)
|
||||
if self._store:
|
||||
self._store.save(creds, single_file=spec.single_file)
|
||||
if spec.type == "ca":
|
||||
self._store.save_chain(creds, "ca", with_root=True)
|
||||
|
||||
if spec.sub_specs:
|
||||
if self._store:
|
||||
sub_store = CertStore(fpath=os.path.join(self._store.path, creds.name))
|
||||
creds.set_store(sub_store)
|
||||
subchain = chain.copy() if chain else []
|
||||
subchain.append(self)
|
||||
creds.issue_certs(spec.sub_specs, chain=subchain)
|
||||
return creds
|
||||
|
||||
|
||||
class CertStore:
|
||||
|
||||
def __init__(self, fpath: str):
|
||||
self._store_dir = fpath
|
||||
if not os.path.exists(self._store_dir):
|
||||
os.makedirs(self._store_dir)
|
||||
self._creds_by_name = {}
|
||||
|
||||
@property
|
||||
def path(self) -> str:
|
||||
return self._store_dir
|
||||
|
||||
def save(self, creds: Credentials, name: Optional[str] = None,
|
||||
chain: Optional[List[Credentials]] = None,
|
||||
single_file: bool = False) -> None:
|
||||
name = name if name is not None else creds.name
|
||||
cert_file = self.get_cert_file(name=name, key_type=creds.key_type)
|
||||
pkey_file = self.get_pkey_file(name=name, key_type=creds.key_type)
|
||||
comb_file = self.get_combined_file(name=name, key_type=creds.key_type)
|
||||
if single_file:
|
||||
pkey_file = None
|
||||
with open(cert_file, "wb") as fd:
|
||||
fd.write(creds.cert_pem)
|
||||
if chain:
|
||||
for c in chain:
|
||||
fd.write(c.cert_pem)
|
||||
if pkey_file is None:
|
||||
fd.write(creds.pkey_pem)
|
||||
if pkey_file is not None:
|
||||
with open(pkey_file, "wb") as fd:
|
||||
fd.write(creds.pkey_pem)
|
||||
with open(comb_file, "wb") as fd:
|
||||
fd.write(creds.cert_pem)
|
||||
if chain:
|
||||
for c in chain:
|
||||
fd.write(c.cert_pem)
|
||||
fd.write(creds.pkey_pem)
|
||||
creds.set_files(cert_file, pkey_file, comb_file)
|
||||
self._add_credentials(name, creds)
|
||||
|
||||
def save_chain(self, creds: Credentials, infix: str, with_root=False):
|
||||
name = creds.name
|
||||
chain = [creds]
|
||||
while creds.issuer is not None:
|
||||
creds = creds.issuer
|
||||
chain.append(creds)
|
||||
if not with_root and len(chain) > 1:
|
||||
chain = chain[:-1]
|
||||
chain_file = os.path.join(self._store_dir, f'{name}-{infix}.pem')
|
||||
with open(chain_file, "wb") as fd:
|
||||
for c in chain:
|
||||
fd.write(c.cert_pem)
|
||||
|
||||
def _add_credentials(self, name: str, creds: Credentials):
|
||||
if name not in self._creds_by_name:
|
||||
self._creds_by_name[name] = []
|
||||
self._creds_by_name[name].append(creds)
|
||||
|
||||
def get_credentials_for_name(self, name) -> List[Credentials]:
|
||||
return self._creds_by_name[name] if name in self._creds_by_name else []
|
||||
|
||||
def get_cert_file(self, name: str, key_type=None) -> str:
|
||||
key_infix = ".{0}".format(key_type) if key_type is not None else ""
|
||||
return os.path.join(self._store_dir, f'{name}{key_infix}.cert.pem')
|
||||
|
||||
def get_pkey_file(self, name: str, key_type=None) -> str:
|
||||
key_infix = ".{0}".format(key_type) if key_type is not None else ""
|
||||
return os.path.join(self._store_dir, f'{name}{key_infix}.pkey.pem')
|
||||
|
||||
def get_combined_file(self, name: str, key_type=None) -> str:
|
||||
return os.path.join(self._store_dir, f'{name}.pem')
|
||||
|
||||
def load_pem_cert(self, fpath: str) -> x509.Certificate:
|
||||
with open(fpath) as fd:
|
||||
return x509.load_pem_x509_certificate("".join(fd.readlines()).encode())
|
||||
|
||||
def load_pem_pkey(self, fpath: str):
|
||||
with open(fpath) as fd:
|
||||
return load_pem_private_key("".join(fd.readlines()).encode(), password=None)
|
||||
|
||||
def load_credentials(self, name: str, key_type=None,
|
||||
single_file: bool = False,
|
||||
issuer: Optional[Credentials] = None):
|
||||
cert_file = self.get_cert_file(name=name, key_type=key_type)
|
||||
pkey_file = cert_file if single_file else self.get_pkey_file(name=name, key_type=key_type)
|
||||
comb_file = self.get_combined_file(name=name, key_type=key_type)
|
||||
if os.path.isfile(cert_file) and os.path.isfile(pkey_file):
|
||||
cert = self.load_pem_cert(cert_file)
|
||||
pkey = self.load_pem_pkey(pkey_file)
|
||||
creds = Credentials(name=name, cert=cert, pkey=pkey, issuer=issuer)
|
||||
creds.set_store(self)
|
||||
creds.set_files(cert_file, pkey_file, comb_file)
|
||||
self._add_credentials(name, creds)
|
||||
return creds
|
||||
return None
|
||||
|
||||
|
||||
class TestCA:
|
||||
|
||||
@classmethod
|
||||
def create_root(cls, name: str, store_dir: str, key_type: str = "rsa2048") -> Credentials:
|
||||
store = CertStore(fpath=store_dir)
|
||||
creds = store.load_credentials(name="ca", key_type=key_type, issuer=None)
|
||||
if creds is None:
|
||||
creds = TestCA._make_ca_credentials(name=name, key_type=key_type)
|
||||
store.save(creds, name="ca")
|
||||
creds.set_store(store)
|
||||
return creds
|
||||
|
||||
@staticmethod
|
||||
def create_credentials(spec: CertificateSpec, issuer: Credentials, key_type: Any,
|
||||
valid_from: timedelta = timedelta(days=-1),
|
||||
valid_to: timedelta = timedelta(days=89),
|
||||
) -> Credentials:
|
||||
"""Create a certificate signed by this CA for the given domains.
|
||||
:returns: the certificate and private key PEM file paths
|
||||
"""
|
||||
if spec.domains and len(spec.domains):
|
||||
creds = TestCA._make_server_credentials(name=spec.name, domains=spec.domains,
|
||||
issuer=issuer, valid_from=valid_from,
|
||||
valid_to=valid_to, key_type=key_type)
|
||||
elif spec.client:
|
||||
creds = TestCA._make_client_credentials(name=spec.name, issuer=issuer,
|
||||
email=spec.email, valid_from=valid_from,
|
||||
valid_to=valid_to, key_type=key_type)
|
||||
elif spec.name:
|
||||
creds = TestCA._make_ca_credentials(name=spec.name, issuer=issuer,
|
||||
valid_from=valid_from, valid_to=valid_to,
|
||||
key_type=key_type)
|
||||
else:
|
||||
raise Exception(f"unrecognized certificate specification: {spec}")
|
||||
return creds
|
||||
|
||||
@staticmethod
|
||||
def _make_x509_name(org_name: str = None, common_name: str = None, parent: x509.Name = None) -> x509.Name:
|
||||
name_pieces = []
|
||||
if org_name:
|
||||
oid = NameOID.ORGANIZATIONAL_UNIT_NAME if parent else NameOID.ORGANIZATION_NAME
|
||||
name_pieces.append(x509.NameAttribute(oid, org_name))
|
||||
elif common_name:
|
||||
name_pieces.append(x509.NameAttribute(NameOID.COMMON_NAME, common_name))
|
||||
if parent:
|
||||
name_pieces.extend([rdn for rdn in parent])
|
||||
return x509.Name(name_pieces)
|
||||
|
||||
@staticmethod
|
||||
def _make_csr(
|
||||
subject: x509.Name,
|
||||
pkey: Any,
|
||||
issuer_subject: Optional[Credentials],
|
||||
valid_from_delta: timedelta = None,
|
||||
valid_until_delta: timedelta = None
|
||||
):
|
||||
pubkey = pkey.public_key()
|
||||
issuer_subject = issuer_subject if issuer_subject is not None else subject
|
||||
|
||||
valid_from = datetime.now()
|
||||
if valid_until_delta is not None:
|
||||
valid_from += valid_from_delta
|
||||
valid_until = datetime.now()
|
||||
if valid_until_delta is not None:
|
||||
valid_until += valid_until_delta
|
||||
|
||||
return (
|
||||
x509.CertificateBuilder()
|
||||
.subject_name(subject)
|
||||
.issuer_name(issuer_subject)
|
||||
.public_key(pubkey)
|
||||
.not_valid_before(valid_from)
|
||||
.not_valid_after(valid_until)
|
||||
.serial_number(x509.random_serial_number())
|
||||
.add_extension(
|
||||
x509.SubjectKeyIdentifier.from_public_key(pubkey),
|
||||
critical=False,
|
||||
)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _add_ca_usages(csr: Any) -> Any:
|
||||
return csr.add_extension(
|
||||
x509.BasicConstraints(ca=True, path_length=9),
|
||||
critical=True,
|
||||
).add_extension(
|
||||
x509.KeyUsage(
|
||||
digital_signature=True,
|
||||
content_commitment=False,
|
||||
key_encipherment=False,
|
||||
data_encipherment=False,
|
||||
key_agreement=False,
|
||||
key_cert_sign=True,
|
||||
crl_sign=True,
|
||||
encipher_only=False,
|
||||
decipher_only=False),
|
||||
critical=True
|
||||
).add_extension(
|
||||
x509.ExtendedKeyUsage([
|
||||
ExtendedKeyUsageOID.CLIENT_AUTH,
|
||||
ExtendedKeyUsageOID.SERVER_AUTH,
|
||||
ExtendedKeyUsageOID.CODE_SIGNING,
|
||||
]),
|
||||
critical=True
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _add_leaf_usages(csr: Any, domains: List[str], issuer: Credentials) -> Any:
|
||||
return csr.add_extension(
|
||||
x509.BasicConstraints(ca=False, path_length=None),
|
||||
critical=True,
|
||||
).add_extension(
|
||||
x509.AuthorityKeyIdentifier.from_issuer_subject_key_identifier(
|
||||
issuer.certificate.extensions.get_extension_for_class(
|
||||
x509.SubjectKeyIdentifier).value),
|
||||
critical=False
|
||||
).add_extension(
|
||||
x509.SubjectAlternativeName([x509.DNSName(domain) for domain in domains]),
|
||||
critical=True,
|
||||
).add_extension(
|
||||
x509.ExtendedKeyUsage([
|
||||
ExtendedKeyUsageOID.SERVER_AUTH,
|
||||
]),
|
||||
critical=True
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _add_client_usages(csr: Any, issuer: Credentials, rfc82name: str = None) -> Any:
|
||||
cert = csr.add_extension(
|
||||
x509.BasicConstraints(ca=False, path_length=None),
|
||||
critical=True,
|
||||
).add_extension(
|
||||
x509.AuthorityKeyIdentifier.from_issuer_subject_key_identifier(
|
||||
issuer.certificate.extensions.get_extension_for_class(
|
||||
x509.SubjectKeyIdentifier).value),
|
||||
critical=False
|
||||
)
|
||||
if rfc82name:
|
||||
cert.add_extension(
|
||||
x509.SubjectAlternativeName([x509.RFC822Name(rfc82name)]),
|
||||
critical=True,
|
||||
)
|
||||
cert.add_extension(
|
||||
x509.ExtendedKeyUsage([
|
||||
ExtendedKeyUsageOID.CLIENT_AUTH,
|
||||
]),
|
||||
critical=True
|
||||
)
|
||||
return cert
|
||||
|
||||
@staticmethod
|
||||
def _make_ca_credentials(name, key_type: Any,
|
||||
issuer: Credentials = None,
|
||||
valid_from: timedelta = timedelta(days=-1),
|
||||
valid_to: timedelta = timedelta(days=89),
|
||||
) -> Credentials:
|
||||
pkey = _private_key(key_type=key_type)
|
||||
if issuer is not None:
|
||||
issuer_subject = issuer.certificate.subject
|
||||
issuer_key = issuer.private_key
|
||||
else:
|
||||
issuer_subject = None
|
||||
issuer_key = pkey
|
||||
subject = TestCA._make_x509_name(org_name=name, parent=issuer.subject if issuer else None)
|
||||
csr = TestCA._make_csr(subject=subject,
|
||||
issuer_subject=issuer_subject, pkey=pkey,
|
||||
valid_from_delta=valid_from, valid_until_delta=valid_to)
|
||||
csr = TestCA._add_ca_usages(csr)
|
||||
cert = csr.sign(private_key=issuer_key,
|
||||
algorithm=hashes.SHA256(),
|
||||
backend=default_backend())
|
||||
return Credentials(name=name, cert=cert, pkey=pkey, issuer=issuer)
|
||||
|
||||
@staticmethod
|
||||
def _make_server_credentials(name: str, domains: List[str], issuer: Credentials,
|
||||
key_type: Any,
|
||||
valid_from: timedelta = timedelta(days=-1),
|
||||
valid_to: timedelta = timedelta(days=89),
|
||||
) -> Credentials:
|
||||
name = name
|
||||
pkey = _private_key(key_type=key_type)
|
||||
subject = TestCA._make_x509_name(common_name=name, parent=issuer.subject)
|
||||
csr = TestCA._make_csr(subject=subject,
|
||||
issuer_subject=issuer.certificate.subject, pkey=pkey,
|
||||
valid_from_delta=valid_from, valid_until_delta=valid_to)
|
||||
csr = TestCA._add_leaf_usages(csr, domains=domains, issuer=issuer)
|
||||
cert = csr.sign(private_key=issuer.private_key,
|
||||
algorithm=hashes.SHA256(),
|
||||
backend=default_backend())
|
||||
return Credentials(name=name, cert=cert, pkey=pkey, issuer=issuer)
|
||||
|
||||
@staticmethod
|
||||
def _make_client_credentials(name: str,
|
||||
issuer: Credentials, email: Optional[str],
|
||||
key_type: Any,
|
||||
valid_from: timedelta = timedelta(days=-1),
|
||||
valid_to: timedelta = timedelta(days=89),
|
||||
) -> Credentials:
|
||||
pkey = _private_key(key_type=key_type)
|
||||
subject = TestCA._make_x509_name(common_name=name, parent=issuer.subject)
|
||||
csr = TestCA._make_csr(subject=subject,
|
||||
issuer_subject=issuer.certificate.subject, pkey=pkey,
|
||||
valid_from_delta=valid_from, valid_until_delta=valid_to)
|
||||
csr = TestCA._add_client_usages(csr, issuer=issuer, rfc82name=email)
|
||||
cert = csr.sign(private_key=issuer.private_key,
|
||||
algorithm=hashes.SHA256(),
|
||||
backend=default_backend())
|
||||
return Credentials(name=name, cert=cert, pkey=pkey, issuer=issuer)
|
||||
442
tests/http/testenv/curl.py
Normal file
442
tests/http/testenv/curl.py
Normal file
|
|
@ -0,0 +1,442 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
from datetime import timedelta, datetime
|
||||
from typing import List, Optional, Dict
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from .env import Env
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ExecResult:
|
||||
|
||||
def __init__(self, args: List[str], exit_code: int,
|
||||
stdout: List[str], stderr: List[str],
|
||||
duration: Optional[timedelta] = None,
|
||||
with_stats: bool = False):
|
||||
self._args = args
|
||||
self._exit_code = exit_code
|
||||
self._stdout = stdout
|
||||
self._stderr = stderr
|
||||
self._duration = duration if duration is not None else timedelta()
|
||||
self._response = None
|
||||
self._responses = []
|
||||
self._results = {}
|
||||
self._assets = []
|
||||
self._stats = []
|
||||
self._json_out = None
|
||||
self._with_stats = with_stats
|
||||
if with_stats:
|
||||
self._parse_stats()
|
||||
else:
|
||||
# noinspection PyBroadException
|
||||
try:
|
||||
out = ''.join(self._stdout)
|
||||
self._json_out = json.loads(out)
|
||||
except:
|
||||
pass
|
||||
|
||||
def __repr__(self):
|
||||
return f"ExecResult[code={self.exit_code}, args={self._args}, stdout={self._stdout}, stderr={self._stderr}]"
|
||||
|
||||
def _parse_stats(self):
|
||||
self._stats = []
|
||||
for l in self._stdout:
|
||||
try:
|
||||
self._stats.append(json.loads(l))
|
||||
except:
|
||||
log.error(f'not a JSON stat: {l}')
|
||||
log.error(f'stdout is: {"".join(self._stdout)}')
|
||||
break
|
||||
|
||||
@property
|
||||
def exit_code(self) -> int:
|
||||
return self._exit_code
|
||||
|
||||
@property
|
||||
def args(self) -> List[str]:
|
||||
return self._args
|
||||
|
||||
@property
|
||||
def outraw(self) -> bytes:
|
||||
return ''.join(self._stdout).encode()
|
||||
|
||||
@property
|
||||
def stdout(self) -> str:
|
||||
return ''.join(self._stdout)
|
||||
|
||||
@property
|
||||
def json(self) -> Optional[Dict]:
|
||||
"""Output as JSON dictionary or None if not parseable."""
|
||||
return self._json_out
|
||||
|
||||
@property
|
||||
def stderr(self) -> str:
|
||||
return ''.join(self._stderr)
|
||||
|
||||
@property
|
||||
def duration(self) -> timedelta:
|
||||
return self._duration
|
||||
|
||||
@property
|
||||
def response(self) -> Optional[Dict]:
|
||||
return self._response
|
||||
|
||||
@property
|
||||
def responses(self) -> List[Dict]:
|
||||
return self._responses
|
||||
|
||||
@property
|
||||
def results(self) -> Dict:
|
||||
return self._results
|
||||
|
||||
@property
|
||||
def assets(self) -> List:
|
||||
return self._assets
|
||||
|
||||
@property
|
||||
def with_stats(self) -> bool:
|
||||
return self._with_stats
|
||||
|
||||
@property
|
||||
def stats(self) -> List:
|
||||
return self._stats
|
||||
|
||||
@property
|
||||
def total_connects(self) -> Optional[int]:
|
||||
if len(self.stats):
|
||||
n = 0
|
||||
for stat in self.stats:
|
||||
n += stat['num_connects']
|
||||
return n
|
||||
return None
|
||||
|
||||
def add_response(self, resp: Dict):
|
||||
self._response = resp
|
||||
self._responses.append(resp)
|
||||
|
||||
def add_results(self, results: Dict):
|
||||
self._results.update(results)
|
||||
if 'response' in results:
|
||||
self.add_response(results['response'])
|
||||
|
||||
def add_assets(self, assets: List):
|
||||
self._assets.extend(assets)
|
||||
|
||||
def check_responses(self, count: int, exp_status: Optional[int] = None,
|
||||
exp_exitcode: Optional[int] = None):
|
||||
assert len(self.responses) == count, \
|
||||
f'response count: expected {count}, got {len(self.responses)}'
|
||||
if exp_status is not None:
|
||||
for idx, x in enumerate(self.responses):
|
||||
assert x['status'] == exp_status, \
|
||||
f'response #{idx} unexpectedstatus: {x["status"]}'
|
||||
if exp_exitcode is not None:
|
||||
for idx, x in enumerate(self.responses):
|
||||
if 'exitcode' in x:
|
||||
assert x['exitcode'] == 0, f'response #{idx} exitcode: {x["exitcode"]}'
|
||||
if self.with_stats:
|
||||
assert len(self.stats) == count, f'{self}'
|
||||
|
||||
def check_stats(self, count: int, exp_status: Optional[int] = None,
|
||||
exp_exitcode: Optional[int] = None):
|
||||
assert len(self.stats) == count, \
|
||||
f'stats count: expected {count}, got {len(self.stats)}'
|
||||
if exp_status is not None:
|
||||
for idx, x in enumerate(self.stats):
|
||||
assert 'http_code' in x, \
|
||||
f'status #{idx} reports no http_code'
|
||||
assert x['http_code'] == exp_status, \
|
||||
f'status #{idx} unexpected http_code: {x["http_code"]}'
|
||||
if exp_exitcode is not None:
|
||||
for idx, x in enumerate(self.stats):
|
||||
if 'exitcode' in x:
|
||||
assert x['exitcode'] == 0, f'status #{idx} exitcode: {x["exitcode"]}'
|
||||
|
||||
|
||||
class CurlClient:
|
||||
|
||||
ALPN_ARG = {
|
||||
'http/0.9': '--http0.9',
|
||||
'http/1.0': '--http1.0',
|
||||
'http/1.1': '--http1.1',
|
||||
'h2': '--http2',
|
||||
'h2c': '--http2',
|
||||
'h3': '--http3-only',
|
||||
}
|
||||
|
||||
def __init__(self, env: Env, run_dir: Optional[str] = None):
|
||||
self.env = env
|
||||
self._curl = os.environ['CURL'] if 'CURL' in os.environ else env.curl
|
||||
self._run_dir = run_dir if run_dir else os.path.join(env.gen_dir, 'curl')
|
||||
self._stdoutfile = f'{self._run_dir}/curl.stdout'
|
||||
self._stderrfile = f'{self._run_dir}/curl.stderr'
|
||||
self._headerfile = f'{self._run_dir}/curl.headers'
|
||||
self._tracefile = f'{self._run_dir}/curl.trace'
|
||||
self._log_path = f'{self._run_dir}/curl.log'
|
||||
self._rmrf(self._run_dir)
|
||||
self._mkpath(self._run_dir)
|
||||
|
||||
@property
|
||||
def run_dir(self) -> str:
|
||||
return self._run_dir
|
||||
|
||||
def download_file(self, i: int) -> str:
|
||||
return os.path.join(self.run_dir, f'download_{i}.data')
|
||||
|
||||
def _rmf(self, path):
|
||||
if os.path.exists(path):
|
||||
return os.remove(path)
|
||||
|
||||
def _rmrf(self, path):
|
||||
if os.path.exists(path):
|
||||
return shutil.rmtree(path)
|
||||
|
||||
def _mkpath(self, path):
|
||||
if not os.path.exists(path):
|
||||
return os.makedirs(path)
|
||||
|
||||
def http_get(self, url: str, extra_args: Optional[List[str]] = None):
|
||||
return self._raw(url, options=extra_args, with_stats=False)
|
||||
|
||||
def http_download(self, urls: List[str],
|
||||
alpn_proto: Optional[str] = None,
|
||||
with_stats: bool = True,
|
||||
with_headers: bool = False,
|
||||
extra_args: List[str] = None):
|
||||
if extra_args is None:
|
||||
extra_args = []
|
||||
extra_args.extend([
|
||||
'-o', 'download_#1.data',
|
||||
])
|
||||
# remove any existing ones
|
||||
for i in range(100):
|
||||
self._rmf(self.download_file(i))
|
||||
if with_stats:
|
||||
extra_args.extend([
|
||||
'-w', '%{json}\\n'
|
||||
])
|
||||
return self._raw(urls, alpn_proto=alpn_proto, options=extra_args,
|
||||
with_stats=with_stats,
|
||||
with_headers=with_headers)
|
||||
|
||||
def http_upload(self, urls: List[str], data: str,
|
||||
alpn_proto: Optional[str] = None,
|
||||
with_stats: bool = True,
|
||||
with_headers: bool = False,
|
||||
extra_args: Optional[List[str]] = None):
|
||||
if extra_args is None:
|
||||
extra_args = []
|
||||
extra_args.extend([
|
||||
'--data-binary', data, '-o', 'download_#1.data',
|
||||
])
|
||||
if with_stats:
|
||||
extra_args.extend([
|
||||
'-w', '%{json}\\n'
|
||||
])
|
||||
return self._raw(urls, alpn_proto=alpn_proto, options=extra_args,
|
||||
with_stats=with_stats,
|
||||
with_headers=with_headers)
|
||||
|
||||
def http_put(self, urls: List[str], data=None, fdata=None,
|
||||
alpn_proto: Optional[str] = None,
|
||||
with_stats: bool = True,
|
||||
with_headers: bool = False,
|
||||
extra_args: Optional[List[str]] = None):
|
||||
if extra_args is None:
|
||||
extra_args = []
|
||||
if fdata is not None:
|
||||
extra_args.extend(['-T', fdata])
|
||||
elif data is not None:
|
||||
extra_args.extend(['-T', '-'])
|
||||
extra_args.extend([
|
||||
'-o', 'download_#1.data',
|
||||
])
|
||||
if with_stats:
|
||||
extra_args.extend([
|
||||
'-w', '%{json}\\n'
|
||||
])
|
||||
return self._raw(urls, intext=data,
|
||||
alpn_proto=alpn_proto, options=extra_args,
|
||||
with_stats=with_stats,
|
||||
with_headers=with_headers)
|
||||
|
||||
def response_file(self, idx: int):
|
||||
return os.path.join(self._run_dir, f'download_{idx}.data')
|
||||
|
||||
def run_direct(self, args, with_stats: bool = False):
|
||||
my_args = [self._curl]
|
||||
if with_stats:
|
||||
my_args.extend([
|
||||
'-w', '%{json}\\n'
|
||||
])
|
||||
my_args.extend([
|
||||
'-o', 'download.data',
|
||||
])
|
||||
my_args.extend(args)
|
||||
return self._run(args=my_args, with_stats=with_stats)
|
||||
|
||||
def _run(self, args, intext='', with_stats: bool = False):
|
||||
self._rmf(self._stdoutfile)
|
||||
self._rmf(self._stderrfile)
|
||||
self._rmf(self._headerfile)
|
||||
self._rmf(self._tracefile)
|
||||
start = datetime.now()
|
||||
with open(self._stdoutfile, 'w') as cout:
|
||||
with open(self._stderrfile, 'w') as cerr:
|
||||
p = subprocess.run(args, stderr=cerr, stdout=cout,
|
||||
cwd=self._run_dir, shell=False,
|
||||
input=intext.encode() if intext else None)
|
||||
coutput = open(self._stdoutfile).readlines()
|
||||
cerrput = open(self._stderrfile).readlines()
|
||||
return ExecResult(args=args, exit_code=p.returncode,
|
||||
stdout=coutput, stderr=cerrput,
|
||||
duration=datetime.now() - start,
|
||||
with_stats=with_stats)
|
||||
|
||||
def _raw(self, urls, intext='', timeout=10, options=None, insecure=False,
|
||||
alpn_proto: Optional[str] = None,
|
||||
force_resolve=True,
|
||||
with_stats=False,
|
||||
with_headers=True):
|
||||
args = self._complete_args(
|
||||
urls=urls, timeout=timeout, options=options, insecure=insecure,
|
||||
alpn_proto=alpn_proto, force_resolve=force_resolve,
|
||||
with_headers=with_headers)
|
||||
r = self._run(args, intext=intext, with_stats=with_stats)
|
||||
if r.exit_code == 0 and with_headers:
|
||||
self._parse_headerfile(self._headerfile, r=r)
|
||||
if r.json:
|
||||
r.response["json"] = r.json
|
||||
return r
|
||||
|
||||
def _complete_args(self, urls, timeout=None, options=None,
|
||||
insecure=False, force_resolve=True,
|
||||
alpn_proto: Optional[str] = None,
|
||||
with_headers: bool = True):
|
||||
if not isinstance(urls, list):
|
||||
urls = [urls]
|
||||
|
||||
args = [self._curl, "-s", "--path-as-is"]
|
||||
if with_headers:
|
||||
args.extend(["-D", self._headerfile])
|
||||
if self.env.verbose > 1:
|
||||
args.extend(['--trace', self._tracefile])
|
||||
if self.env.verbose > 2:
|
||||
args.extend(['--trace', self._tracefile, '--trace-time'])
|
||||
|
||||
for url in urls:
|
||||
u = urlparse(urls[0])
|
||||
if alpn_proto is not None:
|
||||
if alpn_proto not in self.ALPN_ARG:
|
||||
raise Exception(f'unknown ALPN protocol: "{alpn_proto}"')
|
||||
args.append(self.ALPN_ARG[alpn_proto])
|
||||
|
||||
if u.scheme == 'http':
|
||||
pass
|
||||
elif insecure:
|
||||
args.append('--insecure')
|
||||
elif options and "--cacert" in options:
|
||||
pass
|
||||
elif u.hostname:
|
||||
args.extend(["--cacert", self.env.ca.cert_file])
|
||||
|
||||
if force_resolve and u.hostname and u.hostname != 'localhost' \
|
||||
and not re.match(r'^(\d+|\[|:).*', u.hostname):
|
||||
port = u.port if u.port else 443
|
||||
args.extend(["--resolve", f"{u.hostname}:{port}:127.0.0.1"])
|
||||
if timeout is not None and int(timeout) > 0:
|
||||
args.extend(["--connect-timeout", str(int(timeout))])
|
||||
if options:
|
||||
args.extend(options)
|
||||
args.append(url)
|
||||
return args
|
||||
|
||||
def _parse_headerfile(self, headerfile: str, r: ExecResult = None) -> ExecResult:
|
||||
lines = open(headerfile).readlines()
|
||||
if r is None:
|
||||
r = ExecResult(args=[], exit_code=0, stdout=[], stderr=[])
|
||||
|
||||
response = None
|
||||
|
||||
def fin_response(resp):
|
||||
if resp:
|
||||
r.add_response(resp)
|
||||
|
||||
expected = ['status']
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if re.match(r'^$', line):
|
||||
if 'trailer' in expected:
|
||||
# end of trailers
|
||||
fin_response(response)
|
||||
response = None
|
||||
expected = ['status']
|
||||
elif 'header' in expected:
|
||||
# end of header, another status or trailers might follow
|
||||
expected = ['status', 'trailer']
|
||||
else:
|
||||
assert False, f"unexpected line: '{line}'"
|
||||
continue
|
||||
if 'status' in expected:
|
||||
# log.debug("reading 1st response line: %s", line)
|
||||
m = re.match(r'^(\S+) (\d+)( .*)?$', line)
|
||||
if m:
|
||||
fin_response(response)
|
||||
response = {
|
||||
"protocol": m.group(1),
|
||||
"status": int(m.group(2)),
|
||||
"description": m.group(3),
|
||||
"header": {},
|
||||
"trailer": {},
|
||||
"body": r.outraw
|
||||
}
|
||||
expected = ['header']
|
||||
continue
|
||||
if 'trailer' in expected:
|
||||
m = re.match(r'^([^:]+):\s*(.*)$', line)
|
||||
if m:
|
||||
response['trailer'][m.group(1).lower()] = m.group(2)
|
||||
continue
|
||||
if 'header' in expected:
|
||||
m = re.match(r'^([^:]+):\s*(.*)$', line)
|
||||
if m:
|
||||
response['header'][m.group(1).lower()] = m.group(2)
|
||||
continue
|
||||
assert False, f"unexpected line: '{line}, expected: {expected}'"
|
||||
|
||||
fin_response(response)
|
||||
return r
|
||||
|
||||
377
tests/http/testenv/env.py
Normal file
377
tests/http/testenv/env.py
Normal file
|
|
@ -0,0 +1,377 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from configparser import ConfigParser, ExtendedInterpolation
|
||||
from typing import Optional
|
||||
|
||||
from .certs import CertificateSpec, TestCA, Credentials
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def init_config_from(conf_path):
|
||||
if os.path.isfile(conf_path):
|
||||
config = ConfigParser(interpolation=ExtendedInterpolation())
|
||||
config.read(conf_path)
|
||||
return config
|
||||
return None
|
||||
|
||||
|
||||
TESTS_HTTPD_PATH = os.path.dirname(os.path.dirname(__file__))
|
||||
DEF_CONFIG = init_config_from(os.path.join(TESTS_HTTPD_PATH, 'config.ini'))
|
||||
|
||||
TOP_PATH = os.path.dirname(os.path.dirname(TESTS_HTTPD_PATH))
|
||||
CURL = os.path.join(TOP_PATH, 'src/curl')
|
||||
|
||||
|
||||
class EnvConfig:
|
||||
|
||||
def __init__(self):
|
||||
self.tests_dir = TESTS_HTTPD_PATH
|
||||
self.gen_dir = os.path.join(self.tests_dir, 'gen')
|
||||
self.config = DEF_CONFIG
|
||||
# check cur and its features
|
||||
self.curl = CURL
|
||||
self.curl_props = {
|
||||
'version': None,
|
||||
'os': None,
|
||||
'features': [],
|
||||
'protocols': [],
|
||||
'libs': [],
|
||||
'lib_versions': [],
|
||||
}
|
||||
self.curl_protos = []
|
||||
p = subprocess.run(args=[self.curl, '-V'],
|
||||
capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
assert False, f'{self.curl} -V failed with exit code: {p.returncode}'
|
||||
for l in p.stdout.splitlines(keepends=False):
|
||||
if l.startswith('curl '):
|
||||
m = re.match(r'^curl (?P<version>\S+) (?P<os>\S+) (?P<libs>.*)$', l)
|
||||
if m:
|
||||
self.curl_props['version'] = m.group('version')
|
||||
self.curl_props['os'] = m.group('os')
|
||||
self.curl_props['lib_versions'] = [
|
||||
lib.lower() for lib in m.group('libs').split(' ')
|
||||
]
|
||||
self.curl_props['libs'] = [
|
||||
re.sub(r'/.*', '', lib) for lib in self.curl_props['lib_versions']
|
||||
]
|
||||
if l.startswith('Features: '):
|
||||
self.curl_props['features'] = [
|
||||
feat.lower() for feat in l[10:].split(' ')
|
||||
]
|
||||
if l.startswith('Protocols: '):
|
||||
self.curl_props['protocols'] = [
|
||||
prot.lower() for prot in l[11:].split(' ')
|
||||
]
|
||||
self.nghttpx_with_h3 = re.match(r'.* nghttp3/.*', p.stdout.strip())
|
||||
log.debug(f'nghttpx -v: {p.stdout}')
|
||||
|
||||
self.http_port = self.config['test']['http_port']
|
||||
self.https_port = self.config['test']['https_port']
|
||||
self.proxy_port = self.config['test']['proxy_port']
|
||||
self.proxys_port = self.config['test']['proxys_port']
|
||||
self.h3_port = self.config['test']['h3_port']
|
||||
self.httpd = self.config['httpd']['httpd']
|
||||
self.apachectl = self.config['httpd']['apachectl']
|
||||
self.apxs = self.config['httpd']['apxs']
|
||||
if len(self.apxs) == 0:
|
||||
self.apxs = None
|
||||
self._httpd_version = None
|
||||
|
||||
self.examples_pem = {
|
||||
'key': 'xxx',
|
||||
'cert': 'xxx',
|
||||
}
|
||||
self.htdocs_dir = os.path.join(self.gen_dir, 'htdocs')
|
||||
self.tld = 'http.curl.se'
|
||||
self.domain1 = f"one.{self.tld}"
|
||||
self.domain2 = f"two.{self.tld}"
|
||||
self.proxy_domain = f"proxy.{self.tld}"
|
||||
self.cert_specs = [
|
||||
CertificateSpec(domains=[self.domain1, 'localhost'], key_type='rsa2048'),
|
||||
CertificateSpec(domains=[self.domain2], key_type='rsa2048'),
|
||||
CertificateSpec(domains=[self.proxy_domain], key_type='rsa2048'),
|
||||
CertificateSpec(name="clientsX", sub_specs=[
|
||||
CertificateSpec(name="user1", client=True),
|
||||
]),
|
||||
]
|
||||
|
||||
self.nghttpx = self.config['nghttpx']['nghttpx']
|
||||
self.nghttpx_with_h3 = False
|
||||
if len(self.nghttpx) == 0:
|
||||
self.nghttpx = 'nghttpx'
|
||||
if self.nghttpx is not None:
|
||||
p = subprocess.run(args=[self.nghttpx, '-v'],
|
||||
capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
# not a working nghttpx
|
||||
self.nghttpx = None
|
||||
else:
|
||||
self.nghttpx_with_h3 = re.match(r'.* nghttp3/.*', p.stdout.strip()) is not None
|
||||
log.debug(f'nghttpx -v: {p.stdout}')
|
||||
|
||||
self.caddy = self.config['caddy']['caddy']
|
||||
if len(self.caddy.strip()) == 0:
|
||||
self.caddy = None
|
||||
if self.caddy is not None:
|
||||
try:
|
||||
p = subprocess.run(args=[self.caddy, 'version'],
|
||||
capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
# not a working caddy
|
||||
self.caddy = None
|
||||
except:
|
||||
self.caddy = None
|
||||
self.caddy_http_port = self.config['caddy']['http_port']
|
||||
self.caddy_https_port = self.config['caddy']['https_port']
|
||||
|
||||
@property
|
||||
def httpd_version(self):
|
||||
if self._httpd_version is None and self.apxs is not None:
|
||||
p = subprocess.run(args=[self.apxs, '-q', 'HTTPD_VERSION'],
|
||||
capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
raise Exception(f'{self.apxs} failed to query HTTPD_VERSION: {p}')
|
||||
self._httpd_version = p.stdout.strip()
|
||||
return self._httpd_version
|
||||
|
||||
def _versiontuple(self, v):
|
||||
v = re.sub(r'(\d+\.\d+(\.\d+)?)(-\S+)?', r'\1', v)
|
||||
return tuple(map(int, v.split('.')))
|
||||
|
||||
def httpd_is_at_least(self, minv):
|
||||
hv = self._versiontuple(self.httpd_version)
|
||||
return hv >= self._versiontuple(minv)
|
||||
|
||||
def is_complete(self) -> bool:
|
||||
return os.path.isfile(self.httpd) and \
|
||||
os.path.isfile(self.apachectl) and \
|
||||
self.apxs is not None and \
|
||||
os.path.isfile(self.apxs)
|
||||
|
||||
def get_incomplete_reason(self) -> Optional[str]:
|
||||
if not os.path.isfile(self.httpd):
|
||||
return f'httpd ({self.httpd}) not found'
|
||||
if not os.path.isfile(self.apachectl):
|
||||
return f'apachectl ({self.apachectl}) not found'
|
||||
if self.apxs is None:
|
||||
return f"apxs (provided by apache2-dev) not found"
|
||||
if not os.path.isfile(self.apxs):
|
||||
return f"apxs ({self.apxs}) not found"
|
||||
return None
|
||||
|
||||
|
||||
class Env:
|
||||
|
||||
CONFIG = EnvConfig()
|
||||
|
||||
@staticmethod
|
||||
def setup_incomplete() -> bool:
|
||||
return not Env.CONFIG.is_complete()
|
||||
|
||||
@staticmethod
|
||||
def incomplete_reason() -> Optional[str]:
|
||||
return Env.CONFIG.get_incomplete_reason()
|
||||
|
||||
@staticmethod
|
||||
def have_h3_server() -> bool:
|
||||
return Env.CONFIG.nghttpx_with_h3
|
||||
|
||||
@staticmethod
|
||||
def have_h2_curl() -> bool:
|
||||
return 'http2' in Env.CONFIG.curl_props['features']
|
||||
|
||||
@staticmethod
|
||||
def have_h3_curl() -> bool:
|
||||
return 'http3' in Env.CONFIG.curl_props['features']
|
||||
|
||||
@staticmethod
|
||||
def curl_uses_lib(libname: str) -> bool:
|
||||
return libname.lower() in Env.CONFIG.curl_props['libs']
|
||||
|
||||
@staticmethod
|
||||
def curl_lib_version(libname: str) -> str:
|
||||
prefix = f'{libname.lower()}/'
|
||||
for lversion in Env.CONFIG.curl_props['lib_versions']:
|
||||
if lversion.startswith(prefix):
|
||||
return lversion[len(prefix):]
|
||||
return 'unknown'
|
||||
|
||||
@staticmethod
|
||||
def curl_os() -> str:
|
||||
return Env.CONFIG.curl_props['os']
|
||||
|
||||
@staticmethod
|
||||
def curl_version() -> str:
|
||||
return Env.CONFIG.curl_props['version']
|
||||
|
||||
@staticmethod
|
||||
def have_h3() -> bool:
|
||||
return Env.have_h3_curl() and Env.have_h3_server()
|
||||
|
||||
@staticmethod
|
||||
def httpd_version() -> str:
|
||||
return Env.CONFIG.httpd_version
|
||||
|
||||
@staticmethod
|
||||
def httpd_is_at_least(minv) -> bool:
|
||||
return Env.CONFIG.httpd_is_at_least(minv)
|
||||
|
||||
@staticmethod
|
||||
def has_caddy() -> bool:
|
||||
return Env.CONFIG.caddy is not None
|
||||
|
||||
def __init__(self, pytestconfig=None):
|
||||
self._verbose = pytestconfig.option.verbose \
|
||||
if pytestconfig is not None else 0
|
||||
self._ca = None
|
||||
|
||||
def issue_certs(self):
|
||||
if self._ca is None:
|
||||
ca_dir = os.path.join(self.CONFIG.gen_dir, 'ca')
|
||||
self._ca = TestCA.create_root(name=self.CONFIG.tld,
|
||||
store_dir=ca_dir,
|
||||
key_type="rsa2048")
|
||||
self._ca.issue_certs(self.CONFIG.cert_specs)
|
||||
|
||||
def setup(self):
|
||||
os.makedirs(self.gen_dir, exist_ok=True)
|
||||
os.makedirs(self.htdocs_dir, exist_ok=True)
|
||||
self.issue_certs()
|
||||
|
||||
def get_credentials(self, domain) -> Optional[Credentials]:
|
||||
creds = self.ca.get_credentials_for_name(domain)
|
||||
if len(creds) > 0:
|
||||
return creds[0]
|
||||
return None
|
||||
|
||||
@property
|
||||
def verbose(self) -> int:
|
||||
return self._verbose
|
||||
|
||||
@property
|
||||
def gen_dir(self) -> str:
|
||||
return self.CONFIG.gen_dir
|
||||
|
||||
@property
|
||||
def ca(self):
|
||||
return self._ca
|
||||
|
||||
@property
|
||||
def htdocs_dir(self) -> str:
|
||||
return self.CONFIG.htdocs_dir
|
||||
|
||||
@property
|
||||
def domain1(self) -> str:
|
||||
return self.CONFIG.domain1
|
||||
|
||||
@property
|
||||
def domain2(self) -> str:
|
||||
return self.CONFIG.domain2
|
||||
|
||||
@property
|
||||
def proxy_domain(self) -> str:
|
||||
return self.CONFIG.proxy_domain
|
||||
|
||||
@property
|
||||
def http_port(self) -> str:
|
||||
return self.CONFIG.http_port
|
||||
|
||||
@property
|
||||
def https_port(self) -> str:
|
||||
return self.CONFIG.https_port
|
||||
|
||||
@property
|
||||
def h3_port(self) -> str:
|
||||
return self.CONFIG.h3_port
|
||||
|
||||
@property
|
||||
def proxy_port(self) -> str:
|
||||
return self.CONFIG.proxy_port
|
||||
|
||||
@property
|
||||
def proxys_port(self) -> str:
|
||||
return self.CONFIG.proxys_port
|
||||
|
||||
@property
|
||||
def caddy(self) -> str:
|
||||
return self.CONFIG.caddy
|
||||
|
||||
@property
|
||||
def caddy_https_port(self) -> str:
|
||||
return self.CONFIG.caddy_https_port
|
||||
|
||||
@property
|
||||
def caddy_http_port(self) -> str:
|
||||
return self.CONFIG.caddy_http_port
|
||||
|
||||
@property
|
||||
def curl(self) -> str:
|
||||
return self.CONFIG.curl
|
||||
|
||||
@property
|
||||
def httpd(self) -> str:
|
||||
return self.CONFIG.httpd
|
||||
|
||||
@property
|
||||
def apachectl(self) -> str:
|
||||
return self.CONFIG.apachectl
|
||||
|
||||
@property
|
||||
def apxs(self) -> str:
|
||||
return self.CONFIG.apxs
|
||||
|
||||
@property
|
||||
def nghttpx(self) -> Optional[str]:
|
||||
return self.CONFIG.nghttpx
|
||||
|
||||
def authority_for(self, domain: str, alpn_proto: Optional[str] = None):
|
||||
if alpn_proto is None or \
|
||||
alpn_proto in ['h2', 'http/1.1', 'http/1.0', 'http/0.9']:
|
||||
return f'{domain}:{self.https_port}'
|
||||
if alpn_proto in ['h3']:
|
||||
return f'{domain}:{self.h3_port}'
|
||||
return f'{domain}:{self.http_port}'
|
||||
|
||||
def make_data_file(self, indir: str, fname: str, fsize: int) -> str:
|
||||
fpath = os.path.join(indir, fname)
|
||||
s10 = "0123456789"
|
||||
s = (101 * s10) + s10[0:3]
|
||||
with open(fpath, 'w') as fd:
|
||||
for i in range(int(fsize / 1024)):
|
||||
fd.write(f"{i:09d}-{s}\n")
|
||||
remain = int(fsize % 1024)
|
||||
if remain != 0:
|
||||
i = int(fsize / 1024) + 1
|
||||
s = f"{i:09d}-{s}\n"
|
||||
fd.write(s[0:remain])
|
||||
return fpath
|
||||
349
tests/http/testenv/httpd.py
Normal file
349
tests/http/testenv/httpd.py
Normal file
|
|
@ -0,0 +1,349 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import inspect
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from datetime import timedelta, datetime
|
||||
from json import JSONEncoder
|
||||
import time
|
||||
from typing import List, Union, Optional
|
||||
|
||||
from .curl import CurlClient, ExecResult
|
||||
from .env import Env
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Httpd:
|
||||
|
||||
MODULES = [
|
||||
'log_config', 'logio', 'unixd', 'version', 'watchdog',
|
||||
'authn_core', 'authz_user', 'authz_core', 'authz_host',
|
||||
'env', 'filter', 'headers', 'mime',
|
||||
'rewrite', 'http2', 'ssl', 'proxy', 'proxy_http', 'proxy_connect',
|
||||
'mpm_event',
|
||||
]
|
||||
COMMON_MODULES_DIRS = [
|
||||
'/usr/lib/apache2/modules', # debian
|
||||
'/usr/libexec/apache2/', # macos
|
||||
]
|
||||
|
||||
MOD_CURLTEST = None
|
||||
|
||||
def __init__(self, env: Env):
|
||||
self.env = env
|
||||
self._cmd = env.apachectl
|
||||
self._apache_dir = os.path.join(env.gen_dir, 'apache')
|
||||
self._run_dir = os.path.join(self._apache_dir, 'run')
|
||||
self._lock_dir = os.path.join(self._apache_dir, 'locks')
|
||||
self._docs_dir = os.path.join(self._apache_dir, 'docs')
|
||||
self._conf_dir = os.path.join(self._apache_dir, 'conf')
|
||||
self._conf_file = os.path.join(self._conf_dir, 'test.conf')
|
||||
self._logs_dir = os.path.join(self._apache_dir, 'logs')
|
||||
self._error_log = os.path.join(self._logs_dir, 'error_log')
|
||||
self._tmp_dir = os.path.join(self._apache_dir, 'tmp')
|
||||
self._mods_dir = None
|
||||
self._extra_configs = {}
|
||||
assert env.apxs
|
||||
p = subprocess.run(args=[env.apxs, '-q', 'libexecdir'],
|
||||
capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
raise Exception(f'{env.apxs} failed to query libexecdir: {p}')
|
||||
self._mods_dir = p.stdout.strip()
|
||||
if self._mods_dir is None:
|
||||
raise Exception(f'apache modules dir cannot be found')
|
||||
if not os.path.exists(self._mods_dir):
|
||||
raise Exception(f'apache modules dir does not exist: {self._mods_dir}')
|
||||
self._process = None
|
||||
self._rmf(self._error_log)
|
||||
self._init_curltest()
|
||||
|
||||
@property
|
||||
def docs_dir(self):
|
||||
return self._docs_dir
|
||||
|
||||
def clear_logs(self):
|
||||
self._rmf(self._error_log)
|
||||
|
||||
def exists(self):
|
||||
return os.path.exists(self._cmd)
|
||||
|
||||
def set_extra_config(self, domain: str, lines: Optional[Union[str, List[str]]]):
|
||||
if lines is None:
|
||||
self._extra_configs.pop(domain, None)
|
||||
else:
|
||||
self._extra_configs[domain] = lines
|
||||
|
||||
def _run(self, args, intext=''):
|
||||
env = {}
|
||||
for key, val in os.environ.items():
|
||||
env[key] = val
|
||||
env['APACHE_RUN_DIR'] = self._run_dir
|
||||
env['APACHE_RUN_USER'] = os.environ['USER']
|
||||
env['APACHE_LOCK_DIR'] = self._lock_dir
|
||||
env['APACHE_CONFDIR'] = self._apache_dir
|
||||
p = subprocess.run(args, stderr=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
cwd=self.env.gen_dir,
|
||||
input=intext.encode() if intext else None,
|
||||
env=env)
|
||||
start = datetime.now()
|
||||
return ExecResult(args=args, exit_code=p.returncode,
|
||||
stdout=p.stdout.decode().splitlines(),
|
||||
stderr=p.stderr.decode().splitlines(),
|
||||
duration=datetime.now() - start)
|
||||
|
||||
def _apachectl(self, cmd: str):
|
||||
args = [self.env.apachectl,
|
||||
"-d", self._apache_dir,
|
||||
"-f", self._conf_file,
|
||||
"-k", cmd]
|
||||
return self._run(args=args)
|
||||
|
||||
def start(self):
|
||||
if self._process:
|
||||
self.stop()
|
||||
self._write_config()
|
||||
with open(self._error_log, 'a') as fd:
|
||||
fd.write('start of server\n')
|
||||
with open(os.path.join(self._apache_dir, 'xxx'), 'a') as fd:
|
||||
fd.write('start of server\n')
|
||||
r = self._apachectl('start')
|
||||
if r.exit_code != 0:
|
||||
log.error(f'failed to start httpd: {r}')
|
||||
return False
|
||||
return self.wait_live(timeout=timedelta(seconds=5))
|
||||
|
||||
def stop(self):
|
||||
r = self._apachectl('stop')
|
||||
if r.exit_code == 0:
|
||||
return self.wait_dead(timeout=timedelta(seconds=5))
|
||||
return r.exit_code == 0
|
||||
|
||||
def restart(self):
|
||||
self.stop()
|
||||
return self.start()
|
||||
|
||||
def reload(self):
|
||||
self._write_config()
|
||||
r = self._apachectl("graceful")
|
||||
if r.exit_code != 0:
|
||||
log.error(f'failed to reload httpd: {r}')
|
||||
return self.wait_live(timeout=timedelta(seconds=5))
|
||||
|
||||
def wait_dead(self, timeout: timedelta):
|
||||
curl = CurlClient(env=self.env, run_dir=self._tmp_dir)
|
||||
try_until = datetime.now() + timeout
|
||||
while datetime.now() < try_until:
|
||||
r = curl.http_get(url=f'http://{self.env.domain1}:{self.env.http_port}/')
|
||||
if r.exit_code != 0:
|
||||
return True
|
||||
time.sleep(.1)
|
||||
log.debug(f"Server still responding after {timeout}")
|
||||
return False
|
||||
|
||||
def wait_live(self, timeout: timedelta):
|
||||
curl = CurlClient(env=self.env, run_dir=self._tmp_dir)
|
||||
try_until = datetime.now() + timeout
|
||||
while datetime.now() < try_until:
|
||||
r = curl.http_get(url=f'http://{self.env.domain1}:{self.env.http_port}/')
|
||||
if r.exit_code == 0:
|
||||
return True
|
||||
time.sleep(.1)
|
||||
log.debug(f"Server still not responding after {timeout}")
|
||||
return False
|
||||
|
||||
def _rmf(self, path):
|
||||
if os.path.exists(path):
|
||||
return os.remove(path)
|
||||
|
||||
def _mkpath(self, path):
|
||||
if not os.path.exists(path):
|
||||
return os.makedirs(path)
|
||||
|
||||
def _write_config(self):
|
||||
domain1 = self.env.domain1
|
||||
creds1 = self.env.get_credentials(domain1)
|
||||
domain2 = self.env.domain2
|
||||
creds2 = self.env.get_credentials(domain2)
|
||||
proxy_domain = self.env.proxy_domain
|
||||
proxy_creds = self.env.get_credentials(proxy_domain)
|
||||
self._mkpath(self._conf_dir)
|
||||
self._mkpath(self._logs_dir)
|
||||
self._mkpath(self._tmp_dir)
|
||||
self._mkpath(os.path.join(self._docs_dir, 'two'))
|
||||
with open(os.path.join(self._docs_dir, 'data.json'), 'w') as fd:
|
||||
data = {
|
||||
'server': f'{domain1}',
|
||||
}
|
||||
fd.write(JSONEncoder().encode(data))
|
||||
with open(os.path.join(self._docs_dir, 'two/data.json'), 'w') as fd:
|
||||
data = {
|
||||
'server': f'{domain2}',
|
||||
}
|
||||
fd.write(JSONEncoder().encode(data))
|
||||
with open(self._conf_file, 'w') as fd:
|
||||
for m in self.MODULES:
|
||||
if os.path.exists(os.path.join(self._mods_dir, f'mod_{m}.so')):
|
||||
fd.write(f'LoadModule {m}_module "{self._mods_dir}/mod_{m}.so"\n')
|
||||
if Httpd.MOD_CURLTEST is not None:
|
||||
fd.write(f'LoadModule curltest_module \"{Httpd.MOD_CURLTEST}\"\n')
|
||||
conf = [ # base server config
|
||||
f'ServerRoot "{self._apache_dir}"',
|
||||
f'DefaultRuntimeDir logs',
|
||||
f'PidFile httpd.pid',
|
||||
f'ErrorLog {self._error_log}',
|
||||
f'LogLevel {self._get_log_level()}',
|
||||
f'LogLevel http:trace4',
|
||||
f'LogLevel proxy:trace4',
|
||||
f'LogLevel proxy_http:trace4',
|
||||
f'H2MinWorkers 16',
|
||||
f'H2MaxWorkers 128',
|
||||
f'H2Direct on',
|
||||
f'Listen {self.env.http_port}',
|
||||
f'Listen {self.env.https_port}',
|
||||
f'Listen {self.env.proxy_port}',
|
||||
f'Listen {self.env.proxys_port}',
|
||||
f'TypesConfig "{self._conf_dir}/mime.types',
|
||||
]
|
||||
conf.extend([ # plain http host for domain1
|
||||
f'<VirtualHost *:{self.env.http_port}>',
|
||||
f' ServerName {domain1}',
|
||||
f' ServerAlias localhost',
|
||||
f' DocumentRoot "{self._docs_dir}"',
|
||||
f' Protocols h2c http/1.1',
|
||||
])
|
||||
conf.extend(self._curltest_conf())
|
||||
conf.extend([
|
||||
f'</VirtualHost>',
|
||||
f'',
|
||||
])
|
||||
conf.extend([ # https host for domain1, h1 + h2
|
||||
f'<VirtualHost *:{self.env.https_port}>',
|
||||
f' ServerName {domain1}',
|
||||
f' Protocols h2 http/1.1',
|
||||
f' SSLEngine on',
|
||||
f' SSLCertificateFile {creds1.cert_file}',
|
||||
f' SSLCertificateKeyFile {creds1.pkey_file}',
|
||||
f' DocumentRoot "{self._docs_dir}"',
|
||||
])
|
||||
conf.extend(self._curltest_conf())
|
||||
if domain1 in self._extra_configs:
|
||||
conf.extend(self._extra_configs[domain1])
|
||||
conf.extend([
|
||||
f'</VirtualHost>',
|
||||
f'',
|
||||
])
|
||||
conf.extend([ # https host for domain2, no h2
|
||||
f'<VirtualHost *:{self.env.https_port}>',
|
||||
f' ServerName {domain2}',
|
||||
f' Protocols http/1.1',
|
||||
f' SSLEngine on',
|
||||
f' SSLCertificateFile {creds2.cert_file}',
|
||||
f' SSLCertificateKeyFile {creds2.pkey_file}',
|
||||
f' DocumentRoot "{self._docs_dir}/two"',
|
||||
])
|
||||
conf.extend(self._curltest_conf())
|
||||
if domain2 in self._extra_configs:
|
||||
conf.extend(self._extra_configs[domain2])
|
||||
conf.extend([
|
||||
f'</VirtualHost>',
|
||||
f'',
|
||||
])
|
||||
conf.extend([ # http forward proxy
|
||||
f'<VirtualHost *:{self.env.proxy_port}>',
|
||||
f' ServerName {proxy_domain}',
|
||||
f' Protocols h2c, http/1.1',
|
||||
f' ProxyRequests On',
|
||||
f' ProxyVia On',
|
||||
f' AllowCONNECT {self.env.http_port} {self.env.https_port}',
|
||||
f' <Proxy "*">',
|
||||
f' Require ip 127.0.0.1',
|
||||
f' </Proxy>',
|
||||
f'</VirtualHost>',
|
||||
])
|
||||
conf.extend([ # https forward proxy
|
||||
f'<VirtualHost *:{self.env.proxys_port}>',
|
||||
f' ServerName {proxy_domain}',
|
||||
f' Protocols h2, http/1.1',
|
||||
f' SSLEngine on',
|
||||
f' SSLCertificateFile {proxy_creds.cert_file}',
|
||||
f' SSLCertificateKeyFile {proxy_creds.pkey_file}',
|
||||
f' ProxyRequests On',
|
||||
f' ProxyVia On',
|
||||
f' AllowCONNECT {self.env.http_port} {self.env.https_port}',
|
||||
f' <Proxy "*">',
|
||||
f' Require ip 127.0.0.1',
|
||||
f' </Proxy>',
|
||||
f'</VirtualHost>',
|
||||
])
|
||||
fd.write("\n".join(conf))
|
||||
with open(os.path.join(self._conf_dir, 'mime.types'), 'w') as fd:
|
||||
fd.write("\n".join([
|
||||
'text/html html',
|
||||
'application/json json',
|
||||
''
|
||||
]))
|
||||
|
||||
def _get_log_level(self):
|
||||
#if self.env.verbose > 3:
|
||||
# return 'trace2'
|
||||
#if self.env.verbose > 2:
|
||||
# return 'trace1'
|
||||
#if self.env.verbose > 1:
|
||||
# return 'debug'
|
||||
return 'info'
|
||||
|
||||
def _curltest_conf(self) -> List[str]:
|
||||
if Httpd.MOD_CURLTEST is not None:
|
||||
return [
|
||||
f' <Location /curltest/echo>',
|
||||
f' SetHandler curltest-echo',
|
||||
f' </Location>',
|
||||
f' <Location /curltest/put>',
|
||||
f' SetHandler curltest-put',
|
||||
f' </Location>',
|
||||
f' <Location /curltest/tweak>',
|
||||
f' SetHandler curltest-tweak',
|
||||
f' </Location>',
|
||||
]
|
||||
return []
|
||||
|
||||
def _init_curltest(self):
|
||||
if Httpd.MOD_CURLTEST is not None:
|
||||
return
|
||||
local_dir = os.path.dirname(inspect.getfile(Httpd))
|
||||
p = subprocess.run([self.env.apxs, '-c', 'mod_curltest.c'],
|
||||
capture_output=True,
|
||||
cwd=os.path.join(local_dir, 'mod_curltest'))
|
||||
rv = p.returncode
|
||||
if rv != 0:
|
||||
log.error(f"compiling mod_curltest failed: {p.stderr}")
|
||||
raise Exception(f"compiling mod_curltest failed: {p.stderr}")
|
||||
Httpd.MOD_CURLTEST = os.path.join(
|
||||
local_dir, 'mod_curltest/.libs/mod_curltest.so')
|
||||
5
tests/http/testenv/mod_curltest/.gitignore
vendored
Normal file
5
tests/http/testenv/mod_curltest/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
# Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
|
||||
*.slo
|
||||
514
tests/http/testenv/mod_curltest/mod_curltest.c
Normal file
514
tests/http/testenv/mod_curltest/mod_curltest.c
Normal file
|
|
@ -0,0 +1,514 @@
|
|||
/***************************************************************************
|
||||
* _ _ ____ _
|
||||
* Project ___| | | | _ \| |
|
||||
* / __| | | | |_) | |
|
||||
* | (__| |_| | _ <| |___
|
||||
* \___|\___/|_| \_\_____|
|
||||
*
|
||||
* Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
*
|
||||
* This software is licensed as described in the file COPYING, which
|
||||
* you should have received as part of this distribution. The terms
|
||||
* are also available at https://curl.se/docs/copyright.html.
|
||||
*
|
||||
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
* copies of the Software, and permit persons to whom the Software is
|
||||
* furnished to do so, under the terms of the COPYING file.
|
||||
*
|
||||
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
* KIND, either express or implied.
|
||||
*
|
||||
* SPDX-License-Identifier: curl
|
||||
*
|
||||
***************************************************************************/
|
||||
#include <apr_optional.h>
|
||||
#include <apr_optional_hooks.h>
|
||||
#include <apr_strings.h>
|
||||
#include <apr_cstr.h>
|
||||
#include <apr_time.h>
|
||||
#include <apr_want.h>
|
||||
|
||||
#include <httpd.h>
|
||||
#include <http_protocol.h>
|
||||
#include <http_request.h>
|
||||
#include <http_log.h>
|
||||
|
||||
static void curltest_hooks(apr_pool_t *pool);
|
||||
static int curltest_echo_handler(request_rec *r);
|
||||
static int curltest_put_handler(request_rec *r);
|
||||
static int curltest_tweak_handler(request_rec *r);
|
||||
|
||||
AP_DECLARE_MODULE(curltest) = {
|
||||
STANDARD20_MODULE_STUFF,
|
||||
NULL, /* func to create per dir config */
|
||||
NULL, /* func to merge per dir config */
|
||||
NULL, /* func to create per server config */
|
||||
NULL, /* func to merge per server config */
|
||||
NULL, /* command handlers */
|
||||
curltest_hooks,
|
||||
#if defined(AP_MODULE_FLAG_NONE)
|
||||
AP_MODULE_FLAG_ALWAYS_MERGE
|
||||
#endif
|
||||
};
|
||||
|
||||
static int curltest_post_config(apr_pool_t *p, apr_pool_t *plog,
|
||||
apr_pool_t *ptemp, server_rec *s)
|
||||
{
|
||||
void *data = NULL;
|
||||
const char *key = "mod_curltest_init_counter";
|
||||
|
||||
(void)plog;(void)ptemp;
|
||||
|
||||
apr_pool_userdata_get(&data, key, s->process->pool);
|
||||
if(!data) {
|
||||
/* dry run */
|
||||
apr_pool_userdata_set((const void *)1, key,
|
||||
apr_pool_cleanup_null, s->process->pool);
|
||||
return APR_SUCCESS;
|
||||
}
|
||||
|
||||
/* mess with the overall server here */
|
||||
|
||||
return APR_SUCCESS;
|
||||
}
|
||||
|
||||
static void curltest_hooks(apr_pool_t *pool)
|
||||
{
|
||||
ap_log_perror(APLOG_MARK, APLOG_TRACE1, 0, pool, "installing hooks");
|
||||
|
||||
/* Run once after configuration is set, but before mpm children initialize.
|
||||
*/
|
||||
ap_hook_post_config(curltest_post_config, NULL, NULL, APR_HOOK_MIDDLE);
|
||||
|
||||
/* curl test handlers */
|
||||
ap_hook_handler(curltest_echo_handler, NULL, NULL, APR_HOOK_MIDDLE);
|
||||
ap_hook_handler(curltest_put_handler, NULL, NULL, APR_HOOK_MIDDLE);
|
||||
ap_hook_handler(curltest_tweak_handler, NULL, NULL, APR_HOOK_MIDDLE);
|
||||
}
|
||||
|
||||
#define SECS_PER_HOUR (60*60)
|
||||
#define SECS_PER_DAY (24*SECS_PER_HOUR)
|
||||
|
||||
static apr_status_t duration_parse(apr_interval_time_t *ptimeout, const char *value,
|
||||
const char *def_unit)
|
||||
{
|
||||
char *endp;
|
||||
apr_int64_t n;
|
||||
|
||||
n = apr_strtoi64(value, &endp, 10);
|
||||
if(errno) {
|
||||
return errno;
|
||||
}
|
||||
if(!endp || !*endp) {
|
||||
if (!def_unit) def_unit = "s";
|
||||
}
|
||||
else if(endp == value) {
|
||||
return APR_EINVAL;
|
||||
}
|
||||
else {
|
||||
def_unit = endp;
|
||||
}
|
||||
|
||||
switch(*def_unit) {
|
||||
case 'D':
|
||||
case 'd':
|
||||
*ptimeout = apr_time_from_sec(n * SECS_PER_DAY);
|
||||
break;
|
||||
case 's':
|
||||
case 'S':
|
||||
*ptimeout = (apr_interval_time_t) apr_time_from_sec(n);
|
||||
break;
|
||||
case 'h':
|
||||
case 'H':
|
||||
/* Time is in hours */
|
||||
*ptimeout = (apr_interval_time_t) apr_time_from_sec(n * SECS_PER_HOUR);
|
||||
break;
|
||||
case 'm':
|
||||
case 'M':
|
||||
switch(*(++def_unit)) {
|
||||
/* Time is in milliseconds */
|
||||
case 's':
|
||||
case 'S':
|
||||
*ptimeout = (apr_interval_time_t) n * 1000;
|
||||
break;
|
||||
/* Time is in minutes */
|
||||
case 'i':
|
||||
case 'I':
|
||||
*ptimeout = (apr_interval_time_t) apr_time_from_sec(n * 60);
|
||||
break;
|
||||
default:
|
||||
return APR_EGENERAL;
|
||||
}
|
||||
break;
|
||||
case 'u':
|
||||
case 'U':
|
||||
switch(*(++def_unit)) {
|
||||
/* Time is in microseconds */
|
||||
case 's':
|
||||
case 'S':
|
||||
*ptimeout = (apr_interval_time_t) n;
|
||||
break;
|
||||
default:
|
||||
return APR_EGENERAL;
|
||||
}
|
||||
break;
|
||||
default:
|
||||
return APR_EGENERAL;
|
||||
}
|
||||
return APR_SUCCESS;
|
||||
}
|
||||
|
||||
static int status_from_str(const char *s, apr_status_t *pstatus)
|
||||
{
|
||||
if(!strcmp("timeout", s)) {
|
||||
*pstatus = APR_TIMEUP;
|
||||
return 1;
|
||||
}
|
||||
else if(!strcmp("reset", s)) {
|
||||
*pstatus = APR_ECONNRESET;
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int curltest_echo_handler(request_rec *r)
|
||||
{
|
||||
conn_rec *c = r->connection;
|
||||
apr_bucket_brigade *bb;
|
||||
apr_bucket *b;
|
||||
apr_status_t rv;
|
||||
char buffer[8192];
|
||||
const char *ct;
|
||||
long l;
|
||||
|
||||
if(strcmp(r->handler, "curltest-echo")) {
|
||||
return DECLINED;
|
||||
}
|
||||
if(r->method_number != M_GET && r->method_number != M_POST) {
|
||||
return DECLINED;
|
||||
}
|
||||
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r, "echo_handler: processing");
|
||||
r->status = 200;
|
||||
r->clength = -1;
|
||||
r->chunked = 1;
|
||||
apr_table_unset(r->headers_out, "Content-Length");
|
||||
/* Discourage content-encodings */
|
||||
apr_table_unset(r->headers_out, "Content-Encoding");
|
||||
apr_table_setn(r->subprocess_env, "no-brotli", "1");
|
||||
apr_table_setn(r->subprocess_env, "no-gzip", "1");
|
||||
|
||||
ct = apr_table_get(r->headers_in, "content-type");
|
||||
ap_set_content_type(r, ct? ct : "application/octet-stream");
|
||||
|
||||
bb = apr_brigade_create(r->pool, c->bucket_alloc);
|
||||
/* copy any request body into the response */
|
||||
if((rv = ap_setup_client_block(r, REQUEST_CHUNKED_DECHUNK))) goto cleanup;
|
||||
if(ap_should_client_block(r)) {
|
||||
while(0 < (l = ap_get_client_block(r, &buffer[0], sizeof(buffer)))) {
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r,
|
||||
"echo_handler: copying %ld bytes from request body", l);
|
||||
rv = apr_brigade_write(bb, NULL, NULL, buffer, l);
|
||||
if (APR_SUCCESS != rv) goto cleanup;
|
||||
rv = ap_pass_brigade(r->output_filters, bb);
|
||||
if (APR_SUCCESS != rv) goto cleanup;
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r,
|
||||
"echo_handler: passed %ld bytes from request body", l);
|
||||
}
|
||||
}
|
||||
/* we are done */
|
||||
b = apr_bucket_eos_create(c->bucket_alloc);
|
||||
APR_BRIGADE_INSERT_TAIL(bb, b);
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r, "echo_handler: request read");
|
||||
|
||||
if(r->trailers_in && !apr_is_empty_table(r->trailers_in)) {
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE2, 0, r,
|
||||
"echo_handler: seeing incoming trailers");
|
||||
apr_table_setn(r->trailers_out, "h2test-trailers-in",
|
||||
apr_itoa(r->pool, 1));
|
||||
}
|
||||
|
||||
rv = ap_pass_brigade(r->output_filters, bb);
|
||||
|
||||
cleanup:
|
||||
if(rv == APR_SUCCESS ||
|
||||
r->status != HTTP_OK ||
|
||||
c->aborted) {
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r, "echo_handler: done");
|
||||
return OK;
|
||||
}
|
||||
else {
|
||||
/* no way to know what type of error occurred */
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r, "echo_handler failed");
|
||||
return AP_FILTER_ERROR;
|
||||
}
|
||||
return DECLINED;
|
||||
}
|
||||
|
||||
static int curltest_tweak_handler(request_rec *r)
|
||||
{
|
||||
conn_rec *c = r->connection;
|
||||
apr_bucket_brigade *bb;
|
||||
apr_bucket *b;
|
||||
apr_status_t rv;
|
||||
char buffer[16*1024];
|
||||
int i, chunks = 3, error_bucket = 1;
|
||||
size_t chunk_size = sizeof(buffer);
|
||||
const char *request_id = "none";
|
||||
apr_time_t delay = 0, chunk_delay = 0;
|
||||
apr_array_header_t *args = NULL;
|
||||
int http_status = 200;
|
||||
apr_status_t error = APR_SUCCESS, body_error = APR_SUCCESS;
|
||||
|
||||
if(strcmp(r->handler, "curltest-tweak")) {
|
||||
return DECLINED;
|
||||
}
|
||||
if(r->method_number != M_GET && r->method_number != M_POST) {
|
||||
return DECLINED;
|
||||
}
|
||||
|
||||
if(r->args) {
|
||||
args = apr_cstr_split(r->args, "&", 1, r->pool);
|
||||
for(i = 0; i < args->nelts; ++i) {
|
||||
char *s, *val, *arg = APR_ARRAY_IDX(args, i, char*);
|
||||
s = strchr(arg, '=');
|
||||
if(s) {
|
||||
*s = '\0';
|
||||
val = s + 1;
|
||||
if(!strcmp("status", arg)) {
|
||||
http_status = (int)apr_atoi64(val);
|
||||
if(http_status > 0) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("chunks", arg)) {
|
||||
chunks = (int)apr_atoi64(val);
|
||||
if(chunks >= 0) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("chunk_size", arg)) {
|
||||
chunk_size = (int)apr_atoi64(val);
|
||||
if(chunk_size >= 0) {
|
||||
if(chunk_size > sizeof(buffer)) {
|
||||
ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
|
||||
"chunk_size %zu too large", chunk_size);
|
||||
ap_die(HTTP_BAD_REQUEST, r);
|
||||
return OK;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("id", arg)) {
|
||||
/* just an id for repeated requests with curl's url globbing */
|
||||
request_id = val;
|
||||
continue;
|
||||
}
|
||||
else if(!strcmp("error", arg)) {
|
||||
if(status_from_str(val, &error)) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("error_bucket", arg)) {
|
||||
error_bucket = (int)apr_atoi64(val);
|
||||
if(error_bucket >= 0) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("body_error", arg)) {
|
||||
if(status_from_str(val, &body_error)) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("delay", arg)) {
|
||||
rv = duration_parse(&delay, val, "s");
|
||||
if(APR_SUCCESS == rv) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
else if(!strcmp("chunk_delay", arg)) {
|
||||
rv = duration_parse(&chunk_delay, val, "s");
|
||||
if(APR_SUCCESS == rv) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, "query parameter not "
|
||||
"understood: '%s' in %s",
|
||||
arg, r->args);
|
||||
ap_die(HTTP_BAD_REQUEST, r);
|
||||
return OK;
|
||||
}
|
||||
}
|
||||
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r, "error_handler: processing "
|
||||
"request, %s", r->args? r->args : "(no args)");
|
||||
r->status = http_status;
|
||||
r->clength = -1;
|
||||
r->chunked = 1;
|
||||
apr_table_setn(r->headers_out, "request-id", request_id);
|
||||
apr_table_unset(r->headers_out, "Content-Length");
|
||||
/* Discourage content-encodings */
|
||||
apr_table_unset(r->headers_out, "Content-Encoding");
|
||||
apr_table_setn(r->subprocess_env, "no-brotli", "1");
|
||||
apr_table_setn(r->subprocess_env, "no-gzip", "1");
|
||||
|
||||
ap_set_content_type(r, "application/octet-stream");
|
||||
bb = apr_brigade_create(r->pool, c->bucket_alloc);
|
||||
|
||||
if(delay) {
|
||||
apr_sleep(delay);
|
||||
}
|
||||
if(error != APR_SUCCESS) {
|
||||
return ap_map_http_request_error(error, HTTP_BAD_REQUEST);
|
||||
}
|
||||
/* flush response */
|
||||
b = apr_bucket_flush_create(c->bucket_alloc);
|
||||
APR_BRIGADE_INSERT_TAIL(bb, b);
|
||||
rv = ap_pass_brigade(r->output_filters, bb);
|
||||
if (APR_SUCCESS != rv) goto cleanup;
|
||||
|
||||
memset(buffer, 'X', sizeof(buffer));
|
||||
for(i = 0; i < chunks; ++i) {
|
||||
if(chunk_delay) {
|
||||
apr_sleep(chunk_delay);
|
||||
}
|
||||
rv = apr_brigade_write(bb, NULL, NULL, buffer, chunk_size);
|
||||
if(APR_SUCCESS != rv) goto cleanup;
|
||||
rv = ap_pass_brigade(r->output_filters, bb);
|
||||
if(APR_SUCCESS != rv) goto cleanup;
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r,
|
||||
"error_handler: passed %lu bytes as response body",
|
||||
(unsigned long)chunk_size);
|
||||
if(body_error != APR_SUCCESS) {
|
||||
rv = body_error;
|
||||
goto cleanup;
|
||||
}
|
||||
}
|
||||
/* we are done */
|
||||
b = apr_bucket_eos_create(c->bucket_alloc);
|
||||
APR_BRIGADE_INSERT_TAIL(bb, b);
|
||||
rv = ap_pass_brigade(r->output_filters, bb);
|
||||
apr_brigade_cleanup(bb);
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r,
|
||||
"error_handler: response passed");
|
||||
|
||||
cleanup:
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r,
|
||||
"error_handler: request cleanup, r->status=%d, aborted=%d",
|
||||
r->status, c->aborted);
|
||||
if(rv == APR_SUCCESS) {
|
||||
return OK;
|
||||
}
|
||||
if(error_bucket && 0) {
|
||||
http_status = ap_map_http_request_error(rv, HTTP_BAD_REQUEST);
|
||||
b = ap_bucket_error_create(http_status, NULL, r->pool, c->bucket_alloc);
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r,
|
||||
"error_handler: passing error bucket, status=%d",
|
||||
http_status);
|
||||
APR_BRIGADE_INSERT_TAIL(bb, b);
|
||||
ap_pass_brigade(r->output_filters, bb);
|
||||
}
|
||||
return AP_FILTER_ERROR;
|
||||
}
|
||||
|
||||
static int curltest_put_handler(request_rec *r)
|
||||
{
|
||||
conn_rec *c = r->connection;
|
||||
apr_bucket_brigade *bb;
|
||||
apr_bucket *b;
|
||||
apr_status_t rv;
|
||||
char buffer[16*1024];
|
||||
const char *ct;
|
||||
apr_off_t rbody_len = 0;
|
||||
const char *request_id = "none";
|
||||
apr_time_t chunk_delay = 0;
|
||||
apr_array_header_t *args = NULL;
|
||||
long l;
|
||||
int i;
|
||||
|
||||
if(strcmp(r->handler, "curltest-put")) {
|
||||
return DECLINED;
|
||||
}
|
||||
if(r->method_number != M_PUT) {
|
||||
return DECLINED;
|
||||
}
|
||||
|
||||
if(r->args) {
|
||||
args = apr_cstr_split(r->args, "&", 1, r->pool);
|
||||
for(i = 0; i < args->nelts; ++i) {
|
||||
char *s, *val, *arg = APR_ARRAY_IDX(args, i, char*);
|
||||
s = strchr(arg, '=');
|
||||
if(s) {
|
||||
*s = '\0';
|
||||
val = s + 1;
|
||||
if(!strcmp("id", arg)) {
|
||||
/* just an id for repeated requests with curl's url globbing */
|
||||
request_id = val;
|
||||
continue;
|
||||
}
|
||||
else if(!strcmp("chunk_delay", arg)) {
|
||||
rv = duration_parse(&chunk_delay, val, "s");
|
||||
if(APR_SUCCESS == rv) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, "query parameter not "
|
||||
"understood: '%s' in %s",
|
||||
arg, r->args);
|
||||
ap_die(HTTP_BAD_REQUEST, r);
|
||||
return OK;
|
||||
}
|
||||
}
|
||||
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r, "put_handler: processing");
|
||||
r->status = 200;
|
||||
r->clength = -1;
|
||||
r->chunked = 1;
|
||||
apr_table_unset(r->headers_out, "Content-Length");
|
||||
/* Discourage content-encodings */
|
||||
apr_table_unset(r->headers_out, "Content-Encoding");
|
||||
apr_table_setn(r->subprocess_env, "no-brotli", "1");
|
||||
apr_table_setn(r->subprocess_env, "no-gzip", "1");
|
||||
|
||||
ct = apr_table_get(r->headers_in, "content-type");
|
||||
ap_set_content_type(r, ct? ct : "text/plain");
|
||||
|
||||
bb = apr_brigade_create(r->pool, c->bucket_alloc);
|
||||
/* copy any request body into the response */
|
||||
if((rv = ap_setup_client_block(r, REQUEST_CHUNKED_DECHUNK))) goto cleanup;
|
||||
if(ap_should_client_block(r)) {
|
||||
while(0 < (l = ap_get_client_block(r, &buffer[0], sizeof(buffer)))) {
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r,
|
||||
"put_handler: read %ld bytes from request body", l);
|
||||
if(chunk_delay) {
|
||||
apr_sleep(chunk_delay);
|
||||
}
|
||||
rbody_len += l;
|
||||
}
|
||||
}
|
||||
/* we are done */
|
||||
rv = apr_brigade_printf(bb, NULL, NULL, "%"APR_OFF_T_FMT, rbody_len);
|
||||
if(APR_SUCCESS != rv) goto cleanup;
|
||||
b = apr_bucket_eos_create(c->bucket_alloc);
|
||||
APR_BRIGADE_INSERT_TAIL(bb, b);
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, 0, r, "put_handler: request read");
|
||||
|
||||
rv = ap_pass_brigade(r->output_filters, bb);
|
||||
|
||||
cleanup:
|
||||
if(rv == APR_SUCCESS
|
||||
|| r->status != HTTP_OK
|
||||
|| c->aborted) {
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r, "put_handler: done");
|
||||
return OK;
|
||||
}
|
||||
else {
|
||||
/* no way to know what type of error occurred */
|
||||
ap_log_rerror(APLOG_MARK, APLOG_TRACE1, rv, r, "put_handler failed");
|
||||
return AP_FILTER_ERROR;
|
||||
}
|
||||
return DECLINED;
|
||||
}
|
||||
|
||||
186
tests/http/testenv/nghttpx.py
Normal file
186
tests/http/testenv/nghttpx.py
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
#***************************************************************************
|
||||
# _ _ ____ _
|
||||
# Project ___| | | | _ \| |
|
||||
# / __| | | | |_) | |
|
||||
# | (__| |_| | _ <| |___
|
||||
# \___|\___/|_| \_\_____|
|
||||
#
|
||||
# Copyright (C) 2008 - 2022, Daniel Stenberg, <daniel@haxx.se>, et al.
|
||||
#
|
||||
# This software is licensed as described in the file COPYING, which
|
||||
# you should have received as part of this distribution. The terms
|
||||
# are also available at https://curl.se/docs/copyright.html.
|
||||
#
|
||||
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
||||
# copies of the Software, and permit persons to whom the Software is
|
||||
# furnished to do so, under the terms of the COPYING file.
|
||||
#
|
||||
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
||||
# KIND, either express or implied.
|
||||
#
|
||||
# SPDX-License-Identifier: curl
|
||||
#
|
||||
###########################################################################
|
||||
#
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import time
|
||||
from typing import Optional
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from .env import Env
|
||||
from .curl import CurlClient
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Nghttpx:
|
||||
|
||||
def __init__(self, env: Env):
|
||||
self.env = env
|
||||
self._cmd = env.nghttpx
|
||||
self._run_dir = os.path.join(env.gen_dir, 'nghttpx')
|
||||
self._pid_file = os.path.join(self._run_dir, 'nghttpx.pid')
|
||||
self._conf_file = os.path.join(self._run_dir, 'nghttpx.conf')
|
||||
self._error_log = os.path.join(self._run_dir, 'nghttpx.log')
|
||||
self._stderr = os.path.join(self._run_dir, 'nghttpx.stderr')
|
||||
self._tmp_dir = os.path.join(self._run_dir, 'tmp')
|
||||
self._process = None
|
||||
self._process: Optional[subprocess.Popen] = None
|
||||
self._rmf(self._pid_file)
|
||||
self._rmf(self._error_log)
|
||||
self._mkpath(self._run_dir)
|
||||
self._write_config()
|
||||
|
||||
def exists(self):
|
||||
return os.path.exists(self._cmd)
|
||||
|
||||
def clear_logs(self):
|
||||
self._rmf(self._error_log)
|
||||
self._rmf(self._stderr)
|
||||
|
||||
def is_running(self):
|
||||
if self._process:
|
||||
self._process.poll()
|
||||
return self._process.returncode is None
|
||||
return False
|
||||
|
||||
def start_if_needed(self):
|
||||
if not self.is_running():
|
||||
return self.start()
|
||||
return True
|
||||
|
||||
def start(self, wait_live=True):
|
||||
self._mkpath(self._tmp_dir)
|
||||
if self._process:
|
||||
self.stop()
|
||||
args = [
|
||||
self._cmd,
|
||||
f'--frontend=*,{self.env.h3_port};quic',
|
||||
f'--backend=127.0.0.1,{self.env.https_port};{self.env.domain1};sni={self.env.domain1};proto=h2;tls',
|
||||
f'--backend=127.0.0.1,{self.env.http_port}',
|
||||
f'--log-level=INFO',
|
||||
f'--pid-file={self._pid_file}',
|
||||
f'--errorlog-file={self._error_log}',
|
||||
f'--conf={self._conf_file}',
|
||||
f'--cacert={self.env.ca.cert_file}',
|
||||
self.env.get_credentials(self.env.domain1).pkey_file,
|
||||
self.env.get_credentials(self.env.domain1).cert_file,
|
||||
]
|
||||
ngerr = open(self._stderr, 'a')
|
||||
self._process = subprocess.Popen(args=args, stderr=ngerr)
|
||||
if self._process.returncode is not None:
|
||||
return False
|
||||
return not wait_live or self.wait_live(timeout=timedelta(seconds=5))
|
||||
|
||||
def stop_if_running(self):
|
||||
if self.is_running():
|
||||
return self.stop()
|
||||
return True
|
||||
|
||||
def stop(self, wait_dead=True):
|
||||
self._mkpath(self._tmp_dir)
|
||||
if self._process:
|
||||
self._process.terminate()
|
||||
self._process.wait(timeout=2)
|
||||
self._process = None
|
||||
return not wait_dead or self.wait_dead(timeout=timedelta(seconds=5))
|
||||
return True
|
||||
|
||||
def restart(self):
|
||||
self.stop()
|
||||
return self.start()
|
||||
|
||||
def reload(self, timeout: timedelta):
|
||||
if self._process:
|
||||
running = self._process
|
||||
self._process = None
|
||||
os.kill(running.pid, signal.SIGQUIT)
|
||||
end_wait = datetime.now() + timeout
|
||||
if not self.start(wait_live=False):
|
||||
self._process = running
|
||||
return False
|
||||
while datetime.now() < end_wait:
|
||||
try:
|
||||
log.debug(f'waiting for nghttpx({running.pid}) to exit.')
|
||||
running.wait(2)
|
||||
log.debug(f'nghttpx({running.pid}) terminated -> {running.returncode}')
|
||||
break
|
||||
except subprocess.TimeoutExpired:
|
||||
log.warning(f'nghttpx({running.pid}), not shut down yet.')
|
||||
os.kill(running.pid, signal.SIGQUIT)
|
||||
if datetime.now() >= end_wait:
|
||||
log.error(f'nghttpx({running.pid}), terminate forcefully.')
|
||||
os.kill(running.pid, signal.SIGKILL)
|
||||
running.terminate()
|
||||
running.wait(1)
|
||||
return self.wait_live(timeout=timedelta(seconds=5))
|
||||
return False
|
||||
|
||||
def wait_dead(self, timeout: timedelta):
|
||||
curl = CurlClient(env=self.env, run_dir=self._tmp_dir)
|
||||
try_until = datetime.now() + timeout
|
||||
while datetime.now() < try_until:
|
||||
check_url = f'https://{self.env.domain1}:{self.env.h3_port}/'
|
||||
r = curl.http_get(url=check_url, extra_args=['--http3-only'])
|
||||
if r.exit_code != 0:
|
||||
return True
|
||||
log.debug(f'waiting for nghttpx to stop responding: {r}')
|
||||
time.sleep(.1)
|
||||
log.debug(f"Server still responding after {timeout}")
|
||||
return False
|
||||
|
||||
def wait_live(self, timeout: timedelta):
|
||||
curl = CurlClient(env=self.env, run_dir=self._tmp_dir)
|
||||
try_until = datetime.now() + timeout
|
||||
while datetime.now() < try_until:
|
||||
check_url = f'https://{self.env.domain1}:{self.env.h3_port}/'
|
||||
r = curl.http_get(url=check_url, extra_args=[
|
||||
'--http3-only', '--trace', 'curl.trace', '--trace-time'
|
||||
])
|
||||
if r.exit_code == 0:
|
||||
return True
|
||||
log.debug(f'waiting for nghttpx to become responsive: {r}')
|
||||
time.sleep(.1)
|
||||
log.error(f"Server still not responding after {timeout}")
|
||||
return False
|
||||
|
||||
def _rmf(self, path):
|
||||
if os.path.exists(path):
|
||||
return os.remove(path)
|
||||
|
||||
def _mkpath(self, path):
|
||||
if not os.path.exists(path):
|
||||
return os.makedirs(path)
|
||||
|
||||
def _write_config(self):
|
||||
with open(self._conf_file, 'w') as fd:
|
||||
fd.write(f'# nghttpx test config'),
|
||||
fd.write("\n".join([
|
||||
'# do we need something here?'
|
||||
]))
|
||||
Loading…
Add table
Add a link
Reference in a new issue