Add ability to run tests in parallel [PULP-753] #1241
Conversation
gerrod3
force-pushed
the
parallel-test
branch
10 times, most recently
from
5447ea9 to
bd148c5
Compare
| if pulp_ctx.api._dry_run: | ||
| raise click.ClickException( | ||
| _("Trying to cancel {} tasks in safe-mode, aborting").format(len(tasks)) | ||
| ) |
There was a problem hiding this comment.
🤔 Is this changing the behaviour or just preventing an Exception further down in the stack to trigger?
There was a problem hiding this comment.
It's changing the behavior to be more consistent. Before if you ran this command in dry mode it would only give the abort message upon the first task cancellation, meaning it would only error if there was running/waiting tasks. Now it will always give an error (with the amount of available tasks to cancel) when ran in dry mode. Basically the test_tasks check of this command was really inconsistent when running in parallel.
gerrod3
force-pushed
the
parallel-test
branch
2 times, most recently
from
bb3bdfa to
6aa7ae5
Compare
|
I can't figure out why the query_params test is failing. The output looks sorted. I test with the same names and I get no error, albeit I'm not running other tests in the background at the same time. But even with parallel tests I can't see how another test is affecting this one. I look at the error message in the $OUTPUT list and the list is sorted. This failure doesn't happen often, but I seen it on both the reverse and normal alphabetical list. Is it somehow possible we have a bug in pulpcore, a race condition that breaks the ordering maybe having to do with paginating the query? |
Is it a good idea to test whether pulpcore sorts properly in the first place? |
gerrod3
force-pushed
the
parallel-test
branch
3 times, most recently
from
b60de6d to
a399900
Compare
|
I've run the CI 5 times and gotten green each time, I think it's ready. The tests now take 6-8 mins down from 14-18 mins. Interestingly the runner with oas version 3.1.1 seems to be the slowest each time by like a minute, not sure why. |
| non_blocking = not ( | ||
| base_path is not None | ||
| or distribution_ctx.pulp_ctx.has_plugin(PluginRequirement("core", specifier="<3.24.0")) | ||
| ) |
There was a problem hiding this comment.
This sounds like
| non_blocking = not ( | |
| base_path is not None | |
| or distribution_ctx.pulp_ctx.has_plugin(PluginRequirement("core", specifier="<3.24.0")) | |
| ) | |
| non_blocking = ( | |
| base_path is None | |
| and distribution_ctx.pulp_ctx.has_plugin(PluginRequirement("core", specifier=">=3.24.0")) | |
| ) |
What changed in that version?
There was a problem hiding this comment.
Iirc we made the serializer less restrictive on unsetting/replacing the source for the distribution. Before we would have to wait for the update to complete before dispatching the next update else it would fail.
gerrod3
force-pushed
the
parallel-test
branch
from
a399900 to
9329c20
Compare
gerrod3
force-pushed
the
parallel-test
branch
from
9329c20 to
3f092ff
Compare
| {%- endraw %} | ||
| run: | | ||
| .ci/run_container.sh make {% if cookiecutter.unittests %}live{% endif %}test | ||
| .ci/run_container.sh make {% if cookiecutter.unittests %}parallel{% endif %}test |
There was a problem hiding this comment.
So i think the logic here is a little messed up now.
What we have is tests that are marked "live" because they need a running server (this includes all the "scripts" by virtue of the pytest_plugin script lookup but also some python tests manually marked).
We can selectively run them by calling "make livetest". Or, now, run them in parallel using "make paralleltest".
"make unittest" in turn, using -m "not live" will run all other tests.
The thing is our cli plugins may not be ready to run tests in parallel. So I think we should make this a new cookiecutter parameter for the time being.
(I think the reason for having "unittests" as a parameter is that pytest fails if there aren't any.)
2 participants
Need to figure out how to handle tests that call orphan cleanup and make sure they run last and in serial. Most of this is ensuring each test uses unique names to prevent collisions.