詳解pytest分佈式執行插件 pytest-xdist 的高級用法

Posted on by

想要使用多個CPU核心來進行測試，可以使用 -n 參數( 或者 –numprocesses)
(使用8個核心來跑測試用例)

pytest -n 8

使用 -n auto 參數可以利用電腦的所有核心來跑測試用例
測試時使用的算法可以根據--dist命令參數定制：

  • --dist load(默認選項)：給每個CPU核心隨機分配用例，不保證執行順序。
  • --dist loadscope：對於測試函數，測試按模塊分組，對於測試方法，測試按類分組。每組作為一個整體分配給可用的worker。這保證瞭組中的所有測試都在同一進程中運行。如果的模塊級或類級fixtures，這將非常有用。按類分組優先於按模塊分組。
  • --dist loadfile: 測試用例按其所在文件分組。每組作為一個整體分配給可用的worker。這保證瞭文件中的所有測試都在同一個輔助進程中運行。
  • --dist loadgroup: 測試按xdist_group標記分組。每組作為一個整體分配給可用的執行器。這保證瞭具有相同xdist_ group名稱的所有測試都在同一個worker中運行。
@pytest.mark.xdist_group(name="group1")
def test1():
    pass

class TestA:
    @pytest.mark.xdist_group("group1")
    def test2():
        pass-

這將確保test1和TestA:：test2將在同一個worker中運行。沒有xdist_ group標記的測試在–dist=load模式下正常運行。

  • --dist no：正常的pytest執行模式，一次運行一個測試（完全沒有分發）。

例子：
項目目錄結構

xdist_test.py

import logging
import pytest
class TestXdist(object):
    @pytest.mark.xdist_group("group1")
    def test_one(self):
        logging.info("1")
        assert True
    @pytest.mark.xdist_group("group1")
    def test_two(self):
        logging.info("2")
        assert True
    @pytest.mark.xdist_group("group2")
    def test_three(self):
        logging.info("3")
        assert True
    @pytest.mark.xdist_group("group2")
    def test_four(self):
        logging.info("4")
        assert True

xdist_dummy_test.py

import logging
import pytest
class TestXdist(object):
    @pytest.mark.run(order=1)
    @pytest.mark.xdist_group("group1")
    def test_dummy_one(self):
        logging.info("d1")
        assert True
    @pytest.mark.xdist_group("group1")
    @pytest.mark.run(order=2)
    def test_dummy_two(self):
        logging.info("d2")
        assert True
    @pytest.mark.run(order=3)
    @pytest.mark.xdist_group("group2")
    def test_dummy_three(self):
        logging.info("d3")
        assert True
    @pytest.mark.xdist_group("group2")
    @pytest.mark.run(order=4)
    def test_dummy_four(self):
        logging.info("d4")
        assert True
(venv) ➜  pytest pytest -n auto --dist loadscope
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.8.9, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/spock/PycharmProjects/pytest, configfile: pytest.ini
plugins: xdist-2.5.0, forked-1.4.0, ordering-0.6
[gw0] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw1] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw2] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw3] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw4] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw5] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw6] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw7] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
gw0 [8] / gw1 [8] / gw2 [8] / gw3 [8] / gw4 [8] / gw5 [8] / gw6 [8] / gw7 [8]
scheduling tests via LoadScopeScheduling

test2/xdist_dummy_test.py::TestXdist::test_dummy_one 
test/xdist_test.py::TestXdist::test_one 
[gw0] [ 12%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_one 
[gw1] [ 25%] PASSED test/xdist_test.py::TestXdist::test_one 
test2/xdist_dummy_test.py::TestXdist::test_dummy_two 
test/xdist_test.py::TestXdist::test_two 
[gw1] [ 37%] PASSED test/xdist_test.py::TestXdist::test_two 
test/xdist_test.py::TestXdist::test_three 
[gw0] [ 50%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_two 
test2/xdist_dummy_test.py::TestXdist::test_dummy_three 
[gw1] [ 62%] PASSED test/xdist_test.py::TestXdist::test_three 
test/xdist_test.py::TestXdist::test_four 
[gw0] [ 75%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_three 
test2/xdist_dummy_test.py::TestXdist::test_dummy_four 
[gw1] [ 87%] PASSED test/xdist_test.py::TestXdist::test_four 
[gw0] [100%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_four
============================================================================ 8 passed in 0.40s ============================================================================
(venv) ➜  pytest pytest -n auto --dist loadfile 
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.8.9, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/spock/PycharmProjects/pytest, configfile: pytest.ini
plugins: xdist-2.5.0, forked-1.4.0, ordering-0.6
[gw0] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw1] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw2] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw3] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw4] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw5] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw6] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw7] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
gw0 [8] / gw1 [8] / gw2 [8] / gw3 [8] / gw4 [8] / gw5 [8] / gw6 [8] / gw7 [8]
scheduling tests via LoadFileScheduling

test/xdist_test.py::TestXdist::test_one 
test2/xdist_dummy_test.py::TestXdist::test_dummy_one 
[gw1] [ 12%] PASSED test/xdist_test.py::TestXdist::test_one 
[gw0] [ 25%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_one 
test/xdist_test.py::TestXdist::test_two 
test2/xdist_dummy_test.py::TestXdist::test_dummy_two 
[gw1] [ 37%] PASSED test/xdist_test.py::TestXdist::test_two 
[gw0] [ 50%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_two 
test2/xdist_dummy_test.py::TestXdist::test_dummy_three 
test/xdist_test.py::TestXdist::test_three 
[gw1] [ 62%] PASSED test/xdist_test.py::TestXdist::test_three 
[gw0] [ 75%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_three 
test2/xdist_dummy_test.py::TestXdist::test_dummy_four 
test/xdist_test.py::TestXdist::test_four 
[gw1] [ 87%] PASSED test/xdist_test.py::TestXdist::test_four 
[gw0] [100%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_four 

============================================================================ 8 passed in 0.38s ============================================================================
(venv) ➜  pytest pytest -n auto --dist loadgroup
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.8.9, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/spock/PycharmProjects/pytest, configfile: pytest.ini
plugins: xdist-2.5.0, forked-1.4.0, ordering-0.6
[gw0] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw1] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw2] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw3] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw4] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw5] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw6] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw7] Python 3.8.9 (default, May 17 2022, 12:55:41)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
gw0 [8] / gw1 [8] / gw2 [8] / gw3 [8] / gw4 [8] / gw5 [8] / gw6 [8] / gw7 [8]
scheduling tests via LoadGroupScheduling

test2/xdist_dummy_test.py::TestXdist::test_dummy_three@group2 
test2/xdist_dummy_test.py::TestXdist::test_dummy_one@group1 
[gw1] [ 12%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_three@group2 
test2/xdist_dummy_test.py::TestXdist::test_dummy_four@group2 
[gw0] [ 25%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_one@group1 
[gw1] [ 37%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_four@group2 
test2/xdist_dummy_test.py::TestXdist::test_dummy_two@group1 
[gw0] [ 50%] PASSED test2/xdist_dummy_test.py::TestXdist::test_dummy_two@group1 
test/xdist_test.py::TestXdist::test_three@group2 
[gw1] [ 62%] PASSED test/xdist_test.py::TestXdist::test_three@group2 
test/xdist_test.py::TestXdist::test_four@group2 
[gw1] [ 75%] PASSED test/xdist_test.py::TestXdist::test_four@group2 
test/xdist_test.py::TestXdist::test_one@group1 
[gw0] [ 87%] PASSED test/xdist_test.py::TestXdist::test_one@group1 
test/xdist_test.py::TestXdist::test_two@group1 
[gw0] [100%] PASSED test/xdist_test.py::TestXdist::test_two@group1 

============================================================================ 8 passed in 0.40s ============================================================================

到此這篇關於pytest分佈式執行插件 pytest-xdist 的高級用法的文章就介紹到這瞭,更多相關pytest分佈式執行插件 pytest-xdist內容請搜索WalkonNet以前的文章或繼續瀏覽下面的相關文章希望大傢以後多多支持WalkonNet！

推薦閱讀：