[BUG] pip.installed slow for already installed packages

[bdrx312]

Description
The pip.installed state is slow when run for already installed package. It takes 400-700ms per state on various hosts I have tried. This is a problem when re-running a salt top file to apply a single change when there are a lot of pip packages that are managed. Re-running the state.apply should take a few seconds yet can take much longer depending on the number of python packages.

Setup

Please be as specific as possible and give set-up details.

• VM running on a cloud service (AWS ec2 instance rhel 8.10)
• container (ubuntu 22 docker)
• onedir packaging
• masterless

Steps to Reproduce the behavior

• Setup directories and files; set minion to masterless; use state file that installs already installed python package:

  cat > /etc/salt/minion <<'EOF'
  file_client: local
  master_type: disable
  EOF
  
  mkdir /srv/salt
  cat > /srv/salt/top.sls <<'EOF'
  base:
    '*':
      - test
  EOF
  
  cat > /srv/salt/test.sls <<'EOF'
  {% for i in range(100) %}
  "pip package {{i}}":
    pip.installed:
      - name: requests
  {% endfor %}
  EOF

• Run state.apply which takes over a minute instead of a few seconds

  time salt-call --local state.apply

# time salt-call --local state.apply
local:
----------
          ID: pip package 0
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:14.503081
    Duration: 1163.051 ms
     Changes:
----------
          ID: pip package 1
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:15.666359
    Duration: 582.998 ms
     Changes:
----------
          ID: pip package 2
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:16.249571
    Duration: 488.112 ms
     Changes:
----------
          ID: pip package 3
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:16.737891
    Duration: 528.643 ms
     Changes:
----------
          ID: pip package 4
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:17.266742
    Duration: 692.309 ms
     Changes:
----------
          ID: pip package 5
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:17.959389
    Duration: 550.589 ms
     Changes:
----------
          ID: pip package 6
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:18.510188
    Duration: 473.987 ms
     Changes:
----------
          ID: pip package 7
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:18.984420
    Duration: 517.698 ms
     Changes:
----------
          ID: pip package 8
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:19.502335
    Duration: 602.803 ms
     Changes:
----------
          ID: pip package 9
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:20.105362
    Duration: 632.442 ms
     Changes:
----------
          ID: pip package 10
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:07:20.738068
    Duration: 489.732 ms
     Changes:
----------
...
----------
          ID: pip package 90
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:04.533844
    Duration: 557.176 ms
     Changes:
----------
          ID: pip package 91
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:05.091245
    Duration: 519.467 ms
     Changes:
----------
          ID: pip package 92
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:05.610998
    Duration: 515.7 ms
     Changes:
----------
          ID: pip package 93
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:06.127003
    Duration: 608.118 ms
     Changes:
----------
          ID: pip package 94
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:06.735337
    Duration: 555.77 ms
     Changes:
----------
          ID: pip package 95
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:07.291331
    Duration: 508.018 ms
     Changes:
----------
          ID: pip package 96
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:07.799627
    Duration: 509.157 ms
     Changes:
----------
          ID: pip package 97
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:08.309094
    Duration: 509.362 ms
     Changes:
----------
          ID: pip package 98
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:08.818716
    Duration: 538.097 ms
     Changes:
----------
          ID: pip package 99
    Function: pip.installed
        Name: requests
      Result: True
     Comment: Python package requests was already installed
              All specified packages are already installed
     Started: 15:08:09.357041
    Duration: 582.829 ms
     Changes:

Summary for local
--------------
Succeeded: 100
Failed:      0
--------------
Total states run:     100
Total run time:    55.410 s

real    1m1.741s
user    0m52.553s
sys     0m5.382s

Expected behavior
The total time should only be a few seconds when the system is already configured.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
                  Salt: 3007.1+443.g6f12f34502

Python Version:
                Python: 3.10.11 (main, Oct  4 2024, 18:20:58) [GCC 11.4.0]

Dependency Versions:
                  cffi: 1.16.0
              cherrypy: unknown
          cryptography: 42.0.5
              dateutil: 2.8.2
             docker-py: 7.1.0
                 gitdb: 4.0.10
             gitpython: 3.1.43
                Jinja2: 3.1.4
               libgit2: 1.7.1
          looseversion: 1.3.0
              M2Crypto: Not Installed
                  Mako: 1.2.4
               msgpack: 1.0.7
          msgpack-pure: Not Installed
          mysql-python: Not Installed
              networkx: 3.1
             packaging: 23.1
             pycparser: 2.21
              pycrypto: Not Installed
          pycryptodome: 3.19.1
                pygit2: 1.13.1
          python-gnupg: 0.5.2
                PyYAML: 6.0.1
                 PyZMQ: 25.1.2
                relenv: Not Installed
                 smmap: 5.0.1
               timelib: 0.3.0
               Tornado: 6.3.3
                   ZMQ: 4.3.4

Salt Extensions:
 pytest-salt-factories: 1.0.1

Salt Package Information:
          Package Type: pip

System Versions:
                  dist: ubuntu 22.04.5 jammy
                locale: utf-8
               machine: x86_64
               release: 5.15.153.1-microsoft-standard-WSL2
                system: Linux
               version: Ubuntu 22.04.5 jammy

Additional context

Adding caching to the list of installed python packages is one option that could help a lot.