[BUG] 3007.X scheduler_before_connect throws TCP Publish Client encountered an exception, no start event is seen

[amalaguti]

Description
3007.1 Windows minions with option scheduler_before_connect: True set throws exception TCP Publish Client encountered an exception while connecting to 127.0.0.1:4510: TimeoutError() on start, and does not generate /start event on master event bus

[TRACE   ] IPCServer: Handling connection to address: ('127.0.0.1', 52006)
[DEBUG   ] The functions from module 'mine' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded mine.update
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'id': 'minion-win-1', 'fun': 'mine.update', 'fun_args': [], 'schedule': '__mine_interval', 'jid': '20240927221656953118', 'pid': 420}
[TRACE   ] ==== evaluating schedule now None =====
[TRACE   ] Client disconnected from IPC 127.0.0.1:4511
[WARNING ] TCP Publish Client encountered an exception while connecting to 127.0.0.1:4510: TimeoutError(), will reconnect in 1 seconds -   File "C:\Program Files\Salt Project\Salt\Lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,

  File "C:\Program Files\Salt Project\Salt\Lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)

  File "C:\Program Files\Salt Project\Salt\salt-minion.exe\__main__.py", line 7, in <module>
    sys.exit(salt_minion())

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\scripts.py", line 185, in salt_minion
    minion.start()

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\cli\daemons.py", line 344, in start
    self._real_start()

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\cli\daemons.py", line 356, in _real_start
    self.minion.tune_in()

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1193, in tune_in
    self._bind()

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1064, in _bind
    self.event = salt.utils.event.get_event(

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\event.py", line 135, in get_event
    return SaltEvent(

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\event.py", line 265, in __init__
    self.connect_pub()

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\event.py", line 348, in connect_pub
    self.subscriber = salt.transport.ipc_publish_client(

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\transport\base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\transport\base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\transport\tcp.py", line 220, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\transport\base.py", line 398, in __init__
    super().__init__()

[TRACE   ] TCP PubServer sending payload: topic_list=None '/salt/minion/minion_schedule_delete_complete\n\n��completeèschedule��__mine_interval��enabledèfunction�mine.update�jid_includeêmaxrunning\x02�minutes<�name�__mine_interval�return_job£runìrun_on_startåsplay��_stamp�2024-09-27T22:16:56.937494'
[TRACE   ] TCP PubServer finished publishing payload
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem

No /start event is shown, only auth

salt/auth	{
    "_stamp": "2024-09-27T22:22:51.680278",
    "act": "accept",
    "id": "minion-win-1",
    "pub": "-----BEGIN PUBLIC KEY-----REDACTED-----END PUBLIC KEY-----",
    "result": true
}
minion/refresh/minion-win-1	{
    "Minion data cache refresh": "minion-win-1",
    "_stamp": "2024-09-27T22:22:51.961685"
}

The minion seems responsive after start, but not having the /start event seems to be a big issue.

The scheduler_before_connect option is required in this use case and works fine in 3006.x versions.

Setup
3007.x Windos minion with scheduler_before_connect: True

Expected behavior
Minion should start without the exception, the scheduler_before_connect should work as expected and /start event should be seeing on the event bus

[amalaguti]

I discovered a little difference in behavior, if the minion is configured in multimaster (active/active), when the minion is started as service, net start salt-minion, there is no /start event.

But when the minion is started in foreground, salt-minion -l debug, I can randomly get a /start event in one or the other master, but not in both.

With the scheduler_before_connect option removed, both masters get the /start event