Passing SSLContext for the HTTPS+X509 to uproot.dask fails because the context is not serializable #1248
Labels
bug (unverified)
The problem described would be a bug, but needs to be triaged
Comments
bockjoo
added
the
bug (unverified)
nsmith-
changed the title
|
I apologize I pasted a wrong script. works. This is the script that reproduces the issue: |
|
It looks like CA certs that are set up by : disappear when I checked like so: After updating the main script readNanoEventsMyProcessor.py and hacking lib/python3.12/asyncio/base_events.py, or I am not sure how this information can help resolve the issue, though. |
|
After decreasing the step size from 100,000 to 10,000 in the preprocess, the script (See below) is stable and there is no TypeError. |
This might be the same as #1233 |
|
Here is a self-contained pickleable ssl context class: import os
import ssl
import tempfile
from typing import Any
class PickleableSSLContext(ssl.SSLContext):
@classmethod
def create(cls, protocol=ssl.PROTOCOL_TLS_CLIENT):
out = cls(protocol)
out._set_default_state()
return out
def _set_default_state(self):
# this should do the same setup as ssl.create_default_context()
# for now it just loads default certificates
if self.verify_mode != ssl.CERT_NONE:
self.load_default_certs()
def load_cert_chain(self, certfile, keyfile=None, password=None):
with open(certfile, "rb") as fin:
self.certdata = fin.read()
self.keydata = None
if keyfile is not None:
with open(keyfile, "rb") as fin:
self.keydata = fin.read()
self.password = password
self._load_cert_chain()
def _load_cert_chain(self):
with tempfile.TemporaryDirectory() as dirname:
certpath = os.path.join(dirname, "cert.pem")
with open(certpath, "wb") as fout:
fout.write(self.certdata)
keypath = None
if self.keydata is not None:
keypath = os.path.join(dirname, "key.pem")
with open(keypath, "wb") as fout:
fout.write(self.keydata)
super().load_cert_chain(certpath, keypath, self.password)
def __getnewargs__(self):
return (self.protocol,)
def __getstate__(self) -> dict[str, Any]:
return {
"certdata": self.certdata,
"keydata": self.keydata,
"password": self.password,
}
def __setstate__(self, state: dict[str, Any]) -> None:
self.__dict__.update(state)
self._set_default_state()
self._load_cert_chain()It's use is as follows: import os
import pickle
import uproot
from pickleablessl import PickleableSSLContext
sslctx = PickleableSSLContext.create()
sslctx.load_cert_chain(os.environ['X509_USER_PROXY'])
sslctx = pickle.loads(pickle.dumps(sslctx))
url = "https://xrootd-local.unl.edu:1094//store/mc/RunIISummer20UL18NanoAODv9/TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8/NANOAODSIM/106X_upgrade2018_realistic_v16_L1v1-v1/130000/3094251D-BAB4-6446-86F1-2F9C4D8F339D.root"
with uproot.open(url, ssl=sslctx) as file:
print(file["Events"].num_entries)Now the question is, where to put this class? |
|
The picklable ssl context reaches up to get_steps in preprocessor.py, If I mannualy add kw['ssl'] and kwargs['ssl'] , there is no issue. |
|
My apology. It wasn't working inside http.py (called by apply_to_fileset ) by my mistake. |
|
It worked with the SLURMCluster as well. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am using Coffea 2024.6.1 and uproot 5.3.10
To reproduce the issue, this script can be used:
The environmental variable X509_USER_PROXY should point to the voms-proxy file.
The input file is
but may be irrelevant as SSL Connection can not be established as the SSLContext is not serializable.
The stracetrace of the above script looks like:
Thanks for looking into this issue!
The text was updated successfully, but these errors were encountered: