JuliaArrays: VectorOfArrays

[pfackeldey]

Dear Jim,

Awkward Arrays in Julia is a very interesting development which I am happy to see!

I just wanted to point you to a similar Julia project called ArraysOfArrays.jl. There is an implementation of a vector of arrays with different sizes, which might be interesting for you: VectorOfArrays.

I am very sorry for the noise in case you are already aware of this project.

Best, Peter

[Moelf]

Hi Peter, I made sure Jim is aware of the package, in some ways you can replace ListOffsetArray with it, but one goal is to have compatible memory layout with python awkward, which means this package wants offset to start 0-index, of course you hack a OffsetArray into VectorOfVectors, but I won't say that's super clean.

Another thing is that Awkward array composes in multiple ways, and having total control over all types is necessary, which means even if this package tries to re-use ArrayOfArrays.jl, it probably need to wrap it again, that adds a bit messiness too.

[pfackeldey]

Hi @Moelf,

thank you for putting both projects into context, makes sense to me!
I will keep an eye on the development of this project, I'm curious to see where things will go :)

Best, Peter

[jpivarski]

Since ArraysOfArrays and VectorOfArrays are known standards in the Julia world, it would make sense to have some conversions to and from it, for the subset of AwkwardArray that fits their data models.

I still need to make a list of desiderata for this project, but the main purpose is to form a zero-copy bridge with Python Awkward Arrays, for use with PyJulia and PyCall.jl. On the Julia side of the bridge, we should connect to Julia-native ways of doing things, of course. (Python Awkward Array is already a kind of spider...)

One of the things that will be on the feature list will be Julia-side conversion between AwkwardArray and arrow-julia. If UnROOT.jl emits either AwkwardArray or Arrow, then we'd have a replacement for Uproot that can navigate ROOT metadata more quickly, solving this problem: HSF/PyHEP.dev-workshops#15.

That's where this is going: providing reasons to duck out of Python into Julia for short trips so that a mostly-Python data analyst is motivated to learn about Julia in ways that don't involve throwing out their existing infrastructure (as described here). They may eventually decide to become mostly-Julia data analysts, but I'll leave that up to them. The thing is, I don't think many people would even try Julia if they have to adopt it wholesale, right away.

(That was Java's attitude: to promote Java, they reimplemented the entire world in Java. Now it's like a walled garden that's hard to break out of: JNI is not very reliable. Julia does not have this attitude, particularly since it's interested in bare-metal access, rather than a virtual machine, and PyJulia/PyCall.jl demonstrate that openness. I'm sure it's easy to pass rectilinear arrays between Julia and Python, but something more is needed for the irregular arrays. We could always use Arrow at the interface, but there are gaps that an AwkwardArray implementation would fill.)