Microscopy package

[roflmaostc]

cc @RainerHeintzmann @BioTurboNick

Hey all,

I was wondering whether it makes sense to introduce different microscopy types, objectives and sensors.
I'm thinking about composing microscopes from different parts which return resolution criteria, DoF, etc.
Additionally, we could adapt packages like PSF.jl to such a package. Hopefully, other will integrate as well (if possible).

We could built a nice show method which (based on Unitful.jl) would print the physical properties in a nice way.

Something like this (of course manually adding a NA should be possible as well).

julia> Microscope(objective=1JuliaManufacturerPlanFluorite10x, sensor=JuliaManufacturerHighPerformanceCamera, mode=Widefield)
Microscope
      NA = 0.25
      Δx (according abbe) = 1.1µm 
     ...

What do you think about that?

Does it make sense to introduce as many objectives and camera sensors as we can?
So literally having dozen of Zeiss, Nikon, Olympus, .. types? The same for camera sensors.

Is there a common database somewhere similar to glass catalogues in Optic design tools?

Cheers,
Felix

[BioTurboNick]

This could be interesting. I don't have deep knowledge of the optics though, so I would be of limited help there.

[roflmaostc]

One hurdle is to get proper information about the objectives itself.

A similar approach as in [OpticSim.jl[https://github.com/microsoft/OpticSim.jl/blob/main/deps/build.jl) would be nice. During build process the download the glass catalogue and register them into the program.
However, so far I didn't find such a document for Zeiss objectives e.g. (haven't checked other manufacturer)

[RainerHeintzmann]

The question, how to generally define a microscope with an almost complete set of its parameters or more specifically a microscope image has been an ongoing discussion of a whole consortium and led to the OME-tiff XML standard. To me it sounds like, offering an easy to use interface to that data structure, may be the way to go.

Of course a "microscope" toolbox should infer the important limits from such parameters. If you load an image via BioFormats.jl, you get access to such a filled datastructure. However, this is not super easy, as the information is something like this: "3-color image, where the first channel has been taken in widefield mode with the 63x objective, the second one with 488nm confocal illumination and the ... detection filterset, the 3rd channel was using 2-photon imaging at 790nm excitation and the 4th channel is STED with the excitation laser power ..., all images were sampled at 40nmx40nmx160nm.". Sounds like a fair bit of work to convert this into limits.

Anyway, the bottom line is, we should definitely avoid introducing yet another standard for describing microscopes.

In general it would be great, if the meta information (foremost the pixelsize) would stick firmly to the data we process. However, the Julia type system seems to make this "sticking" behaviour next to impossible to achieve. The only Package that I know, which achieves this seems to be OffsetArrays.jl, yet I do not understand, looking at the code, how they managed to achieve this.

[roflmaostc]

There is also OMETIFF.jl.

I don't want to reproduce this standard but instead I want to define a Microscope type within Julia which we can then use for dispatch etc.
Of course, if possible it would be nice to have a parser from "OME-XML -> Julia Microscope Type"

On such a type all functions could depend if they need access to information like NA, magnification.

Instead of providing then some_function(NA=0.3, magnification=10, pixel_size=1u"μm") the user would do

julia> mic = Microscope(NA=0.3, magnification=10, pixel_size=1u"μm");

julia> mic = Microscope(Objective.JuliaOpticsPlanFluor, sensor=Sensor.JuliaImagingCCDII) # or more abstract

julia> some_function(mic)

julia> another_function(mic)

[BioTurboNick]

@RainerHeintzmann - what do you mean by "stick firmly"? Do you mean that the OffsetArray carries the offset with it?

If you mean an image should carry with it how large its pixels are but that current Julia image types don't, we could make a custom image type that just has fields for pixel size and whatever else we want. Anything that processes an abstract image should pass that data through intact, and in the cases where a concrete Image type cannot be avoided, we would need to provide custom methods.

But maybe I'm totally off base with what you're thinking?

[roflmaostc]

@BioTurboNick I believe he refers to something like this

julia> using OffsetArrays, PaddedViews, Unitful, FFTW

julia> arr = randn((2,2))
2×2 Matrix{Float64}:
 0.550614    0.331567
 0.0738847  -1.04439

julia> arr_off = OffsetArray(arr, 2:3, 2:3)
2×2 OffsetArray(::Matrix{Float64}, 2:3, 2:3) with eltype Float64 with indices 2:3×2:3:
 0.550614    0.331567
 0.0738847  -1.04439

julia> arr_pad PaddedView(-1, arr, (3,3), (2,2))
3×3 PaddedView(-1.0, OffsetArray(::Matrix{Float64}, 2:3, 2:3), (Base.OneTo(3), Base.OneTo(3))) with eltype Float64:
 -1.0  -1.0        -1.0
 -1.0   0.550614    0.331567
 -1.0   0.0738847  -1.04439

julia> arr_unit = arr .* 1u"kg" 
2×2 Matrix{Quantity{Float64, 𝐌, Unitful.FreeUnits{(kg,), 𝐌, nothing}}}:
  0.550614 kg  0.331567 kg
 0.0738847 kg  -1.04439 kg

julia> typeof(arr_pad .^2) # type changed
Matrix{Float64} (alias for Array{Float64, 2})

julia> typeof(arr_off .^2) # type unchanged
OffsetMatrix{Float64, Matrix{Float64}} (alias for OffsetArray{Float64, 2, Array{Float64, 2}})

julia> typeof(arr_unit.^2) # type unchanged
Matrix{Quantity{Float64, 𝐌^2, Unitful.FreeUnits{(kg^2,), 𝐌^2, nothing}}} (alias for Array{Quantity{Float64, 𝐌^2, Unitful.FreeUnits{(kg^2,), 𝐌^2, nothing}}, 2})

julia> typeof(fft(arr_pad)) # type changed
Matrix{ComplexF64} (alias for Array{Complex{Float64}, 2})

julia> typeof(fft(arr_off)) # type changed
Matrix{ComplexF64} (alias for Array{Complex{Float64}, 2})
 
julia> typeof(fft(arr_unit)) # error -> cause eltype is _strange_
ERROR: MethodError: no method matching plan_fft(::Matrix{Quantity{Float64, 𝐌, Unitful.FreeUnits{(kg,), 𝐌, nothing}}}, ::UnitRange{Int64})
Closest candidates are:
  plan_fft(::StridedArray{T, N}, ::Any; flags, timelimit) where {T<:Union{ComplexF32, ComplexF64}, N} at /home/fxw/.julia/packages/FFTW/kKdEk/src/fft.jl:685
  plan_fft(::AbstractArray{var"#s6", N} where {var"#s6"<:Real, N}, ::Any; kws...) at /home/fxw/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:199
  plan_fft(::AbstractArray{var"#s15", N} where {var"#s15"<:(Complex{var"#s16"} where var"#s16"<:Union{Integer, Rational}), N}, ::Any; kws...) at /home/fxw/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:201
  ...
Stacktrace:
 [1] plan_fft(x::Matrix{Quantity{Float64, 𝐌, Unitful.FreeUnits{(kg,), 𝐌, nothing}}}; kws::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ AbstractFFTs ~/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:52
 [2] plan_fft(x::Matrix{Quantity{Float64, 𝐌, Unitful.FreeUnits{(kg,), 𝐌, nothing}}})
   @ AbstractFFTs ~/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:52
 [3] fft(x::Matrix{Quantity{Float64, 𝐌, Unitful.FreeUnits{(kg,), 𝐌, nothing}}})
   @ AbstractFFTs ~/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:50
 [4] top-level scope
   @ REPL[19]:1

So OffsetArray survives a broadcasting call (.^2), PaddedView doesn't.
But as soon you call something non-trivial the fancy arrays are converted to plain arrays and all information is lost.
In this case FFTW.jl probably suffers from an external library call, however the broadcast mechanisms seem to be not so easy to implement and hence many special array wrappers fail to execute that correctly.

A solution would be to define own types, but that can go quickly into messy handling whether someone provides an array or an struct with special information.

[RainerHeintzmann]

Yes, this is what I meant. Looks like Felix found a second type which works ;-) The problem is maybe that there is no finalize mechanism in Julia, which ensures that encapsulating types carry over the type signature. But then, I am clearly missing something, since I did not figure out how OffsetArrays does the trick. I tried copying some of the broadcast mechansims over to a different class but this did not really help.

[roflmaostc]

I contacted Zeiss a while ago and it seems like that there is no "objective catalogue" publicly available as it is for "glass catalogues" (Schott, Nikon, ...)

OpticsSim.jl for example automatically imports those glass catalogues to include them into the package.

However, we could probably come up with something that is able to parse a OME XML file. See OMETIFF, which exports a XML displaying the microscope information.
We could transform that to some kind of Julia struct.