Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Erase features in 3D from 3D mask #35

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

brisvag
Copy link
Contributor

@brisvag brisvag commented Sep 6, 2024

First attempt at using fidder to do feature erasure on a volume instead of a stack of 2D images.

To avoid unnecessary refactoring, for now we just made a secondary code path for 3D which is essentially a copy-paste of the 2D one with minor changes.

estimate_local_mean is now replaced by estimate_local_mean_3d, which uses a badly implemented torch_cubic_spline_grids.b_spline_grids.CubicBSplineGrid3d (@alisterburt help pls :P) in order to interpolate in 3D instead of 2D.

Then, we also added an estimate_background_std_3d which is trying to be a bit smarter about noise (since Z noise is quite different from XY noise in tomos).

Any thoughts/suggestions on this approach?

For now, we don't see an obvious improvement on the erased subvolumes (at least visually), but we're testing some downstream processing to see if it affects the STA.

cc @alessiodacapito

Comment on lines 109 to 123

grid = CubicBSplineGrid3d(resolution=resolution)
optimiser = torch.optim.Adam(grid.parameters(), lr=0.01)

for i in range(500):
# what does the model predict for our observations?
prediction = grid(foreground_sample_idx).squeeze()

# zero gradients and calculate loss between observations and model prediction
optimiser.zero_grad()
loss = torch.sum((prediction - w)**2)**0.5

# backpropagate loss and update values at points on grid
loss.backward()
optimiser.step()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not convinced I'm using this right, here...

@estrozi
Copy link

estrozi commented Sep 9, 2024

Hi guys!!! If I understood well the goal, IMHO it would be simpler to adapt/change/retrain membrain-seg to have a "--2d" option so it can segment membranes in 2d (at tilt series level) and then apply fidder. Am I wrong? (of course, easier said than done).
It would also be possible to "project back" the membrain-seg results (3D masks) onto the tilt series performing the corresponding transformations backwards.
Clearly, when re-projected back in 2d the membranes have very different aspects (intensities) depending on the angle their normal make with the tilt axis but they only need to be erased where the intensities are the strongest.
So, 2d projections of the membrain-seg result according to the tilt angles, followed by a peak search on a 2D cross-correlation with the raw images simply to put the mask "back" at the right x-y position, and then fidder can do its magic. No ?
Perhaps it would be even better, instead of "erasing" the membranes using fidder, they could be subtracted by using the projected tomogram, similar to the SPA signal subtraction.
Cheers!

Copy link
Member

@alisterburt alisterburt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Minor usage error on the spline grid but otherwise looks good

I'm not convinced this should live here though - it's about enabling an application that isn't anything to do with fiducials...

How about we part out the erasing code for both 2D + 3D into a cryoeraser package? That way these could live together and fidder could depend on just the 2D bit

@estrozi nice to see you here! I agree 2D erasing of the membranes prior to tomogram reconstruction would be a better solution overall. There might be some value in erasing in 3D though 🤷🏼‍♂️


for i in range(500):
# what does the model predict for our observations?
prediction = grid(foreground_sample_idx).squeeze()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like it's about right except when sampling the model coordinates should be rescaled to [0, 1] where 0 is the center of the first element in the dimension and 1 is the center of the last element in the dimension

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhmh... I see, I'll look into it :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alisterburt does this make sense now? Though unfortunately it doens't look better @alessiodacapito :/ I think the bulk of the work here is being done by the noise generation, not the local mean...

@brisvag
Copy link
Contributor Author

brisvag commented Sep 11, 2024

How about we part out the erasing code for both 2D + 3D into a cryoeraser package? That way these could live together and fidder could depend on just the 2D bit

Yes, I was thinking the same! I think fidder now is really 2 different things in one and the eraser would be useful as standalone. I also think that in that case we should try to merge the paths between 2D and 3D a bit, all this duplication is fragile. But for now I went for separate because it was quicker and easier to review.

It would also be possible to "project back" the membrain-seg results (3D masks) onto the tilt series performing the corresponding transformations backwards.

Yes, that's exactly what I suggested as well, though it's a significantly harder work so we went for the low-hanging fruit first. One major drawback of working in 3D is that the noise pattern is way different (very much not normal distribution, I suspect...) which makes the masking pretty bad. Though, maybe the mistake I made in the grid is also to blame...

@alessiodacapito
Copy link

How about we part out the erasing code for both 2D + 3D into a cryoeraser package? That way these could live together and fidder could depend on just the 2D bit

Yes great idea !

It would also be possible to "project back" the membrain-seg results (3D masks) onto the tilt series performing the corresponding transformations backwards.

I am currently working on it, it's showing good results for now !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants