Redwood C/D - 6504984451: Can you guys hear it?
Wilson, Tom J.: Thanks.
Redwood C/D - 6504984451: So we're in the bridge.
Redwood C/D - 6504984451: And the echo is the echoes.
Redwood C/D - 6504984451: Just… it's all good on. Oh.
Redwood C/D - 6504984451: Alright, so yeah, let's get started. So… This… this session is… Going out a really short.
Redwood C/D - 6504984451: Introductions to problems, so not, like, whole presentations, just…
Redwood C/D - 6504984451: enough to give context on some interesting or perplexing thing, and then we're gonna talk about potential solutions, ways of looking at the problem to diagnose something interesting. But there basically aren't enough
Redwood C/D - 6504984451: opportunities to discuss null results, and null results often lead to really interesting methodological developments. So, even if it couldn't be published because it's, oh, we didn't observe the thing.
Redwood C/D - 6504984451: It's too noisy in this weird way, that kind of thing. It's definitely worth discussing, and collaborating can solve…
Redwood C/D - 6504984451: There's problems that were otherwise.
Redwood C/D - 6504984451: So I will… I will model the first one to just… these can be really quick. So, the… the early leap, the early alerts, these were from…
Redwood C/D - 6504984451: February-ish, there were… there were real bogus scores, reliability scores.
Redwood C/D - 6504984451: there were more alerts with low reliability scores than were initially anticipated, and planning how the alerts would work. So, I was trying to figure out what is the cause of
Redwood C/D - 6504984451: This very large and unexpected number of
Redwood C/D - 6504984451: Middling to low reliability scores.
Redwood C/D - 6504984451: So… Seeing…
Redwood C/D - 6504984451: are they all just one-ups? That was the hypothesis those other people had, and they're not, so there… there's this huge cluster of, when there… there are lots of points in the light curve, so the same object keeps triggering alerts, but they never have high,
Redwood C/D - 6504984451: reliability scores. So… So, that has to be something like image quality or template quality.
Redwood C/D - 6504984451: like, if the template's bad, every time you look there, it's gonna look different from the template. And so…
Redwood C/D - 6504984451: Is there a way that we can…
Redwood C/D - 6504984451: deal with that there's going to be too many alerts to keep this real bogus model.
Redwood C/D - 6504984451: Should we change the algorithm to…
Redwood C/D - 6504984451: The auto-reject based on, like, the history of alerts that are left.
Redwood C/D - 6504984451: That object, that part of the sky, the templates aren't good, they'll always be like that, and so it'll just keep…
Redwood C/D - 6504984451: Triggering too many, so, that…
Redwood C/D - 6504984451: That's an example of, like, a kind of problem that,
Redwood C/D - 6504984451: Some of the discussion already happened, but this is an opportunity if anyone has thoughts on this, and then we'll go on to other members of the ZCC and the other science collaborations who applied.
Redwood C/D - 6504984451: One shot.
Redwood C/D - 6504984451: Yeah, so, let me just get the…
Redwood C/D - 6504984451: There's, Phil Marshall here. So, we've seen an effect just like this as well, trying to find lensed quasars in the alert stream.
Redwood C/D - 6504984451: And…
Redwood C/D - 6504984451: have concluded that persistence of, of appearance in the alert stream is actually a really good signal for us. It's one of the ways we can reject things that haven't been flagged as bogus, but aren't… aren't lens quasars in the alert stream. But yeah, we've…
Redwood C/D - 6504984451: concluded that persistence is important, but I think we've also seen the problem that you're flagging, which is that there are some, some real astronomical objects that are repeatedly being, appearing in the alert stream
Redwood C/D - 6504984451: with, unimpressive reliability scores, and it looks to me as though they might well be, you know, bad PSF models.
Redwood C/D - 6504984451: That kind of thing. I mean, you'd mentioned bad templates, I think it's in the same category of just problems with the… with the processing that leads to things persisting in the alert stream when we'd really rather they didn't. But I did want to flag that…
Redwood C/D - 6504984451: For the lens quasar search, we're looking for things that are… That, that aren't point-like.
Redwood C/D - 6504984451: But we nevertheless still want to find, and so we're in this grey area where, some Lanza AGN may well be flagged as bogus, even though we don't want them to be.
Redwood C/D - 6504984451: So, in our future, we may well be trying to feed back to Ruben a set of, the same set of simulated Lensed AGN that we've been training on.
Redwood C/D - 6504984451: feed it to Rubin in order to improve the real bogus classifier and not reject lens quasars. So I wanted to flag that up as well as something that we're… that we're working on.
Redwood C/D - 6504984451: Thank you. Yeah, so there was a session
Redwood C/D - 6504984451: Yesterday, that… that talked about the real bogus classifier, and…
Redwood C/D - 6504984451: and different strategies for it, and I didn't even know it was on the table to, like.
Redwood C/D - 6504984451: Can we revise it? Can we give it more training? Try again? Like, is it… yeah, revising it over time? I don't know if that's the kind of thing that has to be written into a plan of, like, yes, after this much time, we'll…
Redwood C/D - 6504984451: Redo it, and it'll be different, and then I'll…
Redwood C/D - 6504984451: You are more likely to know the answer to this process for that.
Redwood C/D - 6504984451: Are you saying the real bogus versions?
Redwood C/D - 6504984451: Like, different versions of… Yeah, the real bogus has continuous versions. Three are already available and described in the MTN377, so they typically will be attached to data releases.
Redwood C/D - 6504984451: Because the real book is not only used to give out the real book… the score to the audience, but it's also used to select sources that we then extract.
Redwood C/D - 6504984451: Because we have many more sources than we expected, at least for now, than if, you know, image quality improves, templates improve, that might no longer be the case. But I think we would realistically have a version of the real bogus that is different for every data release, at least in the first few years, and then
Redwood C/D - 6504984451: In addition, different, yeah, versions could be attached to the AP products.
Redwood C/D - 6504984451: If we have a significant performance improvement, we would probably update the version for the alert production.
Redwood C/D - 6504984451: In real time.
Redwood C/D - 6504984451: Okay, so mine was just, like, an example of this, we'll go into…
Redwood C/D - 6504984451: Take it away.
Redwood C/D - 6504984451: And you're being timed.
Redwood C/D - 6504984451: Pardon? Yeah, you got 5 minutes.
Redwood C/D - 6504984451: Hello everybody, my name is Prince. I am developing AI models for object detection, instant segmentation, and classification.
Redwood C/D - 6504984451: So this is our deep disk framework.
Redwood C/D - 6504984451: It's basically able to take in the DP1 tiles, or any kind of Khpish tiles from astronomical data, and it can give you the classification, so it can tell you if it's a star or a galaxy, it can tell you the bounding boxes, so it'll put a box around the item.
Redwood C/D - 6504984451: And that's, like, the mask head, so you could also get the mask of these things. So this is a sample, like, Scarlett-curated pseudo-ground code that I created, and a deep disk prediction, for example.
Redwood C/D - 6504984451: You guys can glance to it for a second?
Redwood C/D - 6504984451: Tell us what we're looking for.
Redwood C/D - 6504984451: So yeah, these are all the… all the blues are galaxies, and all the signs are stars here.
Redwood C/D - 6504984451: And basically, yeah, like, it's basically able to put a box around all the galaxies, for example, and make the mask around them, and it's able to say that if the box is basically blue, it will be a galaxy.
Redwood C/D - 6504984451: And this is basically something that I've published in my first paper on classification head. Our model reaches near parity with the random first classifier, and since it's basically a scene-level model, it basically looks at the image itself, it doesn't rely on the catalog as much.
Redwood C/D - 6504984451: Currently I'm working on self-supervised models, so here, since I use curated ground truth here, on the left side, this is… because DP1 doesn't have two ground truth, I use a curated ground truth. I'm trying to remove the reliance on this, so I could just directly feed in the images, and it will be able to spit out all the predictions.
Redwood C/D - 6504984451: Now, the problem, the one case that I have, like, a specific null case is that, initially, I worked on full DP1, and I produced these AI models that, would work with the full DP1, but when I… when it went to publishing, I only published on these two fields.
Redwood C/D - 6504984451: So, this is because, of the stellar density being very low here, and the galaxy density being extremely high, and also because VR1 and all the upcoming ones will be more focused towards,
Redwood C/D - 6504984451: the deeper fields, basically. So this is why, like, we focused on this, but… and there's always a trade-off, like, if the model works really well on these fields, it might not work as well on others, basically.
Redwood C/D - 6504984451: So, yeah,
Redwood C/D - 6504984451: That's, like, one of the things… I mean, eventually I have to make a model that will work on all of them, so I would appreciate any comments or any feedback.
Redwood C/D - 6504984451: One example I can give is, because the galaxy density is really high here, and the star density was low, you could use something like star rating, so you could augment the data of stars, just so it sees more and more of stars. And that way you can improve your stellar, like, classification stuff, or, you know, the stellar accuracies.
Redwood C/D - 6504984451: But yeah, like, as you can see, there is different amounts. So, like, for example, this one has near-zero galaxies, and it has a lot of stars. A model trained on this would not do as good on the low galactic one.
Redwood C/D - 6504984451: Initially, when I was doing, I was still getting, like, 50-60%, but accuracy, but yeah, so the main part is that those results are not publishable, because mostly with these AI stuff, like, you're trying to prove that they are really good, and you reach this 80-90% accuracy.
Redwood C/D - 6504984451: Yeah, so I suppose this is a, supervised learning. Sorry. Yeah, so, I suppose this is a supervised learning. So, how did you train, like.
Redwood C/D - 6504984451: So yeah, this is, currently, my first paper is on supervised learning part, and for sure, we need the ground truth annotation for it. So for that, I used scarlet-curated ground truth.
Redwood C/D - 6504984451: For a classification part, to know if it's a star or a galaxy for just a training sample, I use the ref extendedness parameter. Okay. Or, the box… for the boxes and the mask, I use Scarlet, so Scarlet is able to tell you pixel-wise, that all these pixels, for example, are belonging to me.
Redwood C/D - 6504984451: From that pixel, I can look at the maximum width of the pixel, the maximum height of the pixel, and then that would make the box around me.
Redwood C/D - 6504984451: So this is basically the curated pseudo-ground tool that I use to train, and if you look closely here, like, this BrightStar contamination has been one of a major, like, issues with DP1, where, there's the… if there's a bright star, it would kind of, like.
Redwood C/D - 6504984451: it messes up with the scarlet. For example, the Scarlett tries to de-blend, and it would put, like, a lot of boxes around here.
Redwood C/D - 6504984451: So, I came up with some techniques to clean up that, basically, and I trained it on that, and if you see my poster later on, you might have a closer look on it, but it's able to even do better than the pseudo-ground troop that I trained it on.
Redwood C/D - 6504984451: So, one counter question, so how you generate the ground truth?
Redwood C/D - 6504984451: So your model cannot surpass, like, cannot do better than how you generate the ground truth. Now, if generating the ground truth consumes more time, then it makes sense, but if generating the ground truth is, like, pretty instantaneous, then what's the selling point here?
Redwood C/D - 6504984451: Well, one… for one, like,
Redwood C/D - 6504984451: Well, okay, so for creating the ground truth part, Alam says, for creating the ground truth part, like.
Redwood C/D - 6504984451: First of all, there's no ground truth for DP1 itself.
Redwood C/D - 6504984451: So there's no, like, point that it's, like, surpassing what's… so this is just a scarlet, curated suit of ground truth. We don't have a ground truth, and we don't… we can't, like, stop the science there, right? Oh, I don't have annotations, I don't have ground truth, so let's just not work on this project, right? Like, science has to progress, and so I made this, like, curated ground truth in order to train it, and in many of the tiles that I've seen visually, and, even, for example, like.
Redwood C/D - 6504984451: If you see here,
Redwood C/D - 6504984451: Technically, I use DP1, but for example, the unrecognized blend, when I compare it with Hubble Space Telescope, our model is able to bring this 3% lower, for example. I trained on ref extending its parameter.
Redwood C/D - 6504984451: And, that's basically morphology only. If you look at the morphology-only plots for, like, this, for example, you'll see a lot of stars misclassified as galaxies, like, it dips basically here.
Redwood C/D - 6504984451: Because this is, like, the… as you go to the fainter regime, galaxies start looking more like point-like. So this drops here for morphology only, and this will go up like this. So if you look at the…
Redwood C/D - 6504984451: There's a paper by Gato et al, and they have shown that already, that the random forest is able to kind of, like, recover similar things, basically. But yeah, even though I trained it on RefX standardness, I trained it on this curated ground truth, it does actually do better than what I trained it on. Okay, makes sense, yeah.
Redwood C/D - 6504984451: Yeah, this seems to suggest a need for metrics that don't require a ground truth. Exactly, so this should suggest, on metrics that don't require ground truth, so that's my main PhD-like thesis, project. With DP2 and further, I will be using something called Deno V3.
Redwood C/D - 6504984451: Dean OV3 can work on, like, it's a self-supervised learning model, so it relies on much less amount of annotations. So even if, for example, in DP1, I had a decent amount of overlap with the Hubble Space Telescope data, I could use the Hubble Space Telescope data as the ground truth and train it on that, and then apply it. So, yeah, that's something that I'm currently working on.
Redwood C/D - 6504984451: And, yeah, it's, like, the main part. One more thing is that on this… this thing, you could also add, like, the redshift head, so you could also get redshift from our model, and you could also get, like, the key points of these galaxies, like spiral arms and all that.
Redwood C/D - 6504984451: physically.
Redwood C/D - 6504984451: Yeah, but I'm trying to remove this, because, yeah, this is the main thing, that the reliance on the pseudo-ground truth is not the best thing, and eventually, as LSSC data progresses, we will have lesser and lesser ground truth.
Redwood C/D - 6504984451: So, yeah, I'm trying to have this Deno V3 model. This is a meta-AI model, basically, and you could work it with only, like, 5 or 6% of data.
Redwood C/D - 6504984451: even if I find 5% ground truth somewhere in the overlap of full DR1 or DR2, you can use these models to, like, train admins. They'll still do really well.
Redwood C/D - 6504984451: Just a comparison point, like, self-supervised, like, when you have the labels and stuff completely correct.
Redwood C/D - 6504984451: people reach accuracies of 85-ish percent, and even in the self-supervised learning regime, you can reach a nice, like, 75-76% accuracy if you use Dinov3, like, models. So I would… we would trade off that 10% accuracy with actually training it on the nice, like, ground truth that we can get, even if it's, like, 5%.
Redwood C/D - 6504984451: Okay, thank you.
Redwood C/D - 6504984451: There's gonna be 3 presenters now? Are they… They're, like, 3 separate presentations. They are 3 separate… How you should interpret this. One after another, but they're on the same topic. Bryce, are they here?
Redwood C/D - 6504984451: So we will, I guess, advance the slides for you, or should we… are they all online? Yes, Hurum says… Hurum says… Hurum says I'm on Zoom, and… Oh, okay.
Redwood C/D - 6504984451: Try saying something, because we don't hear.
Hurum Maksora Tohfa: Hi, can you hear me?
Redwood C/D - 6504984451: Yes.
Redwood C/D - 6504984451: Thank you.
Redwood C/D - 6504984451: You should be able to share.
Redwood C/D - 6504984451: Alright.
Hurum Maksora Tohfa: Awesome.
Hurum Maksora Tohfa: Can you hear me?
Redwood C/D - 6504984451: Yep.
Hurum Maksora Tohfa: Okay, great. So…
Redwood C/D - 6504984451: We…
Hurum Maksora Tohfa: We are giving a talk about, all the AI methods that, we have tried with Ruben. I'll talk about the first effort that's, AI Donut, just before we get into everything else, to give, an overview.
Hurum Maksora Tohfa: since Rubin has such a wide field of view, small changes in temperature or tilt can cause the telescope to go out of focus just a little bit, so we have to
Hurum Maksora Tohfa: actively correct for it, throughout the night. And, it has a bendy mirror, which has a bunch of pistons behind it, and,
Hurum Maksora Tohfa: We kind of combine those things with, 50 degrees of freedom.
Hurum Maksora Tohfa: And we could use those corrections to adjust for any changes that happens.
Hurum Maksora Tohfa: So, to do this, we take out-of-focus, pictures. The reason for doing out-of-focus pictures is, if you look at the PSF perfect optics and distorted optics, it's not, super…
Hurum Maksora Tohfa: different in a… just, like, by looking at eye, you can't exactly see what actually caused the distortion. So instead, what we do is we take.
Redwood C/D - 6504984451: That's true.
Hurum Maksora Tohfa: images. That way, because of the shadow of the M2 mirror, you have these, donut-looking objects. And then, if you have, certain kind of aberrations, that becomes easier to calculate. So here's, for example, is a pentofoil. Obviously, this is very exaggerated.
Hurum Maksora Tohfa: But, we can see, like, these kind of effects.
Hurum Maksora Tohfa: So, for the objects in the field, we take, these out-of-focus images. You can see on the right, that's what they look like, and then we cut out, these stamps, and then we go from, the focal plane to people
Hurum Maksora Tohfa: plane, and what we are currently doing is, we estimate a set of Zernickies, and then we run a forward modeling and try to see if that matches what we see, and then calculate, and then use those Zernickies to correct for,
Hurum Maksora Tohfa: correct for… correct to get the telescope in focus again. So what we have worked on is we trained, CNN,
Hurum Maksora Tohfa: That learns the, feature space of the donuts, and then we give it the image features, and the focal type, and position, and wavelength, and then,
Hurum Maksora Tohfa: predict the Zernickies instead of doing the forward modeling, which can take a long time.
Hurum Maksora Tohfa: At first, we have done that on the data from April in 2025,
Hurum Maksora Tohfa: And we have tried a few models, with, a few different variations of the model to see what works best. Since we had some, simulated data before.
Hurum Maksora Tohfa: We tried to see if we could transfer learn, but what we end up finding is training on real data works the best.
Hurum Maksora Tohfa: So then we take it, and then we, test it on Sky in September last year. We found that,
Hurum Maksora Tohfa: that AI Donut was able to focus the telescope in much less time, and it was able to also process 5 times more data. And of course, it is extremely fast, and it was also robust to bending and vignetting. So, obviously, we want to, explore what happens, when we,
Hurum Maksora Tohfa: extend the data across the months, because the weather condition and everything changes. Once we did that, we tried some simulated testing, and we see that AI Donnet is able to converge.
Hurum Maksora Tohfa: To the correct Zernikis, and it's also robust to, lens, as, you can see, and also bad, image quality on certain nights.
Hurum Maksora Tohfa: And these are our results on the test set, and it seems to work well on lower-ordered Zernikis, as well as the high-order Zernikis, which is really great. And we are waiting for on-sky testing sometime soon. So yeah, I'll pass it on to Taiwan.
Redwood C/D - 6504984451: We can pause for questions. Yes, yes, yes, that's true.
Redwood C/D - 6504984451: Anyone?
Hurum Maksora Tohfa: Okay.
Redwood C/D - 6504984451: Yeah, there's, like…
Redwood C/D - 6504984451: There are a few slides.
Redwood C/D - 6504984451: It's a lot.
Redwood C/D - 6504984451: I mean, I mean… I'll just skip these two slides. I'm sorry, but,
Redwood C/D - 6504984451: If you can go back to the slides later, because I don't know who… I think they got inserted in out of order, so yeah, go ahead.
Redwood C/D - 6504984451: Yeah, okay, thank you. Oh, okay, hi everyone,
Redwood C/D - 6504984451: My name is Tabian Liu, and I'm a PhD student here at Stanford and SLAC, and I'm happy to be able to share my… also, I'm a part of AI here, so it's about AI for remote.
Redwood C/D - 6504984451: Okay. So, we know from Hammer's talk, the AI don't take the, default images as input, so…
Redwood C/D - 6504984451: So my project is… is kind of a project that trend… that,
Redwood C/D - 6504984451: It's based on the AirDonne method, and we want to transfer it to the science cameras. So how can we get the default images from science cameras? We do kind of a so-called triple exposure. So, the general practice is, firstly, we
Redwood C/D - 6504984451: Def folks the whole, science camera plane to the intrapocal position, take one exposure, so where we can… where we can get the intrapocal donut, and then…
Redwood C/D - 6504984451: extra focal exposure, get the extra focus on it, and then the final, the last infocus exposure. So, through the kind of a,
Redwood C/D - 6504984451: well, triplet exposure, we can get the paired domain images, which can be fed into the AI domain.
Redwood C/D - 6504984451: Okay, so these slides, I want to show some, information about how the neural network is constructed and how we split the data. So, yeah, the neural network is
Redwood C/D - 6504984451: I mean, very similar to the wavefront sensor AI donut, so we… so the both the intrafocal and extra-focal donut will be… will pass through the VESTED 18 encoder, and then the imagery features will be combined with, some metadata, like the…
Redwood C/D - 6504984451: field… the field position, the band, the detector ID, and Raptor IDs.
Redwood C/D - 6504984451: And then those… all those pictures will be fed into a multilateral perception to make predictions. So there are 22…
Redwood C/D - 6504984451: prediction has in total. So 21 of them will predict the Zerniki coefficients, which describes the refund. And there's another one, predicting the
Redwood C/D - 6504984451: the pair blur, which is just an atmospheric scene. So the target of this prediction is just to be in alignment with the Danish
Redwood C/D - 6504984451: Which is our ground truth, Danish output.
Redwood C/D - 6504984451: Okay, and… okay, so this image shows how… how the data is paid. So we use 20 nights for training, and we take kind of an alternating manner. So, one 20 night, one testing night, one 20 night, one testing night, and…
Redwood C/D - 6504984451: Seems like it then.
Redwood C/D - 6504984451: Okay, so we have a… so the training… the… I mean, the amount of training data is around 2 million from the peers, and also 2 million for… for training, 2 million for tests.
Redwood C/D - 6504984451: Well, okay, so these slides want to show somewhere. So, on the left, it's,
Redwood C/D - 6504984451: example of the… our AI donor production. So, to compare it with our…
Redwood C/D - 6504984451: with a Danish label, which is used as our ground truth. And the right, on the right, is kind of an overall performance evaluation. So you see, on the…
Redwood C/D - 6504984451: Other clean… there are very clean donut stems. The AI donut can… can generate very similar results as the all-ground trees. So you see these columns, so we… which are… these columns, which are the comparison of, predictions, they're very close, so…
Redwood C/D - 6504984451: the result, the result looks good, yeah. And under wide, so…
Redwood C/D - 6504984451: Each diagram is one zerith mode, and the x-axis is a…
Redwood C/D - 6504984451: Danish label, the ground truth, and the YYX is, add on the prediction. So, for perfect reproduction, we will expect the Y equal to X, diagonal, straight line. However, it's not that perfect. So, in reality, we see some,
Redwood C/D - 6504984451: Like, these small outlier points, and
Redwood C/D - 6504984451: Also, the kind of, you see, non-zero biases, and also non-zero…
Redwood C/D - 6504984451: I mean, slopes, I mean, the slopes are not 1, so, which means it's not perfect. So…
Redwood C/D - 6504984451: The good thing is that we can still see some, like, clear correlation.
Redwood C/D - 6504984451: Just one sign, yeah. So maybe it's… I mean, it is still useful under some conditions, but we need more training with more data.
Redwood C/D - 6504984451: Probably with more data or more nights covering different part of the space, like ambient temperature, like, different mechanical state.
Redwood C/D - 6504984451: Excellent.
Redwood C/D - 6504984451: Okay, so this is the last slide. We want to talk about some more basic questions. So, why add on full remote?
Redwood C/D - 6504984451: So the first… the first reason is very… is the same as,
Redwood C/D - 6504984451: So we wanted to accelerate the image processing speed. And the second reason, which is…
Redwood C/D - 6504984451: yeah, it's also similar to the add-on-Franc sensors, that it… we think there's some evidence that shows it's more… it's more robust to the image quality, so there are two examples. This one is,
Redwood C/D - 6504984451: So there's… so you see, at the edge, there are the… There is a neighbor, donut.
Redwood C/D - 6504984451: So this is the first example, with kind of a low-quality image, and the second example is that, so low, low signal-to-noise ratio, and there's some… some strange, noise point.
Redwood C/D - 6504984451: this price point. So you see, on both cases, the AI donut outperforms the…
Redwood C/D - 6504984451: initial, like, our traditional algorithm, the Danish algorithm, because these… these features will drag the, output of the Danish very quickly, which is not good.
Redwood C/D - 6504984451: But… But we are still working on some dedicated training for these local quality images to…
Redwood C/D - 6504984451: Makes… makes the preliminary evidence to be a…
Redwood C/D - 6504984451: to be true, it should be more or less. So…
Redwood C/D - 6504984451: Yes, that's all I want to share. Thanks.
Redwood C/D - 6504984451: Oh yeah, I was gonna say, if there are any quick questions while you're… I blocked your eyes.
Redwood C/D - 6504984451: So, walk around.
Redwood C/D - 6504984451: I have a question about those biases.
Redwood C/D - 6504984451: Previous slide, you know what?
Redwood C/D - 6504984451: So, yeah, what… what causes…
Redwood C/D - 6504984451: The reasonable bias? That's not reflected in the next plot of, like, how good is it overall, that's probably… Yeah, yeah, I mean, this is just a single example, but there is kind of statistics. It's very difficult to locate one single reason, because, you know, it's a machine learning.
Redwood C/D - 6504984451: Like, how are you investigating potential reasons?
Redwood C/D - 6504984451: Will… Do you want help investigating the potential? Yes, yes, yes. Well, we'll have to…
Redwood C/D - 6504984451: Actually, the initial version is much… is even worse, much worse. Like, the slope will be compressed to some length.
Redwood C/D - 6504984451: a big population at the center. And, we have to, improve our training method and, improve, like, enlarge our data set.
Redwood C/D - 6504984451: So more training to help fix the virus. Yeah, so, and kind of fixed, so… There's, already happened.
Redwood C/D - 6504984451: fix, fix, I mean, for improvement.
Redwood C/D - 6504984451: With, larger, larger data set, and… Just improvement of training.
Redwood C/D - 6504984451: So… but still, at this stage, I think we have to consider some physical reasons, like what I mentioned, the…
Redwood C/D - 6504984451: particle space. But, like, we only have 20 months. This is very small. So, maybe we can expand it to…
Redwood C/D - 6504984451: more monsters?
Redwood C/D - 6504984451: Or… more, like, optical state, or, I mean, temperature, things like that, so we have…
Redwood C/D - 6504984451: We can expect this can be improved in the future.
Redwood C/D - 6504984451: Thank you. Thanks.
Redwood C/D - 6504984451: All right, so, thanks, Taruan. I'm gonna continue off of the lost amount of work that Hurum and, Taiyun has done already on AI Donuts, and extend it out further with a model that we're calling Tarts.
Redwood C/D - 6504984451: So, so far on our discussion about AI donuts, and also Danish, is the underlying assumption needs to be true, which is that Danish is the optimal algorithm for estimating the wavefronts.
Redwood C/D - 6504984451: And are we exactly sure that's the case, is what we're going to try to look at right now. So, there's some hints that perhaps this is not the best, but a functional algorithm is that if we're to do a simple test of looking at how well does the lowest orders and keep terms correlate to itself.
Redwood C/D - 6504984451: As it evolves through time. And you can see on the top right here, when you look at Danish, at the lowest order left-hand corner, you see that it pretty much immediately de-correlates from itself after, like, two images taken.
Redwood C/D - 6504984451: And you also can look at, perhaps, like, spatial resolute… spatial noise. If you were to look at when you're just observing, just monitoring, not doing active controls, that the highest order terms of the Xernity expansions are quite noisy.
Redwood C/D - 6504984451: And we're suggesting, potentially, that might be a better solution, when I'm hinting at the other models here.
Redwood C/D - 6504984451: And so, with that assumption put in place, how can we potentially improve upon Danish
Redwood C/D - 6504984451: and also AI donuts in the future, extending that further on the work that people have done already. So we're calling this model TARTS, and it stands for a mouthful. It's Triple Stage Alignment Reconstruction Transformer Systems, and that is
Redwood C/D - 6504984451: 100% chat GPT generated, just to fit with the theme of pastries, because donuts, Danish, and not tarts. But yeah, alright, let's continue about this. So…
Redwood C/D - 6504984451: one method for it to potentially do better is, what if you just got better simulations, right? So, as Hurum has mentioned, that Danish has this underlying ray tracing model that's super fast, do all this for ray tracing in real time. But there's also much more expensive ray tracer models out there, for example, MSIP.
Redwood C/D - 6504984451: which takes on the order of hours to run, which is something you definitely can't be doing on Sky in real time.
Redwood C/D - 6504984451: So what if we just emulated that? Something that includes so much more physics, the atmosphere, CCD physics, the underlying optical model, throw the whole kitchen sink at it, see what happens. So that's one thing that we could potentially do better.
Redwood C/D - 6504984451: The other thing is, what if we just had a larger model, okay? So, this part, what we call the WaveNet, is pretty much inheriting off of AI Donut, okay? What if we just extended that with two more models?
Redwood C/D - 6504984451: Okay, so we have one model that does the donut detection, so we're now not just using, like, the 8 donuts that Danish is using, or, like, the 20-ish donuts that, AI Donut is using, but we're using basically everything that we can possibly get out of these corners, right? So this is on the order of maybe 80 donuts, for example, per exposure, per corner.
Redwood C/D - 6504984451: So, individually, these donuts come up with an estimate of their wavefront, cool, but each donut estimates the same wavefront in the end, supposedly. And so, potentially, you can use all of those donuts to get the even better estimate. And so you have an aggregator at the bottom here that gives you a more global estimate here.
Redwood C/D - 6504984451: But you're probably wondering to yourself, Peter, you can't just be training on simulated data, that's probably not gonna work. So how do you incorporate real data into your model without biasing or not having any labels at all? And so this comes in the field of domain adaptation, and there's a method that we can talk about more, just called Derogram, to bridge this sim-to-real gap here.
Redwood C/D - 6504984451: Okay, so what are the results here? When you get larger simulations, bigger models, and more data, do you actually perform any better when you go on Sky and try to test things? So, just like AI Donuts, we also went on Sky, I guess, more recently now. This was July 14th, or something like that, and so here are some results.
Redwood C/D - 6504984451: We ran our model actively controlling in initial alignment, on R-band in a fixed pointing. And what you're looking at here is the evolution of the PSF full attack max, the median of that in arts seconds as it evolves through time.
Redwood C/D - 6504984451: And what we see here is that the red is tarts, and the blue is Danish, and within the same amount of sequences, we do see that we're converging to a lower floor here, which is pretty exciting. But what is even more exciting is that if we were to let this run for even longer, we're able to get sub.6 arcseconds pretty consistently for on the order of maybe 10-ish images, which is pretty much the duration of that test.
Redwood C/D - 6504984451: Here's some pretty pictures of the focal plane here. This is the medium PSS per each detector, and here are images of, you know, the PSS per part of that focal plane. There's, like, red boxes around it.
Redwood C/D - 6504984451: makes it seem like it's doing bad, but it's, like, pretty good. It's, like, 0.55 arcseconds. So… this is looking pretty good.
Redwood C/D - 6504984451: We also can measure the, gradients of the PSF, so we want the focal point to be pretty uniform in performance, and we see that when we do go on Sky? And you see that when we turn on turns with our model, you see basically a step function change in that performance there, which is indicating potentially actual improvement in the performance.
Redwood C/D - 6504984451: I'll just… Craig, credit score to Craig for how she did that analysis for us.
Redwood C/D - 6504984451: Okay, so, so far, things look good, but again, this is a very idealistic scenario. This is one single pointing, sample size of, like, four, right? So, can't really draw conclusive evidence yet.
Redwood C/D - 6504984451: So are we winning? Not yet. But it's showing some real signs of bridging a simulation to a real gap, which is pretty exciting. You know, from Hurum's original analysis, we saw that, for example, you know, simulations might not be the best solution to solving this problem, and that you need to have real data with labels.
Redwood C/D - 6504984451: But this current analysis has shown that, like, we can actually, you know, bridge that gap with simulations and no labels from real data. And, you know, in the field of machine learning, just to us, it's like, this is like the holy grail of bridging sim to real, so that's the exciting part for us, and
Redwood C/D - 6504984451: That gives us confidence to try to maybe, in the next step, to see to extend this to, all the bands, so not just R, and also try to see if we can, in a more realistic survey mode, deliver good performance as well. And we'll get more statistics about, you know, how our performance is sample size greater than 4, hopefully, but…
Redwood C/D - 6504984451: Yeah, summary notes, but yeah, I'm happy to open floor for any questions, and generally, also, Taiyan can also answer questions as well. Just, I had, like, two takes, one… Just a second.
Redwood C/D - 6504984451: I had, like, two things. One, I realized that the other two people were using ResNet18 or something, so that was the CNN-based architecture, and you're using a transformer-based architecture? It's a transformer and a CNN. Okay, so did you, would you say, like, the advantage came mostly because of using the transformer architecture, or…
Redwood C/D - 6504984451: Yeah, so, the transformer… when we did our ablation study of removing this guy, we saw that there were performance degradation in the estimates when ground truth compared to simulation. We didn't burn any on sky time to…
Redwood C/D - 6504984451: tested ablation, but we did see performance improvement. So it's not super substantial, but there were performance improvements by adding this aggregator at the end. Right, okay. And my second thing was, I actually did something very similar, like, I see you have the real data.
Redwood C/D - 6504984451: And because the people who worked before me, they had been training it on simulated data the whole time, so that when LSSTDP1 comes out, we can see how it works on real data. So, when I applied it on real data,
Redwood C/D - 6504984451: Basically.
Redwood C/D - 6504984451: getting it from the normal models to simulated data, simulation, and then to real data, this did exactly the same for me as just taking the base models and directly just applying it with real data. So I would say, like, because I see you're using, like, a bridging between those two. I would say, like, if you try just directly using the real data to train on.
Redwood C/D - 6504984451: it might… you might not need any, like, going through the simulations. So we don't have… we don't… we should not necessarily trust any of the labels that we have for real data. Oh, okay, okay. Yeah, I see that.
Redwood C/D - 6504984451: Sure.
Redwood C/D - 6504984451: we could do that, and that is a lot of the work that Hurum and Taiyuan is doing, which is their training on Danish outputs. We're kind of attacking this problem orthogonally, in the sense that, what if we had something completely different than Danish? Could we still
Redwood C/D - 6504984451: form.
Redwood C/D - 6504984451: similar, or better, or whatever the case is today, or whatever that might be. That makes sense, yeah.
Redwood C/D - 6504984451: And if I can make an observation.
Redwood C/D - 6504984451: So it seems like it's another case where metrics that don't require round proof There's a lot of subterrys.
Redwood C/D - 6504984451: be valuable, so… Yes. Yes. Metrics in different contexts. Yes, exactly.
Redwood C/D - 6504984451: Got it.
Redwood C/D - 6504984451: Are there other questions for all three?
Redwood C/D - 6504984451: Speakers on this topic?
Redwood C/D - 6504984451: Yeah, let's give all three a round of applause.
Redwood C/D - 6504984451: I changed the slides, but… Probably not.
Redwood C/D - 6504984451: This is an excerpt back upside.
Redwood C/D - 6504984451: Great, yes, let's go.
Redwood C/D - 6504984451: Hello, everyone. This is Jay Bhattacharya from Amherst College. I'm a postdoctoral researcher, and I'm working with Dr. Mia De Los Teres, studying, low-mass dwarf galaxies.
Redwood C/D - 6504984451: And the problem that we are, interested in right now is, like, to, to generate a catalog of low redshift dwarf galaxies that can be of,
Redwood C/D - 6504984451: use for, for the, for the,
Redwood C/D - 6504984451: Nomass Galaxy Low Surface Brightness, Galaxy community. So…
Redwood C/D - 6504984451: So, so the problem, is, like, at, say, redshifts less than 0.1, it is very hard to, constrain, like, photometric redshifts and, other methods, like surface brightness fluctuation, I mean…
Redwood C/D - 6504984451: which has been used to measure, like, redshifts at scale for the local volume, which isn't, like, applicable, let's say, when you are, like, looking at 100 megaparsec or something like that. So, so we are trying to, use the
Redwood C/D - 6504984451: use the XSAGA framework, so that was, this, this ResNet-based classifier that was, designed by our collaborator, John Wu.
Redwood C/D - 6504984451: To, to, identify such low redshift candidates, from, from the legacy survey using, spectroscopic redshifts from, from, from the SAGA, survey. So…
Redwood C/D - 6504984451: We are trying to repurpose that and apply that to LSST.
Redwood C/D - 6504984451: And, today, I will, like, give a… I mean, we already… President… me and, yeah, we presented this at the low surface brightness galaxy Session, but I think after DP2, we are also, like, recalibrating as to, like, what's of, like, the…
Redwood C/D - 6504984451: greatest interest here, and, and also looking at some, some, some, limitations of, of, of, of this. So…
Redwood C/D - 6504984451: So right now, what we… what we are…
Redwood C/D - 6504984451: planning with the GP2 release is that to train this on, say, on a spectroscopic catalog of galaxies, typically greater than 10,000 galaxies, and to deliver this catalog. And over here, what you see are these outputs of the original XSaga.
Redwood C/D - 6504984451: to show what Xaga is… does, so Xaga classifies, like.
Redwood C/D - 6504984451: low near, nearby galaxies against more distant high redshift galaxies. So the galaxies in the upper row, these are the high redshift distant galaxies. They have this lower value of this probability, that's, provided by the
Redwood C/D - 6504984451: the CNN, and the low redshift galaxies you see in the bottom, bottom row. So these are the objects of our interest, and, these have a higher probability. So…
Redwood C/D - 6504984451: So yeah, this is what we are trying to do. So this is legacy survey imaging. We want to, like.
Redwood C/D - 6504984451: do this for LSST, because right now it's TP2, but once we have TR1, and as the survey progresses, it's going to get better, and as we cover a larger area of the sky, I think we can, like, generate a very large sample of such low-mass star galaxies.
Redwood C/D - 6504984451: Yeah, so, so, the… right now, what we have done is we have designed this pipeline that can efficiently, like, extract UGRA cutouts in bulk very efficiently, and we also, like, perform, like, preliminary training using, like, XSACA. And,
Redwood C/D - 6504984451: And yeah, so these are some… some…
Redwood C/D - 6504984451: images from DP1 that we use, like, these are on the top row shows the nearby galaxies, and the bottom shows, like, the distant galaxies. So, as you can see, the nearby galaxies are more extended, and I mean, that's what we are, we are trying for the model to learn, and thereby classify these galaxies.
Redwood C/D - 6504984451: So, so yeah, this is… this is the main project that I've been working on. But aside from this, I'm also interested in galaxy morphologies, because, recently, I think two weeks back, I, I, completed this paper, this work that I've been,
Redwood C/D - 6504984451: engaged in, so that's on the non-parametric morphologies of low-mass galaxies, and I'm using multiband photometry. Not just a signal band, but GRIZ. Once again, this is legacy survey.
Redwood C/D - 6504984451: So, so the utility of that is, if we study the non-parametric morphologies, like the GD coefficient, F2G, concentration, asymmetry, smoothness parameters.
Redwood C/D - 6504984451: it has been found that these, parameters… these, measures are, like, can be… have correlations with the physical properties of the galaxy, like stellar mass, star formation rate, so… so there's some, utility here, I think, that
Redwood C/D - 6504984451: can, like, explore further. So the… on the left-hand side, you see this plot from the paper, where we are doing, like, the image segmentation, and then measuring the morphologies using this code called stat morph. And the thing with the low-mass galaxies that you, like, look at here closely is, like, there, sometimes the galaxies are off, like, irregular shape.
Redwood C/D - 6504984451: Like, sometimes there isn't, like, a… I mean, there's, like, star-forming regions, and also there's lots of stuff going on, and for this, the two parameters, asymmetry and smoothness, I think that
Redwood C/D - 6504984451: that will be very useful, and I think we can capture in a greater detail than the legacy survey, with, with LSST. So, so, so that's what I'm, like, interested in to,
Redwood C/D - 6504984451: whether, if we were to, like, use, something like a neural net or something, or we do, like, the non-parametric
Redwood C/D - 6504984451: measurements and do something like dimension energy reduction. Actually, we do a little bit of that in our paper, so, we apply UMAP to see, like, how the… how the parameters are… are distributed in this space. So, so yeah, this is something I'm working on. I think I'm also, like, part of, like, the… the work of the
Redwood C/D - 6504984451: Galaxy Morphology challenge that's, being undertaken by the Galaxy Science, collaboration. So, so yeah, it is an interesting time, and I also want to, like, compare how, like, these measurements, are between, say, Legacy Survey and SST, LTP2. So, so yeah, so this is, some of the interesting…
Redwood C/D - 6504984451: things that I've been doing, and I would like to answer any questions. If you have, that I would like to cut you by as well.
Redwood C/D - 6504984451: Start discussions, yeah.
Redwood C/D - 6504984451: Okay.
Redwood C/D - 6504984451: I'm curious to know, I'm sure I could read the paper, but you can also tell me about how, you, were able to cluster your sources, and which algorithm you used.
Redwood C/D - 6504984451: Is this even on? It's for the people online. Oh, okay, not for the room, so I need to speak louder anyway. Yeah, which, learning method or algorithm did you use to, to create the representation that you've visualized in your UMAP space there?
Redwood C/D - 6504984451: Oh, so, so yeah, I, I had something like 20, sort of, like,
Redwood C/D - 6504984451: Data points for each galaxy, and the number of galaxies were, like, 2,000.
Redwood C/D - 6504984451: So, so yeah, so that's, I mean, a large thing, and I think, I thought, like, UMAP, this, this uniform manifold, some sort of, like, projection, this was, like, one of, like, the,
Redwood C/D - 6504984451: methods that people tend to use for dimensional deduction reduction to see how things are testing. So, this is more of, like, a
Redwood C/D - 6504984451: Proof of concept, rather than the main thing in this paper.
Redwood C/D - 6504984451: But, I think, I know, like, other groups, they are also, like, considering, like, methods like climate change reduction to, like, study, like, galactic morphologies.
Redwood C/D - 6504984451: So, yeah, over here, I used UMAP to answer your… did I answer it? Okay, I'm sure you have, just didn't understand the method. Have you only used, optical photometry in this? Okay.
Redwood C/D - 6504984451: Okay, I'll read the paper. Thanks.
Redwood C/D - 6504984451: Just a quick question, why did you…
Redwood C/D - 6504984451: Just a quick question, why did you not use the U-band?
Redwood C/D - 6504984451: Oh, so this is Legacy Survey. Oh, okay, okay, sorry, I thought it's LSSCDP1. Yeah, this is… this is Legacy Survey, but I mean, these are mostly star-forming galaxies, and I mean, if we use, like, the U-band in, like, the DP2, I think that will… that will provide us more information.
Redwood C/D - 6504984451: It sounds like it would be a good… Hackathon.
Redwood C/D - 6504984451: project. Yes. Yeah.
Redwood C/D - 6504984451: relate them directly to each other. Yes, yeah, and also, yeah, and also these depend on, like, noise and resolution, very much so, so, like, comparing with how this, like, depends on, like, noise and resolution across legacy survey and
Redwood C/D - 6504984451: TP2, that would also be something interesting.
Redwood C/D - 6504984451: Okay, well, thank you.
Redwood C/D - 6504984451: Yeah, so, I have just one slide. There's… yeah, that's fine. So, I wanted to discuss the problem, so I'll not sell any solution.
Redwood C/D - 6504984451: So I touched upon this problem yesterday in the strong Lanza session here.
Redwood C/D - 6504984451: And, yeah, so let me start. Like, the name says that there's an image time series, there will be multiple bands, so all of these data points are basically an image in multiple bands.
Redwood C/D - 6504984451: So you know that AISST, like, scans the whole sky in multiple bands. The problem is that?
Redwood C/D - 6504984451: At a given point in time, you have observation in only one band. That makes the…
Redwood C/D - 6504984451: Time series in different bands asynchronously.
Redwood C/D - 6504984451: Now, So, we deal with this for the strong lens transient surge.
Redwood C/D - 6504984451: But I thought that this might be one of the generic problems that the whole LSST transient community should encounter.
Redwood C/D - 6504984451: forget about any logistic challenges, because, like, suppose there are many groups who analyze one frame, maybe a co-add, so this is basically the static, you find VITs, resonates, units, for not.
Redwood C/D - 6504984451: The transient community actually uses this, but you replace each… this n by n pixel image by one photometric data, so that becomes your light curve.
Redwood C/D - 6504984451: But what if you want to analyze the full information?
Redwood C/D - 6504984451: I was searching… The solution for this problem for years.
Redwood C/D - 6504984451: I thought that somebody in the computer fission community, they must have, given
Redwood C/D - 6504984451: Some well-known, well-established foundational model, like some model that we can reuse. We do not need to reinvent the solution.
Redwood C/D - 6504984451: I could not find one.
Redwood C/D - 6504984451: I asked many people, like, in Germany, US, you know, do you know? I could not find it. Okay, so we have solved it, I solved 50%, Claude's solved maybe the other 50%?
Redwood C/D - 6504984451: But the… so my question here is not the solution. My question is that how important this particular question… so I'll ask you a very simple question. How many of you think that these problems should be solved, and could be useful for your science case?
Redwood C/D - 6504984451: Like, just, just, just to clarify, you mean that the question being that, The probabilities for…
Redwood C/D - 6504984451: each time step don't know about. Yes, I'm so sorry I didn't explain the last part. With online prediction, that means…
Redwood C/D - 6504984451: You do not wait for the series to complete, so after each observation, you predict.
Redwood C/D - 6504984451: Suppose you are here, then you are seeing this, and the past is accessible to you.
Redwood C/D - 6504984451: So, you have only data or observation till this point, and based on that, you have to predict. So, naturally, as the observation goes on, goes on, you are accruing more and more data, so your prediction improves and improves.
Redwood C/D - 6504984451: So that, that is the last one. I'm so sorry that I forwarded the, explaining. So, yeah, so anyone thinks that this is worth solving? Yes? I mean, I think in…
Redwood C/D - 6504984451: Like, not in physical systems so much, but in engineering systems, this is a problem that exists and is worked on, where you have multiple sensors that predict… So, all the papers I have… I went through CVPR myself, Claude went through maybe the others.
Redwood C/D - 6504984451: CDPR is their, like, their flagship conference for the computer vision community, and they write
Redwood C/D - 6504984451: Yeah, like, burning questions, the answers there. I couldn't find it. So, if you remove one problem from this particular problem, things become much easier.
Redwood C/D - 6504984451: If you, like, reduce this, From a 2D image to one number, solutions there.
Redwood C/D - 6504984451: If you align the frames, this is the classic video transfer, like the video transfer, and all the self-driving car with different sensors, so the problem is that their frames are aligned, or…
Redwood C/D - 6504984451: they are so, finely spaced that you are allowed to do interpolation. So here, another thing I, forgot to mention, you're not allowed to do any type of interpolition. So you have to deal with only the observation.
Redwood C/D - 6504984451: I think the bottleneck is the image-based problem. I think the bottleneck would be the fact that this is an image-based problem, but there's multi-dimensional
Redwood C/D - 6504984451: Sensors that are not…
Redwood C/D - 6504984451: multidimensional inputs that are not images for which this problem is worked on, right? For the self-driving car, suppose you have multiple cameras, so then that is an image-based problem. But in that case, the frames are…
Redwood C/D - 6504984451: Allied. But suppose you are taking some radar data, which is not… I do not know whether the radars are, like, can be one-dimensional, but of course, there could be some…
Redwood C/D - 6504984451: asynchronous one-dimensional data, and for… for that, the solution exists. The solutions are not groundbreaking. So you just basically need, transformers, you mix the spatial and temporal transformer in some certain
Redwood C/D - 6504984451: Okay? But what surprised me, that I could not find a solution, and yes, so there, like, still, I'm pretty sure there exists some obscure paper, which I could not find. But, yeah, so that actually surprised me.
Redwood C/D - 6504984451: And there was a second slide that's the solution, but yeah, so this is not so, interesting. So you can think of the solution. If you are a master of transformers, then I'm pretty sure, like, if you can…
Redwood C/D - 6504984451: like, design your own attention mechanism, then you can solve it. So there is no, not some issues, and anyway, we are writing paper.
Redwood C/D - 6504984451: That's not a thing. But I wanted to highlight that this problem exists, and if I understand it correctly.
Redwood C/D - 6504984451: Then, that gives you the best solution, like, be it a…
Redwood C/D - 6504984451: Binary classification, be it some sort of regression, be it generation, it doesn't matter.
Redwood C/D - 6504984451: So, how you, like, analyze it, so, yeah. So, I want to edit, sorry.
Redwood C/D - 6504984451: Alex, can I?
Redwood C/D - 6504984451: Yeah, I, I wonder if I… it's,
Redwood C/D - 6504984451: Why does it have to be images? Because… because there are solutions when it's just, like, if it were photometric light curves instead of looking at the images. What do you… do you lose something by using those solutions instead of the images? It depends on the problems, right? So, for example,
Redwood C/D - 6504984451: Thanks.
Redwood C/D - 6504984451: So, we stumbled upon this problem when we were analyzing, we were trying to detect the lens transients as early as possible.
Redwood C/D - 6504984451: But there could be many other problems. So, whenever you reduce the whole information into one number, you must be losing something. So that might not be so important for some science cases. For example, if you are classifying transients from the light curve, that's fine.
Redwood C/D - 6504984451: But there could be some, like, there should be, like, you are always losing some information to get to the solution. But you can also do that classification from the images, and another part is that
Redwood C/D - 6504984451: The future time domain surveys, so this will, I think, again, like, what do I know? But still, like, with my small, experience, I think that this would be a very generic problem that you have to deal with.
Redwood C/D - 6504984451: If you embed your images, if you use an embedder, like, because…
Redwood C/D - 6504984451: So we agreed that the problem…
Redwood C/D - 6504984451: That solutions to the problem exist.
Redwood C/D - 6504984451: If the input is not images, but if you can embed your images in an input that is much simpler while not being a single data point. Doesn't that bring you back to existing solution engineering domain?
Redwood C/D - 6504984451: So, we… the solutions we found are basically this. So, you embed the… so basically a mixture of marriage between VITs and temporal transformers, and basically, it follows this technique. So, it basically not… so we do not embed the full image into one token.
Redwood C/D - 6504984451: Rather, you take a VIT approach, that you patchify the images. So, all these things are technical, right? So, if you are interested, you can always talk to me, you can even talk to Claude, Claude can explain it better.
Redwood C/D - 6504984451: Much better than me. But I just wanted to inform people that this problem… for me, like, I was surprised at why nobody, came a solution to this.
Redwood C/D - 6504984451: Yeah, I'm afraid you can just…
Redwood C/D - 6504984451: I'll say that there's… there's been some research… So, I already showed these yesterday, so…
Redwood C/D - 6504984451: Well, I meant, there's been some recent work by an ISSC member, Grant Joe, who's a grad student at CMU, that… it's not quite… so instead of images, it's like…
Redwood C/D - 6504984451: A multiplied flight curve is one of… can be thought of as multi-channel, and also the probability vector from the previous classifications as another set of channels.
Redwood C/D - 6504984451: Because there's… it's over time, and it's multi-class, so there are… there are higher dimensional methods that deal with this, not from the images, but you… even if you turn the images into, like.
Redwood C/D - 6504984451: it doesn't have to be two parameters, but, like, some… some latent space that can still be a fair number of parameters. There are ways of dealing with this.
Redwood C/D - 6504984451: Yeah, of course, like, if there… so, if there existed a solution, I didn't have to find one. So, my complaint is that I could not find one. It is a very recent paper. So, okay, we'll… Okay, so we'll communicate on this. Thank you.
Redwood C/D - 6504984451: That would be me. Do you mind, refreshing the browser?
Redwood C/D - 6504984451: It should be a slide before that.
Redwood C/D - 6504984451: Oh, there it is. Okay, okay. It's there.
Redwood C/D - 6504984451: Okay.
Redwood C/D - 6504984451: So, kind of part of my responsibility, being a South African astronomer here at CAPAC, where I've been visiting, for 9 months now. I'll be here for another 3 months on my Fulbright, 1-year fellowship, is to disseminate as much information as possible about
Redwood C/D - 6504984451: SKA, square kilometer array, and, developments thereof.
Redwood C/D - 6504984451: since the American community has unfortunately somehow been kind of left out of all of that, given that the USA is not a partner country. I'm probably saying some controversial things, so I'll stop.
Redwood C/D - 6504984451: But, thankfully, there are opportunities for partnership, collaborative.
Redwood C/D - 6504984451: partnerships between Rubin LSST and the SKA, and this has been made possible through the in-kind programs that have been developed. There's several of them. In-kind partnerships with Chile, of course, which is hosting the actual LSST CAM and the observatory itself, but also
Redwood C/D - 6504984451: with South Africa through our in-kind partnership that is operated by the South African Astronomical Observatory.
Redwood C/D - 6504984451: Where we currently have 10 PIs selected, so these are all faculty members at various universities and institutes in South Africa, who have their own scientific foci, and also groups of junior associates, as we call them, students.
Redwood C/D - 6504984451: Or postdocs, early career scientists, and the idea is to make sure that data rights access
Redwood C/D - 6504984451: is made available to, South African, researchers through this program. So, given the timeline of the SKA, optimistically, we expect to have a completed telescope by the early to mid-2030s.
Redwood C/D - 6504984451: So, given also the timeline of Rubin LSST and the decadal survey, there'll be an excellent opportunity to combine deep radio continuum observations, such as radio continuum.
Redwood C/D - 6504984451: polarization products, polarized emission, observations, and also H1, spectral line emission, which will be, more sensitive than has ever been, given the scale of the SKA.
Redwood C/D - 6504984451: So that's happening, and it's something, that if American researchers, I, I'd love you to be aware of.
Redwood C/D - 6504984451: So that you can form partnerships with, astronomers in South Africa. So just a snapshot, an example of something that I've already got a student of mine working on, Maria Zoras, who is at UNISA, which is my home institute. She is now combining radio
Redwood C/D - 6504984451: continuum observations that we already have from Meerkat, which is the 64 antenna, array that has already been commissioned now.
Redwood C/D - 6504984451: Five years ago.
Redwood C/D - 6504984451: And combining this with archival, optical, and near-infrared photometry to try to, firstly label the data, according to empirical models, such as the infrared radio correlation, which gives us our QIR, that allows us to determine where our star-forming galaxies and radio-loud AGN
Redwood C/D - 6504984451: And, other as-yet-unknown classifications of radio if it's galaxies. So this… the whole focus here is galaxies, essentially. So she's, using, the empirical data to inform the models.
Redwood C/D - 6504984451: And, effectively apply a SIMCLR procedure to get a learned representation of the data based on labels that, she's assigned to the data, informed by the empirical
Redwood C/D - 6504984451: Work of the… in the archives, in the… in the literature.
Redwood C/D - 6504984451: And projected it into UMAP 2D space, as we saw in Joy's plot. That's a way to visualize clusters of sources.
Redwood C/D - 6504984451: And then used HDBCAN to find groupings of these sources, and she's been able to, within this UMAP space, find where the star-forming galaxies Radio AGN, radio quiet, radio access. So the idea, what she's doing, since it's just an MSC project, is to see if, the empirical
Redwood C/D - 6504984451: relations can be reproduced by these,
Redwood C/D - 6504984451: algorithms and machine learning methods that have been used time and time again, mainly in industry, but now we're applying them to astronomy cases. So, if the question here is just open for discussion, you're welcome to find me.
Redwood C/D - 6504984451: wherever you see me, and, we want to figure out how, DP2 Rubin photometry will be able to… to possibly improve our ability to find out where all these different classifications are. So that's something that,
Redwood C/D - 6504984451: I will be definitely able to get as much, information on improving from those who've worked on these kinds of projects. Yes, thank you very much.
Redwood C/D - 6504984451: Are there… are there specific questions?
Redwood C/D - 6504984451: Nice. Okay.
Redwood C/D - 6504984451: Thank you for sharing this. This is very exciting, Nabilh. So, over here, so what radio data are you using? Yeah, so the radio data we have is from the MITE survey, so it's a…
Redwood C/D - 6504984451: Meerkat Continuum Survey that aims to target deep drilling fields such as Cosmos, XMM, LSS, Elias S1,
Redwood C/D - 6504984451: And that's at, in the L-band, so between 1 to 2 gigahertz. And we've already got all of that imaging completed, and it's, it's, been calibrated.
Redwood C/D - 6504984451: And from that, we're able to run our source finders to… to pick out where we get contiguous, kind of, 5 sigma plus radio emission to identify where our…
Redwood C/D - 6504984451: radio-emitting galaxies most likely are, and we combine that with multi-wavelength archival data. Yeah, so that's the story, how we get the data.
Redwood C/D - 6504984451: And now we're applying these, methods to try to, develop more automated methods for classification.
Redwood C/D - 6504984451: Because all empirical methods can be very slow when we have large volumes of data.
Redwood C/D - 6504984451: Let me see. If there's… can I ask another question, quick question, yeah. So, when we talk about radio data, it's like, intensity, so, I mean, what will SK provide us, like, intensity, polarization, and will you… will that be helpful in this case? Yes.
Redwood C/D - 6504984451: So the SKA, so you can think of the data products where, and they've been calibrated and imaged as, your continuum imaging, which is your intensity map, so that's the eye parameter, the Stokes eye parameter, and then you'll have the QUV,
Redwood C/D - 6504984451: Which provides your polarization products, and then you've also got your H1 21cm spectral line cubes that will also be provided in 4K mode. So that's the three, kind of, categories that we'll get with SKA, yeah. Sure, thank you.
Redwood C/D - 6504984451: Yeah, I'll hold my questions, because they're not really statistical. No, this is very… this is very cool.
Redwood C/D - 6504984451: Super excited about the possibility of having labels…
Redwood C/D - 6504984451: That, that couldn't have been derived from one dataset, but seeing can we
Redwood C/D - 6504984451: predict from one to another, to see if this is happening in the supermit yet.
Redwood C/D - 6504984451: So yeah, cool.
Redwood C/D - 6504984451: Okay, so…
Redwood C/D - 6504984451: Do I just stop and they hear me? Yeah, cool. So, I was asked to maybe present a slide
Redwood C/D - 6504984451: like, earlier today, so apologies for how rough this is. Essentially, the USDF team is trying something different.
Redwood C/D - 6504984451: We are looking to monitor all the infrastructure that is below prompt processing and alert production, so this is, like, the network, the file system, storage, so that we don't run into problems, during observing. And one of the ways we want to do this is assisted by LLM. We already have code that essentially hits,
Redwood C/D - 6504984451: About 150 of these systems.
Redwood C/D - 6504984451: And checks them for, readiness, for, functionality. But, we're in the process of connecting it up to various LLM
Redwood C/D - 6504984451: MCP sort of situations, and also,
Redwood C/D - 6504984451: The idea, sort of, is that we will have pre-observing checks, like a daytime checkout sort of thing for our infrastructure. We'll have during-observing checks, where some, some, a few tests will run on a loop.
Redwood C/D - 6504984451: And somebody could be notified if something goes wrong. And we'll have post-observing checks to make sure that everything's in a good state at the end.
Redwood C/D - 6504984451: So, essentially, this is just starting out. I've essentially just pinged every system I want to test, and for…
Redwood C/D - 6504984451: people, there's a CLI tool that outputs data that looks kind of like this, right? This is from, like, a post-observing analysis of a day where the Ceph cluster, like, went out, and you can see, like, oh, all this stuff is wrong, and
Redwood C/D - 6504984451: all these other systems went fine. But I figured it might be helpful to ask you guys
Redwood C/D - 6504984451: sort of the questions I have, which are, like.
Redwood C/D - 6504984451: what's a helpful way to display information on, like, infrastructure systems, like, during observing, and, like, who's going to be awake, and how do we get that information succinctly to a person?
Redwood C/D - 6504984451: how can we reasonably compare information that's of, like, wildly different formats, right? Like, I know we have Profana dashboards, there's a lot of them, but we're kind of hoping we can get to really drill down into, like.
Redwood C/D - 6504984451: What is wrong?
Redwood C/D - 6504984451: And across the whole system.
Redwood C/D - 6504984451: And there's a whole follow-up check architecture that we want to enable,
Redwood C/D - 6504984451: This sort of, like, yes, here's the response.
Redwood C/D - 6504984451: And then also, like, my…
Redwood C/D - 6504984451: forays thus far into, like, a local LLM for interpreting this data have really led me to realize that,
Redwood C/D - 6504984451: we have a lot of it. And contact management is really hard. You can't just, like, call one check and have it be, like, processed and tell you the root cause was this one, you know? So, like, how would you sign a system, that can manage, like.
Redwood C/D - 6504984451: metadata on 150 systems, like a thousand-line JSON file. So yeah, this is…
Redwood C/D - 6504984451: Also, if you… if you think monitoring infrastructure systems under AP is a cool idea, and you want to work on that, feel free to let me and Adam and Adi know. But yeah, I was asked if I had anything to bring people.
Redwood C/D - 6504984451: This is what I've got, so…
Redwood C/D - 6504984451: I'm excited to see what you think.
Redwood C/D - 6504984451: Yep.
Redwood C/D - 6504984451: Actually, I think that might be… I think you're in the room. Yeah, how many, how much tokens do you expect to be loaded into contact, say, if you have trouble managing?
Redwood C/D - 6504984451: So, the idea here would be to run something on, like, a local node, like a Gemma 4 model. So I don't know how many tokens that actually expense, or if token… I don't think tokens are involved in this, but I may be wrong. Well, how much… how much tokens are you inputting into keeping contacts?
Redwood C/D - 6504984451: I don't have a good answer to that question, okay.
Redwood C/D - 6504984451: But I will happily talk with you later about it. Sure.
Redwood C/D - 6504984451: because there's, like, ways of people in the industry that manage context with things called, like, Ralph Loops, and compaction, and doing, like, summarize… auto-summarizing of, like, things and context, and then reloading that summary into context, and tricks that people do, so there's, like, a…
Redwood C/D - 6504984451: cool thing about it. Cool, yeah.
Redwood C/D - 6504984451: Yeah, I'll look into it.
Redwood C/D - 6504984451: Yeah, for reference, at this point, I have a JSON file, so… Gotcha.
Redwood C/D - 6504984451: Yeah.
Redwood C/D - 6504984451: Thoughts, questions?
Redwood C/D - 6504984451: I hope on the UX side, for your first bullet, that you actually are talking to the staff who are awake in the middle of the night as to how they want to see the information. Yes.
Redwood C/D - 6504984451: Adam, you have anything to add?
Redwood C/D - 6504984451: Yeah, that's a… it's something that… it's a conversation we've had going for a while now, since even before we started thinking about this in… in the Agentic LLM kind of way, even when we were thinking about just straight-ahead deterministic monitoring and… and,
Redwood C/D - 6504984451: Yeah, I think we realized pretty quickly that anybody who's looking at this as a…
Redwood C/D - 6504984451: They're sitting in a different role, and they're looking at other things simultaneously, and so, yeah, just in terms of the basic monitoring dashboards, we already… those break out into things that you might imagine exposing to the observers at night, which have to be very unobtrusive and only really show them something if they can
Redwood C/D - 6504984451: If it's really something they need to act on, and they know what action they need to take by seeing it, but they definitely don't need to be overwhelmed with a bunch of
Redwood C/D - 6504984451: down in the weeds information about what's going on under the hood. And then there's, like, another level of, sort of, Tier 2 support, maybe, where, okay, there is a problem, and can you at least get a system to triage it to the level of, like, do you need to wake up a network person, or…
Redwood C/D - 6504984451: a telescope person, or what kind of… what subsystem is the problem coming from? And then, yeah, even beyond that, I think it's interesting to see, and this is where we, you know, potentially can go deeper on it at the infrastructure layer within the USDF here, is, like.
Redwood C/D - 6504984451: can you actually have an autonomous expert system go really deep into the, like, diagnosing the root cause in this fairly complex, environment here? So…
Redwood C/D - 6504984451: I think we're enthusiastic, but it's still, still early, and of course, we're still early in operations as well.
Redwood C/D - 6504984451: Yeah, so, what type of safety guardrail do you have, like, and how confident you are?
Redwood C/D - 6504984451: Yeah. Okay. So, I guess I should clarify, I'm asking at a point where I have not connected up an LLM yet. But, this is partly because I've been spending a lot of time on making sure that there are guardrails in place. So, the architecture is,
Redwood C/D - 6504984451: There's, I have a Python tool that I call through CLI that, does all these checks, and then, there's a specific endpoint that,
Redwood C/D - 6504984451: the AIC people use, and those MC… those, are tracked in a session. So I have an orchestrator, which at the… that, like, the system.
Redwood C/D - 6504984451: spun up, it starts a session, and then during that session, there's essentially a hard limit on how many times any of these components can be hit or queried. And there's,
Redwood C/D - 6504984451: like, individually, but also overall, also in a certain time frame. And then, for testing, but also for human diagnosis, I have,
Redwood C/D - 6504984451: I have an interactive mode that I'm using to sort of,
Redwood C/D - 6504984451: put the guard barrels in place, one by one, test them myself, and, like, see that the code path actually works just fine. So, it's all in progress, but I've spent about… Yeah, actually, my fear is that what even beaches the guardrails.
Redwood C/D - 6504984451: I do not know the answer, but I'm also figuring out that how to put, like, a hard guardrail that can never be breached.
Redwood C/D - 6504984451: I'm pretty sure that community knows this, I…
Redwood C/D - 6504984451: So, for example, RF minus… so, RF minus RF dot can never be executed, or whatever, like, the world is destroyed, I don't care. But this… so, one is that you…
Redwood C/D - 6504984451: Alias it, like, so you hired this comment.
Redwood C/D - 6504984451: But these are, like, very elementary stuff. I'm pretty sure that the industry has figured out something better, I guess.
Redwood C/D - 6504984451: Yeah, I guess part of the answer is just to proceed cautiously and start with something that's only offering reports and summaries and perhaps debugging things, and then the next stage might be it's suggesting actions that humans might take, but not actually taking those actions autonomously, and then
Redwood C/D - 6504984451: once we… there's… I guess there's a trust-building exercise, almost, that it's just like you do with people, although it's not exactly people. And yeah, at some point, if it's actually taking autonomous actions, right, then it still needs to be very well prescribed as to what those actions can be, and
Redwood C/D - 6504984451: how its permissions are established and inherited, and how it, you know, making sure it doesn't circumvent them. Yeah, it's… I mean, it seems like a…
Redwood C/D - 6504984451: I don't even… Want to speculate what our cybersecurity colleagues are thinking about all of this right now.
Redwood C/D - 6504984451: Yeah, for now, the flint is…
Redwood C/D - 6504984451: check on things, and get a report. So it's just a lot of queries.
Redwood C/D - 6504984451: I have one question, it's a little bit logistical.
Redwood C/D - 6504984451: So, this… this information, is it possible to make it public and make, like, a little data challenge of, like, come up with a way to visualize and predict on this? Or is it too… too secret to really see?
Redwood C/D - 6504984451: I think.
Redwood C/D - 6504984451: I would be surprised if stuff that's happening in SDF, which is most of this.
Redwood C/D - 6504984451: was publicly available in any way. Even as Ruben and SLAC staff, I had to get, like, a special token to look at Girvana.
Redwood C/D - 6504984451: So…
Redwood C/D - 6504984451: One thing we've talked about doing is going back and assembling the logs and telemetry from past incidents where the root cause is known, and using that, potentially dividing that into training and validation.
Redwood C/D - 6504984451: Samples and seeing what, you know, which types of agents are best at
Redwood C/D - 6504984451: Synthesizing all of that unstructured data and getting to the right answer when we know, based on the history of what happened, what the actual problem was.
Redwood C/D - 6504984451: But, yeah, that's just an idea right now. We haven't… I don't think we've tried to actually assemble that data set.
Redwood C/D - 6504984451: Yeah, we've, we've identified a few, incidents and fixes.
Redwood C/D - 6504984451: Great, you have actually done… that's true, you have done some of this, but I think we haven't…
Redwood C/D - 6504984451: Where are… would we potentially… we'll have to think about this some more… whether… whether we might even be able to…
Redwood C/D - 6504984451: package this and distribute it to other… that's an interesting idea.
Redwood C/D - 6504984451: And then the next question after that is, okay, if you did have something that performs particularly well, then…
Redwood C/D - 6504984451: Like, what's the process of bringing that back inside the fence and integrating it?
Redwood C/D - 6504984451: Looking forward to iterating on this, in the future.
Redwood C/D - 6504984451: Thank you.
Wilson, Tom J.: Alex, can I, interrupt? I think you've missed at least me. This is the last slide on the thing, right?
Redwood C/D - 6504984451: Council?
Redwood C/D - 6504984451: in an unexpected…
Wilson, Tom J.: I got caught in your cached Google Slides.
Wilson, Tom J.: I don't know if you've missed anybody else. I was… I put myself somewhere in the middle, was the mistake I made.
Wilson, Tom J.: Above the red.
Redwood C/D - 6504984451: Have you moved them to the end?
Wilson, Tom J.: No, I…
Redwood C/D - 6504984451: Just quickly move them to the end, I can refresh and…
Redwood C/D - 6504984451: Because my mouse is so tiny on this very.
Wilson, Tom J.: Right. Okay, yeah, I can do it then.
Redwood C/D - 6504984451: Okay, cool.
Wilson, Tom J.: I don't know.
Redwood C/D - 6504984451: Oops.
Wilson, Tom J.: Okay, there we go. I'm at the bottom now.
Redwood C/D - 6504984451: Okay.
Wilson, Tom J.: Hopefully, it was just me that That's why I thought.
Wilson, Tom J.: You can go to next slide, please, as I need the one. I just put that one in to not get caught out.
Wilson, Tom J.: Yeah, hi everyone. I'm sort of coming at this from an ISSC point of view from the other direction, because I've done the ISSC bit, almost, with this work. Apologies to anyone who was in the previous session, where you'll have heard this talk just a second ago.
Wilson, Tom J.: But the very quick summary, now that we're approaching the hour, is that I have…
Wilson, Tom J.: I have done all of the complicated statistical analysis of subtracting out all of the contributions to measured Rubin astrometry that you don't want.
Wilson, Tom J.: And I'm now hopefully comparing, as well as you can, the centroid precisions… the centroids measured by the Rubin pipeline to the formal covariance matrices, as also determined by the pipeline.
Wilson, Tom J.: Where essentially you… you would envision some sort of Monte Carlo simulation of repeat measurements which build you a standard deviation of… of some sort of truth error residual.
Wilson, Tom J.: And you compare that to the covariances, and you hope that Y equals X, and therefore your scatter in your positions equals your uncertainty.
Wilson, Tom J.: And the very high-level summary here is that it doesn't… that we fit this very simple model in the middle of the slide there, adding in quadrature some bright and systematic with a slope-corrected covariance between our
Wilson, Tom J.: these empirical scatter and error residuals, and the quoted pipeline uncertainties. And in very small figures, because I had to put all of them on this one slide, you see that in one side we have
Wilson, Tom J.: M of 1, if we use the faint data from Hubble, so we're down at 26th, 27th magnitude. So M is 1, which is great. You can ignore the N on the right-hand side, but when we, for the brighter sources, brighter than 20th magnitude in, Gaia and LSST,
Wilson, Tom J.: We see a, a systematic, slope in the faint but 20th magnitude, data, and a, a bright end,
Wilson, Tom J.: flaw to the astronometry of something like 5 or 10 milliarseconds.
Wilson, Tom J.: But of course, the problem I have is that I've just done an end-to-end test, so I can't answer why this is true. And I sort of wanted to come from this, the other side, and ask, is anyone else looking into the LSST astrometry, for any of the data previews, one or, you know, two, two going forwards?
Wilson, Tom J.: And, you know, maybe we can sort of meet in the middle, where I've got
Wilson, Tom J.: I've got an answer, but I don't know why, and if there are other people who are looking at astronometry from a why-are-things-happening point of view, we had,
Wilson, Tom J.: I talk in the SMWLV session, from Clare Saunders about, the turbulence model, for example, the Gaussian processing fit that should subtract out the… these residuals, and, you know, they will make a difference. So things like that.
Wilson, Tom J.: It was more just a plug for… for friends, more than anything.
Wilson, Tom J.: If anyone was, interested in astronometry from an ISSE point of view.
Wilson, Tom J.: Or are you all galaxy people and don't care where your things are, so long as they're uniformly distributed in the sky?
Redwood C/D - 6504984451: Okay, thank you, this is really interesting. I…
Redwood C/D - 6504984451: People just be impressed that this session has covered so many different areas, like, there's… there's…
Redwood C/D - 6504984451: The different areas of science we can do, also how the data is processed, the intermediate
Redwood C/D - 6504984451: But fundamental data products.
Redwood C/D - 6504984451: images versus… Things that you can put in catalogs.
Redwood C/D - 6504984451: All… all of it has come up, so, we're out of time.
Wilson, Tom J.: So, if you don'
Wilson, Tom J.: at the bottom, you'll find my email address, you can email me if you… if you want to be friends.
Redwood C/D - 6504984451: Also, yeah…
Redwood C/D - 6504984451: through the ISSC, all things are possible. You can follow up on Slack, we can… we can keep the discussion going.
Redwood C/D - 6504984451: I just wanted to bring up a quick point with Sara Bonito, who wanted to be here but could not be because timings, she's joining virtually. But she faced one problem where she found some NEN measurements in RubenLSST data, specifically when looking at the magnitudes for Carina Nebula.
Redwood C/D - 6504984451: And she just wanted to raise that, and maybe, I don't know if others have seen that problems or experiencing that, but if you have, if that's something on your mind, I think that's also something to think about.
Redwood C/D - 6504984451: Okay, so Sarah's not here. Yeah, she's not here, she just asked me to sort of raise this. If you can, forward it along to ISSC, ask the ISSC, if that's a good place to continue the discussion.
Redwood C/D - 6504984451: Cool.
Redwood C/D - 6504984451: Thank you, everyone, but especially Sid and Tom for filling in as, Co-facilitators.
Redwood C/D - 6504984451: Governor's face.