Rubin2020_PCW slack archives day1-mon-slot2b-algorithms-ws 2020-07-15---2020-08-11

Wed 2020-07-15 03:06PM
@Ranpal (she/her/hers) has joined the channel
Thu 2020-07-16 01:18PM
@Melissa Graham has joined the channel
Wed 2020-07-22 11:56AM
@Jeff Carlin has joined the channel
Fri 2020-07-24 04:57PM
@Colin Slater has joined the channel
Colin Slater Fri 2020-07-24 04:58PM
@Colin Slater set the channel topic: Algorithms workshop follow-up session.
Fri 2020-07-31 03:02AM
@Ilaria Musella has joined the channel
Fri 2020-07-31 08:31AM
@Robert Szabo has joined the channel
Mon 2020-08-03 11:38AM
@K-T Lim has joined the channel
Mon 2020-08-03 07:27PM
@Ian Sullivan (he/him) has joined the channel
Tue 2020-08-04 02:31PM
@Adam Thornton has joined the channel
Tue 2020-08-04 07:13PM
@Arun Kannawadi (he/him) has joined the channel
Wed 2020-08-05 02:20AM
@leanne has joined the channel
Wed 2020-08-05 04:51AM
@Tom J Wilson has joined the channel
Wed 2020-08-05 11:30AM
@Eduardo Gonzalez Solares has joined the channel
Wed 2020-08-05 06:14PM
@Markus Rabus has joined the channel
Wed 2020-08-05 10:42PM
@Luis Pineda has joined the channel
Thu 2020-08-06 12:35PM
@parejkoj has joined the channel
Fri 2020-08-07 12:16AM
@mrawls has joined the channel
Melissa Graham Fri 2020-08-07 02:11PM
@Melissa Graham set the channel topic: Algorithms workshop follow-up session.
Sat 2020-08-08 12:06AM
@Lee Kelvin has joined the channel
Sun 2020-08-09 02:39PM
@John Swinbank has joined the channel
Sun 2020-08-09 07:54PM
@rmandelb has joined the channel
leanne Sun 2020-08-09 10:01PM
Reminder: this session is on Monday!
07:30 HST - 10:30 PT - 13:30 EDT - 19:30 CEST - 03:30 AET +1
Zoom Meeting ID:_
943 9906 9251
Zoom Link:_
Password:_ 257500
leanne Sun 2020-08-09 11:33PM
@leanne set the channel topic: Algorithms workshop follow-up Q&amp;A session.
Mon 2020-08-10 03:41AM
@Naoki Yasuda has joined the channel
Mon 2020-08-10 06:43AM
@quasar_panda has joined the channel
Robert Lupton Mon 2020-08-10 01:05PM
Nice to see my friends drifting in (some in the middle of the night)
Chris Walter Mon 2020-08-10 01:27PM
Weirdly, I didn't see this channel in the sidebar and had to click on a link in the schedule to get to it..
Robert Lupton Mon 2020-08-10 01:27PM
It was in my sidebar
Morgan Schmitz Mon 2020-08-10 01:28PM
I had to manually add it, but I think that's intentional? You are only added to plenary channels at first, so you can pick only those corresponding to the breakouts you plan on attending?
mrawls Mon 2020-08-10 01:28PM
All attendees are automatically in all the plenary channels, but can pick and choose between breakout channels (there's only 53 of us in here right now vs. over 600 in the others)
Tom J Wilson Mon 2020-08-10 01:28PM
By default I think most of us are only added to plenary-level sessions by default; to see all options and join parallel sessions et al. click the "plus" symbol by the "Channels" at the top of the list of channels and join all extra channels
Nicholas Walton Mon 2020-08-10 01:28PM
for most of the sessions you need to specifically add the channel - you only get the std set automatically so to speak - those that you have been pre-added to by the slack channel admins.
Chris Walter Mon 2020-08-10 01:28PM
Ah. OK thanks.
K-T Lim Mon 2020-08-10 01:28PM
I think you only get the plenaries added by default unless you're a presenter/owner/panel member; everything else is self-signup.
Robert Lupton Mon 2020-08-10 01:30PM
The link is also in the google-calendar which is pinned to #general (don't forget to say thank you if you subscribe)
Merlin Mon 2020-08-10 01:38PM
@Robert Lupton your internet is problematically bad. Could you tether a cell phone perhaps?
Paul Price Mon 2020-08-10 01:38PM
Or @Jim Bosch (he/him) could present?
leanne Mon 2020-08-10 01:39PM
Much better
mwv Mon 2020-08-10 01:43PM
Thanks, @Robert Lupton : That proper motion discussion and plan to consider re-running a small area of DC2 with proper motion as a test make sense.
mwv Mon 2020-08-10 01:43PM
Thanks, @Yusra AlSayyad . The 3 levels of participation and start date of "now" are exactly helpful!
Mon 2020-08-10 01:43PM
@Yusra AlSayyad has joined the channel
Robert Lupton Mon 2020-08-10 01:44PM
We had all sorts of fun improving LSB science (to c. 29-30 mag/arcsec^2) in HSC and ruining weak lensing
rmandelb Mon 2020-08-10 01:48PM
Yes, that was lots of fun. Speaking as a weak lenser. :slightly_smiling_face:
Robert Lupton Mon 2020-08-10 01:47PM
There's some chat on #dm-coadds over on LSST slack
mwv Mon 2020-08-10 01:48PM
@Jim Bosch (he/him) What's the trade-off of having to think about rejecting single images with edges/mask when N~20 where the relative impact is significant but so is the loss of S/N, as compared to rejecting single images with edges/masks when N~200 where the relative impact is small in both PSF and S/N?
What's the scaling of needing to have a well-controlled PSF vs. depth?
Nicholas Walton Mon 2020-08-10 01:48PM
@leanne - Quick QUESTION ... and perhaps this will be covered in another session - but what is the current timeline for end to end testing of the DM system? How much simulated survey data will you use when testing the complete DM system needed to support DP0.1/DP0.2?
Colin Slater Mon 2020-08-10 01:52PM
I see this and will make sure you get an answer
Nicholas Walton Mon 2020-08-10 01:52PM
@Colin Slater thnx
William O'Mullane Mon 2020-08-10 01:52PM
you can also have a look at
William O'Mullane Mon 2020-08-10 01:53PM
Though it does not cover data volumes
Nicholas Walton Mon 2020-08-10 01:55PM
@William O'Mullane I was wondering if the timeline has been updated (since the last release of the test plan doc) for the operational rehearsals?
William O'Mullane Mon 2020-08-10 02:02PM
slightly but the IDF is intended to allow is stay on track.
William O'Mullane Mon 2020-08-10 02:03PM
Your volume question then depends ... we have not commited yet to how much we will reprocess in DP0.2.
William O'Mullane Mon 2020-08-10 02:04PM
For DP0.1 - serving we are still pending MOUs.
Nicholas Walton Mon 2020-08-10 02:14PM
@William O'Mullane quick sanity check ... what was the outcome of LDM-503-09 (OpR for comm 1)
leanne Mon 2020-08-10 02:39PM
Thanks @William O'Mullane @Colin Slater for fielding Nic's question
William O'Mullane Mon 2020-08-10 03:03PM
@Nicholas Walton
Nicholas Walton Mon 2020-08-10 03:54PM
@William O'Mullane Thanks. I guess you haven't written up the more recent OpR? I have a few followup q's - best via email.
William O'Mullane Mon 2020-08-10 04:23PM
there is an empty document with the title in it ..
Nicholas Walton Mon 2020-08-10 04:23PM
William O'Mullane Mon 2020-08-10 04:23PM
brant Mon 2020-08-10 01:49PM
Where is the best place to look for details on the coaddition methods to be used by the DM system?
brant Mon 2020-08-10 01:52PM
I did not register for the algorithms workshop and cannot seem to access the talks posted there.
Jeff Carlin Mon 2020-08-10 01:52PM
The (possibly outdated/incomplete) Tech Note that Jim referred to is at
Robert Lupton Mon 2020-08-10 01:52PM
@leanne : did you make the algo workshop talks available to PCW attendees?
mwv Mon 2020-08-10 01:53PM
@brant I understand you're saying you can't, but just to double-check, can you access this direct link:
leanne Mon 2020-08-10 01:53PM
Yes - they are all linked from the workshop webpage
brant Mon 2020-08-10 01:55PM
The Agenda for Science Sessions was not rendering in my browser correctly, but I reloaded the page and can now access the PDFs. Thank you.
mwv Mon 2020-08-10 01:52PM
Thanks, @Jim Bosch (he/him)
Robert Lupton Mon 2020-08-10 01:52PM
@Jim Bosch (he/him) : how much discussion of chi^2 coadds did your talk have?
Jim Bosch (he/him) Mon 2020-08-10 01:53PM
I believe I mentioned them, but I'm sure I didn't go into any of the formalism or make any definitive statements about whether we'd use them.
Robert Lupton Mon 2020-08-10 01:54PM
The context here is that I've been thinking quite hard about the tradeoffs, and I think that we'll probably be making more use of them than we might have once thought
Meg Schwamb Mon 2020-08-10 01:57PM
I have a general question. It seems that all the questions submitted were from DESC. Perhaps other science collaborations are not as advanced in their understanding of Rubin algorithms. Where is the best place for beginners to dive in on Rubin DM algorithms and where is the best place to keep up to date on updates/progress?
mwv Mon 2020-08-10 01:58PM
For the record, I believe that @Eric Gawiser submitted his question as a member of the Galaxy Science Collaboration.
mwv Mon 2020-08-10 01:58PM
But that distinction just emphasizes your point. @Meg Schwamb
[edited: Sorry, @Meg Schwamb , I auto-tabbed-completed incorrectly]
Nicholas Walton Mon 2020-08-10 01:59PM
This probably reflects the fact that the DESC group is better funded - so more people/time to work on Rubin/LSST prep.
mwv Mon 2020-08-10 02:01PM
And has a more unified goal. Sure, supernovae and weak lensing, but the reason we care about them is the same. So it's somewhat easier to agree on what the important question are.
Colin Slater Mon 2020-08-10 02:02PM
We have one more pre-organized question, I'll bring this up after that
Eric Gawiser Mon 2020-08-10 02:03PM
The organizers of the Rubin Algorithms workshop did a great job making all of the talks (slides and videos) available to the LSST community. I suggest those as a way to get up to speed for people who were not able to attend the original workshop. See
mwv Mon 2020-08-10 02:04PM
The workshop slides and talks were indeed excellent.
Yusra AlSayyad Mon 2020-08-10 02:04PM
We were hoping to hear followup questions about the algorithms discussed in the algorithms workshop. @mwv relaying a lot of questions about that which was not discussed. I provided a list of those issues that we would not discuss in my intro to pipelines talk. lastish slide
mwv Mon 2020-08-10 02:05PM
But I agree with @Nicholas Walton 's point that if an area of science doesn't get the funding to focus on even spending the time to think and digest this stuff, it's hard to make contributions in the same way.
Yusra AlSayyad Mon 2020-08-10 02:09PM
All of these topics (second page of timelines) for example were explicitly NOT discussed in the algorithms workshop
Eric Gawiser Mon 2020-08-10 02:11PM
One interpretation of that is that everything covered in the algorithms workshop was wonderfully clear :slightly_smiling_face: but that some of the topics that were not discussed there are still of great interest...
Jim Bosch (he/him) Mon 2020-08-10 02:23PM
It's also think it's often fine for a science collaboration to not be following algorithm details to the extent DESC is. I think there are many science cases where the relevant algorithms are well-understood and are not likely to be a limiting factor in doing science with Rubin data, and many more where the algorithms are mostly well understood and a much smaller amount of interaction with DM on them is probably sufficient. DESC is unusual (I'm not saying unique!) in having many science cases for which algorithms are definitely going to be a limiting factor and a lot of close interaction is warranted.
mwv Mon 2020-08-10 02:27PM
I think one challenge is that DESC has science cases and approaches that have been used in previous large-scale surveys. Thus there are people used to thinking about the algorithms in the context of large-scale surveys. I think the HSC expertise that influences DM is also largely based around large-scale cosmologyish surveys.
mwv Mon 2020-08-10 02:28PM
But the truly revolutionary part of Rubin Observatory LSST will be taking all of the the science cases that previously had been small, bespoke, limited N affairs, and realizing them in the context of a large-scale survey.
mwv Mon 2020-08-10 02:29PM
I think it's much harder to make that connection for reasons of expertise, perspective of various people involved, and funding and funding models (as noted above). But I think that's where there's great potential.
mwv Mon 2020-08-10 02:29PM
(Yes, there are stellar variability, transients, and solar system scientists in DM. I didn't mean to imply that there was no internal expertise in some of these other areas.)
Michael Strauss (he/his) Mon 2020-08-10 01:57PM
Whose responsibility will it be to make a photo-z algorithm robust for LSST data? That is, an algorithm is not necessarily the same thing as making code that will work in the Rubin framework for ~10^10 galaxies.
Surhud More Mon 2020-08-10 01:58PM
Is there a possibility to include more than one phot-z algorithm?
Dave Young (QUB, UK) Mon 2020-08-10 02:01PM
quick fix for after the session - hyperlinks seem to have been stripped from the slide-deck PDF. Thanks
Will Clarkson Mon 2020-08-10 02:02PM
Question I believe for @Colin Slater (I think maybe for the "additional questions" section of this session?) - can you update us please on the status and plans for crowded field stellar photometry and/or astrometry in DM?
jgizis Mon 2020-08-10 02:21PM
Thanks for the update @Colin Slater
Will Clarkson Mon 2020-08-10 02:23PM
Colin has a nice talk in the 2020 March workshop. Talk link:

It references the following doc:
Will Clarkson Mon 2020-08-10 02:44PM
Thanks, @Colin Slater !!
Colin Slater Mon 2020-08-10 02:49PM
Thanks for the good quesiton
Robert Lupton Mon 2020-08-10 02:04PM
I think that the "photo-z pdf space" question isn't likely to be a problem. We only need to characterise each cell in colour-colour-...colour (or flux-colour-colour) space, and that's a significantly smaller count than all the faint fuzzy objects.
mwv Mon 2020-08-10 02:05PM
I think that has to be right as well.
Chris Walter Mon 2020-08-10 02:07PM
What is GAaP implementation?
Arun Kannawadi (he/him) Mon 2020-08-10 02:14PM
GAaP stands for Gaussian Aperture and PSF. It involves Gaussianizing the PSFs and then using a Gaussian aperture (instead of top-hat) for measuring photometry. The aperture+PSF is designed to be the same across all bands, so that you measure consistent colors.
mwv Mon 2020-08-10 02:09PM
@Jim Bosch (he/him) Apologies for putting you on the spot about this, but I didn't actual dates in the answers on the first slide of "Q6: Timelines". Is it reasonable to expect that those date estimates will be made available to Science Collaborations?
mwv Mon 2020-08-10 02:09PM
(I appreciate that there were dates for the topics discussed on the second slide of Q6: Timelines.)
Keith Bechtol Mon 2020-08-10 02:15PM
One option for people learning about how to use the Stack is the Stack Club Course and associated Stack Club
Keith Bechtol Mon 2020-08-10 02:18PM
Here's the main Stack Club repo (with many additional tutorial notebooks)
Keith Bechtol Mon 2020-08-10 02:20PM
On the main LSSTC Slack workspace, this is discussed on #stack-club
Tom Loredo Mon 2020-08-10 02:20PM
My recollection is that the Stack Club course (this one, or perhaps a shorter one) filled up immediately. Perhaps you need to allocate more resources to this, or offer it more often.
Keith Bechtol Mon 2020-08-10 02:20PM
Yes, there was far more demand than slots available
Tom Loredo Mon 2020-08-10 02:20PM
PS: Thanks for posting the link to the course on Chat; I didn't know about that!
Tom J Wilson Mon 2020-08-10 02:17PM
Where would we go if we do want incredibly technical details instead of these high-level overviews?
leanne Mon 2020-08-10 02:19PM
The Data Products Definition Document : and the Rubin Science Pipelines Design document:
Robert Lupton Mon 2020-08-10 02:17PM
One slightly-out-of-date place is the Bosch et al. HSC pipeline paper
Robert Lupton Mon 2020-08-10 02:18PM
Then asking questions on should get you answers
Michael Strauss (he/his) Mon 2020-08-10 02:19PM
The direct link to the paper Robert cites is
Robert Lupton Mon 2020-08-10 02:18PM
.... and the code is all public on github!
John Swinbank Mon 2020-08-10 02:19PM
...and documented at . But of course the code and the documentation are still a work in progress.
Amanda Bauer Mon 2020-08-10 02:20PM
is LSE-163 (DPDD) doc not on ? a quick search did not find it...
Robert Lupton Mon 2020-08-10 02:20PM works
Merlin Mon 2020-08-10 02:21PM
As does for the general case.
Robert Lupton Mon 2020-08-10 02:21PM
(not that that addresses Amanda's point)
John Swinbank Mon 2020-08-10 02:22PM
To actually answer the question - I see LSE-163 on . But I get weird redirects trying to link to it. Let's follow up offline, @Amanda Bauer .
Simon Krughoff Mon 2020-08-10 02:23PM
It's also the last document in my search list.
Simon Krughoff Mon 2020-08-10 02:23PM
Maybe it's because it is a LaTeX document?
K-T Lim Mon 2020-08-10 02:34PM
Works fine for me.
Simon Krughoff Mon 2020-08-10 02:34PM
What do you mean?
K-T Lim Mon 2020-08-10 02:34PM
I mean that it shows up, in proper sequence, and the link works.
K-T Lim Mon 2020-08-10 02:35PM
(I have "Sort by ID" in the search results)
Simon Krughoff Mon 2020-08-10 02:35PM
What was your search term?
K-T Lim Mon 2020-08-10 02:35PM
K-T Lim Mon 2020-08-10 02:35PM
I selected "LSE" in the "Documents" section.
K-T Lim Mon 2020-08-10 02:36PM
If I type "LSE-163" as a search term, it shows up as the first result.
Simon Krughoff Mon 2020-08-10 02:36PM
Oh, I think the original question was how to use the search box.
K-T Lim Mon 2020-08-10 02:37PM
It does not show up if you search for "DPDD"
K-T Lim Mon 2020-08-10 02:37PM
But "Data Products Definition Document" works fine.
Tom J Wilson Mon 2020-08-10 02:20PM
Great, thanks all!
Peregrine Mon 2020-08-10 02:22PM
What is the relation between the project CF work and the TVS CF task force?
Will Clarkson Mon 2020-08-10 02:25PM
@Colin Slater I think peregrine's follow-on from my question here may be directed at you
Fred Moolekamp Mon 2020-08-10 02:29PM
I've been working with the TVS CF task force to run scarlet on NGC-6569 and compare with results that @Massimo Dall'Ora has obtained.
Will Clarkson Mon 2020-08-10 02:31PM
Thanks for the update, @Fred Moolekamp !!
Massimo Dall'Ora Mon 2020-08-10 03:59PM
To extend the Colin's answer: both approaches are forced photometry based. In the LSST pipeline, individual images are fitted with the positions coming from a stack. In the TVS CF, images are simultaneously fitted, to cross-match positions and fluxes. Of course, the latter is much more time-consuming, and it is eventually being offered as in-kind contribution, provided that it reaches fainter magnitudes and/or higher accuracy.
jgizis Mon 2020-08-10 02:23PM
@Robert Lupton mentioned DM parallaxes and a model for faint stars. This might be too far in the weeds for the video but I'd like to hear about about the plans as they are developed.
jgizis Mon 2020-08-10 02:29PM
Robert Lupton Mon 2020-08-10 02:32PM
@Clare Saunders is doing the gbdes experiments --- sorry, Clare, I should have given you credit
Andrés Plazas (he/him/his) Mon 2020-08-10 02:24PM
Jim's paper:
Robert Lupton Mon 2020-08-10 02:25PM
Or if you're boycotting arXiv
Nicholas Walton Mon 2020-08-10 02:32PM
Lee Kelvin Mon 2020-08-10 02:32PM
Sara (Rosaria) Bonito Mon 2020-08-10 02:32PM
Tom J Wilson Mon 2020-08-10 02:32PM
Thanks everyone
Mon 2020-08-10 02:32PM
@Massimo Dall'Ora has joined the channel
Ranpal (she/her/hers) Tue 2020-08-11 08:31AM
The recording of the live session is here::