Rubin2020_PCW slack archives day1-mon-slot2b-algorithms-ws 2020-07-15---2020-08-11

Wed 2020-07-15 03:06PM

@Ranpal (she/her/hers) has joined the channel

Thu 2020-07-16 01:18PM

@Melissa Graham has joined the channel

Wed 2020-07-22 11:56AM

@Jeff Carlin has joined the channel

Fri 2020-07-24 04:57PM

@Colin Slater has joined the channel

Colin Slater Fri 2020-07-24 04:58PM

@Colin Slater set the channel topic: Algorithms workshop follow-up session.

Fri 2020-07-31 03:02AM

@Ilaria Musella has joined the channel

Fri 2020-07-31 08:31AM

@Robert Szabo has joined the channel

Mon 2020-08-03 11:38AM

@K-T Lim has joined the channel

Mon 2020-08-03 07:27PM

@Ian Sullivan (he/him) has joined the channel

Tue 2020-08-04 02:31PM

@Adam Thornton has joined the channel

Tue 2020-08-04 07:13PM

@Arun Kannawadi (he/him) has joined the channel

Wed 2020-08-05 02:20AM

@leanne has joined the channel

Wed 2020-08-05 04:51AM

@Tom J Wilson has joined the channel

Wed 2020-08-05 11:30AM

@Eduardo Gonzalez Solares has joined the channel

Wed 2020-08-05 06:14PM

@Markus Rabus has joined the channel

Wed 2020-08-05 10:42PM

@Luis Pineda has joined the channel

Thu 2020-08-06 12:35PM

@parejkoj has joined the channel

Fri 2020-08-07 12:16AM

@mrawls has joined the channel

Melissa Graham Fri 2020-08-07 02:11PM

@Melissa Graham set the channel topic: Algorithms workshop follow-up session.
<https://project.lsst.org/meetings/rubin2020/agenda/session/rubin-algorithms-workshop-follow-answers-community-questions>

Sat 2020-08-08 12:06AM

@Lee Kelvin has joined the channel

Sun 2020-08-09 02:39PM

@John Swinbank has joined the channel

Sun 2020-08-09 07:54PM

@rmandelb has joined the channel

leanne Sun 2020-08-09 10:01PM

Reminder: this session is on Monday!
Time:_
07:30 HST - 10:30 PT - 13:30 EDT - 19:30 CEST - 03:30 AET +1
Zoom Meeting ID:_
943 9906 9251
Zoom Link:_
https://stanford.zoom.us/j/94399069251
Password:_ 257500

leanne Sun 2020-08-09 11:33PM

@leanne set the channel topic: Algorithms workshop follow-up Q&A session.
<https://project.lsst.org/meetings/rubin2020/agenda/session/rubin-algorithms-workshop-follow-answers-community-questions>

Mon 2020-08-10 03:41AM

@Naoki Yasuda has joined the channel

Mon 2020-08-10 06:43AM

@quasar_panda has joined the channel

Robert Lupton Mon 2020-08-10 01:05PM

Nice to see my friends drifting in (some in the middle of the night)

Chris Walter Mon 2020-08-10 01:27PM

Weirdly, I didn't see this channel in the sidebar and had to click on a link in the schedule to get to it..

Robert Lupton Mon 2020-08-10 01:27PM: It was in my sidebar
Morgan Schmitz Mon 2020-08-10 01:28PM: I had to manually add it, but I think that's intentional? You are only added to plenary channels at first, so you can pick only those corresponding to the breakouts you plan on attending?
mrawls Mon 2020-08-10 01:28PM: All attendees are automatically in all the plenary channels, but can pick and choose between breakout channels (there's only 53 of us in here right now vs. over 600 in the others)
Tom J Wilson Mon 2020-08-10 01:28PM: By default I think most of us are only added to plenary-level sessions by default; to see all options and join parallel sessions et al. click the "plus" symbol by the "Channels" at the top of the list of channels and join all extra channels
Nicholas Walton Mon 2020-08-10 01:28PM: for most of the sessions you need to specifically add the channel - you only get the std set automatically so to speak - those that you have been pre-added to by the slack channel admins.
Chris Walter Mon 2020-08-10 01:28PM: Ah. OK thanks.

K-T Lim Mon 2020-08-10 01:28PM

I think you only get the plenaries added by default unless you're a presenter/owner/panel member; everything else is self-signup.

Robert Lupton Mon 2020-08-10 01:30PM

The link is also in the google-calendar which is pinned to #general (don't forget to say thank you if you subscribe)

Merlin Mon 2020-08-10 01:38PM

@Robert Lupton your internet is problematically bad. Could you tether a cell phone perhaps?

Paul Price Mon 2020-08-10 01:38PM

Or @Jim Bosch (he/him) could present?

leanne Mon 2020-08-10 01:39PM

Much better

mwv Mon 2020-08-10 01:43PM

Thanks, @Robert Lupton : That proper motion discussion and plan to consider re-running a small area of DC2 with proper motion as a test make sense.

mwv Mon 2020-08-10 01:43PM

Thanks, @Yusra AlSayyad . The 3 levels of participation and start date of "now" are exactly helpful!

Mon 2020-08-10 01:43PM

@Yusra AlSayyad has joined the channel

Robert Lupton Mon 2020-08-10 01:44PM

We had all sorts of fun improving LSB science (to c. 29-30 mag/arcsec^2) in HSC and ruining weak lensing

rmandelb Mon 2020-08-10 01:48PM: Yes, that was lots of fun. Speaking as a weak lenser. :slightly_smiling_face:

Robert Lupton Mon 2020-08-10 01:47PM

There's some chat on #dm-coadds over on LSST slack

mwv Mon 2020-08-10 01:48PM

@Jim Bosch (he/him) What's the trade-off of having to think about rejecting single images with edges/mask when N~20 where the relative impact is significant but so is the loss of S/N, as compared to rejecting single images with edges/masks when N~200 where the relative impact is small in both PSF and S/N?
What's the scaling of needing to have a well-controlled PSF vs. depth?

Nicholas Walton Mon 2020-08-10 01:48PM

@leanne - Quick QUESTION ... and perhaps this will be covered in another session - but what is the current timeline for end to end testing of the DM system? How much simulated survey data will you use when testing the complete DM system needed to support DP0.1/DP0.2?

Colin Slater Mon 2020-08-10 01:52PM: I see this and will make sure you get an answer
Nicholas Walton Mon 2020-08-10 01:52PM: @Colin Slater thnx
William O'Mullane Mon 2020-08-10 01:52PM: you can also have a look at http://LDM-503.lsst.io
William O'Mullane Mon 2020-08-10 01:53PM: Though it does not cover data volumes
Nicholas Walton Mon 2020-08-10 01:55PM: @William O'Mullane I was wondering if the timeline has been updated (since the last release of the test plan doc) for the operational rehearsals?
William O'Mullane Mon 2020-08-10 02:02PM: slightly but the IDF is intended to allow is stay on track.
William O'Mullane Mon 2020-08-10 02:03PM: Your volume question then depends ... we have not commited yet to how much we will reprocess in DP0.2.
William O'Mullane Mon 2020-08-10 02:04PM: For DP0.1 - serving we are still pending MOUs.
Nicholas Walton Mon 2020-08-10 02:14PM: @William O'Mullane quick sanity check ... what was the outcome of LDM-503-09 (OpR for comm 1)
leanne Mon 2020-08-10 02:39PM: Thanks @William O'Mullane @Colin Slater for fielding Nic's question
William O'Mullane Mon 2020-08-10 03:03PM: @Nicholas Walton https://dmtn-119.lsst.io/
Nicholas Walton Mon 2020-08-10 03:54PM: @William O'Mullane Thanks. I guess you haven't written up the more recent OpR? I have a few followup q's - best via email.
William O'Mullane Mon 2020-08-10 04:23PM: there is an empty document with the title in it ..
Nicholas Walton Mon 2020-08-10 04:23PM: :smile:
William O'Mullane Mon 2020-08-10 04:23PM: dmtn-159

brant Mon 2020-08-10 01:49PM

Where is the best place to look for details on the coaddition methods to be used by the DM system?

brant Mon 2020-08-10 01:52PM: I did not register for the algorithms workshop and cannot seem to access the talks posted there.
Jeff Carlin Mon 2020-08-10 01:52PM: The (possibly outdated/incomplete) Tech Note that Jim referred to is at https://dmtn-015.lsst.io/
Robert Lupton Mon 2020-08-10 01:52PM: @leanne : did you make the algo workshop talks available to PCW attendees?
mwv Mon 2020-08-10 01:53PM: @brant I understand you're saying you can't, but just to double-check, can you access this direct link:
https://project.lsst.org/meetings/law/sites/lsst.org.meetings.law/files/Building%20and%20using%20coadds.pdf
leanne Mon 2020-08-10 01:53PM: Yes - they are all linked from the workshop webpage
brant Mon 2020-08-10 01:55PM: The Agenda for Science Sessions was not rendering in my browser correctly, but I reloaded the page and can now access the PDFs. Thank you.

mwv Mon 2020-08-10 01:52PM

Thanks, @Jim Bosch (he/him)

Robert Lupton Mon 2020-08-10 01:52PM

@Jim Bosch (he/him) : how much discussion of chi^2 coadds did your talk have?

Jim Bosch (he/him) Mon 2020-08-10 01:53PM: I believe I mentioned them, but I'm sure I didn't go into any of the formalism or make any definitive statements about whether we'd use them.
Robert Lupton Mon 2020-08-10 01:54PM: The context here is that I've been thinking quite hard about the tradeoffs, and I think that we'll probably be making more use of them than we might have once thought

Meg Schwamb Mon 2020-08-10 01:57PM

I have a general question. It seems that all the questions submitted were from DESC. Perhaps other science collaborations are not as advanced in their understanding of Rubin algorithms. Where is the best place for beginners to dive in on Rubin DM algorithms and where is the best place to keep up to date on updates/progress?

mwv Mon 2020-08-10 01:58PM: For the record, I believe that @Eric Gawiser submitted his question as a member of the Galaxy Science Collaboration.
mwv Mon 2020-08-10 01:58PM: But that distinction just emphasizes your point. @Meg Schwamb
[edited: Sorry, @Meg Schwamb , I auto-tabbed-completed incorrectly]
Nicholas Walton Mon 2020-08-10 01:59PM: This probably reflects the fact that the DESC group is better funded - so more people/time to work on Rubin/LSST prep.
mwv Mon 2020-08-10 02:01PM: And has a more unified goal. Sure, supernovae and weak lensing, but the reason we care about them is the same. So it's somewhat easier to agree on what the important question are.
Colin Slater Mon 2020-08-10 02:02PM: We have one more pre-organized question, I'll bring this up after that
Eric Gawiser Mon 2020-08-10 02:03PM: The organizers of the Rubin Algorithms workshop did a great job making all of the talks (slides and videos) available to the LSST community. I suggest those as a way to get up to speed for people who were not able to attend the original workshop. See https://project.lsst.org/meetings/law/content/registration-opening-soon
mwv Mon 2020-08-10 02:04PM: The workshop slides and talks were indeed excellent.
Yusra AlSayyad Mon 2020-08-10 02:04PM: We were hoping to hear followup questions about the algorithms discussed in the algorithms workshop. @mwv relaying a lot of questions about that which was not discussed. I provided a list of those issues that we would not discuss in my intro to pipelines talk. lastish slide
mwv Mon 2020-08-10 02:05PM: But I agree with @Nicholas Walton 's point that if an area of science doesn't get the funding to focus on even spending the time to think and digest this stuff, it's hard to make contributions in the same way.
Yusra AlSayyad Mon 2020-08-10 02:09PM: All of these topics (second page of timelines) for example were explicitly NOT discussed in the algorithms workshop
Eric Gawiser Mon 2020-08-10 02:11PM: One interpretation of that is that everything covered in the algorithms workshop was wonderfully clear :slightly_smiling_face: but that some of the topics that were not discussed there are still of great interest...
Jim Bosch (he/him) Mon 2020-08-10 02:23PM: It's also think it's often fine for a science collaboration to not be following algorithm details to the extent DESC is. I think there are many science cases where the relevant algorithms are well-understood and are not likely to be a limiting factor in doing science with Rubin data, and many more where the algorithms are mostly well understood and a much smaller amount of interaction with DM on them is probably sufficient. DESC is unusual (I'm not saying unique!) in having many science cases for which algorithms are definitely going to be a limiting factor and a lot of close interaction is warranted.
mwv Mon 2020-08-10 02:27PM: I think one challenge is that DESC has science cases and approaches that have been used in previous large-scale surveys. Thus there are people used to thinking about the algorithms in the context of large-scale surveys. I think the HSC expertise that influences DM is also largely based around large-scale cosmologyish surveys.
mwv Mon 2020-08-10 02:28PM: But the truly revolutionary part of Rubin Observatory LSST will be taking all of the the science cases that previously had been small, bespoke, limited N affairs, and realizing them in the context of a large-scale survey.
mwv Mon 2020-08-10 02:29PM: I think it's much harder to make that connection for reasons of expertise, perspective of various people involved, and funding and funding models (as noted above). But I think that's where there's great potential.
mwv Mon 2020-08-10 02:29PM: (Yes, there are stellar variability, transients, and solar system scientists in DM. I didn't mean to imply that there was no internal expertise in some of these other areas.)

Michael Strauss (he/his) Mon 2020-08-10 01:57PM

Whose responsibility will it be to make a photo-z algorithm robust for LSST data? That is, an algorithm is not necessarily the same thing as making code that will work in the Rubin framework for ~10^10 galaxies.

Surhud More Mon 2020-08-10 01:58PM

Is there a possibility to include more than one phot-z algorithm?

Dave Young (QUB, UK) Mon 2020-08-10 02:01PM

quick fix for after the session - hyperlinks seem to have been stripped from the slide-deck PDF. Thanks

Will Clarkson Mon 2020-08-10 02:02PM

Question I believe for @Colin Slater (I think maybe for the "additional questions" section of this session?) - can you update us please on the status and plans for crowded field stellar photometry and/or astrometry in DM?

jgizis Mon 2020-08-10 02:21PM: Thanks for the update @Colin Slater
Will Clarkson Mon 2020-08-10 02:23PM: Colin has a nice talk in the 2020 March workshop. Talk link: https://project.lsst.org/meetings/law/sites/lsst.org.meetings.law/files/Crowded%20Field%20Processing%20-%20Colin%20Slater.pdf

It references the following doc: https://dmtn-129.lsst.io/
Will Clarkson Mon 2020-08-10 02:44PM: Thanks, @Colin Slater !!
Colin Slater Mon 2020-08-10 02:49PM: Thanks for the good quesiton

Robert Lupton Mon 2020-08-10 02:04PM

I think that the "photo-z pdf space" question isn't likely to be a problem. We only need to characterise each cell in colour-colour-...colour (or flux-colour-colour) space, and that's a significantly smaller count than all the faint fuzzy objects.

mwv Mon 2020-08-10 02:05PM: I think that has to be right as well.

Chris Walter Mon 2020-08-10 02:07PM

What is GAaP implementation?

Arun Kannawadi (he/him) Mon 2020-08-10 02:14PM: GAaP stands for Gaussian Aperture and PSF. It involves Gaussianizing the PSFs and then using a Gaussian aperture (instead of top-hat) for measuring photometry. The aperture+PSF is designed to be the same across all bands, so that you measure consistent colors.

mwv Mon 2020-08-10 02:09PM

@Jim Bosch (he/him) Apologies for putting you on the spot about this, but I didn't actual dates in the answers on the first slide of "Q6: Timelines". Is it reasonable to expect that those date estimates will be made available to Science Collaborations?

mwv Mon 2020-08-10 02:09PM

(I appreciate that there were dates for the topics discussed on the second slide of Q6: Timelines.)

Keith Bechtol Mon 2020-08-10 02:15PM

One option for people learning about how to use the Stack is the Stack Club Course and associated Stack Club
https://github.com/LSSTScienceCollaborations/StackClubCourse

Keith Bechtol Mon 2020-08-10 02:18PM: Here's the main Stack Club repo (with many additional tutorial notebooks)
https://github.com/LSSTScienceCollaborations/StackClub
Keith Bechtol Mon 2020-08-10 02:20PM: On the main LSSTC Slack workspace, this is discussed on #stack-club

https://lsstc.slack.com/archives/C9YRAS4HM
Tom Loredo Mon 2020-08-10 02:20PM: My recollection is that the Stack Club course (this one, or perhaps a shorter one) filled up immediately. Perhaps you need to allocate more resources to this, or offer it more often.
Keith Bechtol Mon 2020-08-10 02:20PM: Yes, there was far more demand than slots available
Tom Loredo Mon 2020-08-10 02:20PM: PS: Thanks for posting the link to the course on Chat; I didn't know about that!

Tom J Wilson Mon 2020-08-10 02:17PM

Where would we go if we do want incredibly technical details instead of these high-level overviews?

leanne Mon 2020-08-10 02:19PM: The Data Products Definition Document : http://ls.st/LSE-163 and the Rubin Science Pipelines Design document: http://ls.st/LDM-151

Robert Lupton Mon 2020-08-10 02:17PM

One slightly-out-of-date place is the Bosch et al. HSC pipeline paper

Robert Lupton Mon 2020-08-10 02:18PM

Then asking questions on http://community.lsst.org should get you answers

Michael Strauss (he/his) Mon 2020-08-10 02:19PM: The direct link to the paper Robert cites is https://ui.adsabs.harvard.edu/abs/2018PASJ...70S...5B/abstract

Robert Lupton Mon 2020-08-10 02:18PM

.... and the code is all public on github!

John Swinbank Mon 2020-08-10 02:19PM

...and documented at http://pipelines.lsst.io . But of course the code and the documentation are still a work in progress.

Amanda Bauer Mon 2020-08-10 02:20PM: is LSE-163 (DPDD) doc not on http://lsst.io ? a quick search did not find it...
Robert Lupton Mon 2020-08-10 02:20PM: http://ls.st/dpdd works
Merlin Mon 2020-08-10 02:21PM: As does http://ls.st/lse-163 for the general case.
Robert Lupton Mon 2020-08-10 02:21PM: (not that that addresses Amanda's point)
John Swinbank Mon 2020-08-10 02:22PM: To actually answer the question - I see LSE-163 on http://www.lsst.io . But I get weird redirects trying to link to it. Let's follow up offline, @Amanda Bauer .
Simon Krughoff Mon 2020-08-10 02:23PM: It's also the last document in my search list.
Simon Krughoff Mon 2020-08-10 02:23PM: Maybe it's because it is a LaTeX document?
K-T Lim Mon 2020-08-10 02:34PM: Works fine for me.
Simon Krughoff Mon 2020-08-10 02:34PM: What do you mean?
K-T Lim Mon 2020-08-10 02:34PM: I mean that it shows up, in proper sequence, and the link works.
K-T Lim Mon 2020-08-10 02:35PM: (I have "Sort by ID" in the search results)
Simon Krughoff Mon 2020-08-10 02:35PM: What was your search term?
K-T Lim Mon 2020-08-10 02:35PM: Nothing.
K-T Lim Mon 2020-08-10 02:35PM: I selected "LSE" in the "Documents" section.
K-T Lim Mon 2020-08-10 02:36PM: If I type "LSE-163" as a search term, it shows up as the first result.
Simon Krughoff Mon 2020-08-10 02:36PM: Oh, I think the original question was how to use the search box.
K-T Lim Mon 2020-08-10 02:37PM: It does not show up if you search for "DPDD"
K-T Lim Mon 2020-08-10 02:37PM: But "Data Products Definition Document" works fine.

Tom J Wilson Mon 2020-08-10 02:20PM

Great, thanks all!

Peregrine Mon 2020-08-10 02:22PM

What is the relation between the project CF work and the TVS CF task force?

Will Clarkson Mon 2020-08-10 02:25PM: @Colin Slater I think peregrine's follow-on from my question here may be directed at you
Fred Moolekamp Mon 2020-08-10 02:29PM: I've been working with the TVS CF task force to run scarlet on NGC-6569 and compare with results that @Massimo Dall'Ora has obtained.
Will Clarkson Mon 2020-08-10 02:31PM: Thanks for the update, @Fred Moolekamp !!
Massimo Dall'Ora Mon 2020-08-10 03:59PM: To extend the Colin's answer: both approaches are forced photometry based. In the LSST pipeline, individual images are fitted with the positions coming from a stack. In the TVS CF, images are simultaneously fitted, to cross-match positions and fluxes. Of course, the latter is much more time-consuming, and it is eventually being offered as in-kind contribution, provided that it reaches fainter magnitudes and/or higher accuracy.

jgizis Mon 2020-08-10 02:23PM

@Robert Lupton mentioned DM parallaxes and a model for faint stars. This might be too far in the weeds for the video but I'd like to hear about about the plans as they are developed.

jgizis Mon 2020-08-10 02:29PM: Thanks!
Robert Lupton Mon 2020-08-10 02:32PM: @Clare Saunders is doing the gbdes experiments --- sorry, Clare, I should have given you credit

Andrés Plazas (he/him/his) Mon 2020-08-10 02:24PM