Rubin2020_PCW slack archives day4-thu-slot2a-alert-brokers 2020-07-15---2020-08-14

Wed 2020-07-15 02:57PM

@Ranpal (she/her/hers) has joined the channel

Thu 2020-07-16 01:19PM

@Melissa Graham has joined the channel

Wed 2020-07-22 11:58AM

@Jeff Carlin has joined the channel

Melissa Graham Fri 2020-07-24 06:10PM

@Melissa Graham set the channel topic: Thur Aug 13, 10:30-11:30am PDT
Zoom: <https://stanford.zoom.us/j/93955636098>
Chairs: Eric Bellm and Melissa Graham
Webpage (with agenda, links to slides): <https://project.lsst.org/meetings/rubin2020/agenda/session/community-alert-brokers>

Fri 2020-07-24 06:10PM

@Eric C. Bellm has joined the channel

Thu 2020-07-30 01:25AM

@Pkubanek has joined the channel

Thu 2020-07-30 03:19AM

@Christian Setzer has joined the channel

Fri 2020-07-31 03:04AM

@Ilaria Musella has joined the channel

Fri 2020-07-31 08:33AM

@Robert Szabo has joined the channel

Mon 2020-08-03 11:41AM

@K-T Lim has joined the channel

Wed 2020-08-05 07:45AM

@Tomás Müller has joined the channel

Wed 2020-08-05 02:11PM

@M Temple has joined the channel

Wed 2020-08-05 10:43PM

@Luis Pineda has joined the channel

Thu 2020-08-06 06:11PM

@Kara Ponder has joined the channel

Thu 2020-08-06 06:58PM

@Ignacio Reyes has joined the channel

Thu 2020-08-06 07:35PM

@Youtsumi has joined the channel

Fri 2020-08-07 12:19AM

@mrawls has joined the channel

Melissa Graham Fri 2020-08-07 02:35PM

@here , note that the Project, along with several broker teams, have made pre-view materials (e.g., short informative videos, slide decks) available for PCW attendees, and it's all on the https://project.lsst.org/meetings/rubin2020/agenda/session/community-alert-brokers .

Mon 2020-08-10 12:12AM

@Lauren Corlies has joined the channel

Mon 2020-08-10 06:44AM

@quasar_panda has joined the channel

Mon 2020-08-10 01:12PM

@Christopher Waters has joined the channel

Mon 2020-08-10 01:12PM

@Jeyhan Kartaltepe has joined the channel

Melissa Graham Wed 2020-08-12 11:56PM

Community Alert Brokers
In this session the Project will provide a brief update on the broker selection process, and many broker teams have provided informative materials and updates for the science community. Most of this session's time will be dedicated to Q&A between the Project, broker teams, and the broader science community. Everyone is welcome to attend this session and ask questions about brokering alerts!

Resources Available on the https://project.lsst.org/meetings/rubin2020/agenda/session/community-alert-brokers
a video presentation by Eric Bellm titled "Status of Alert Broker Selection and Sample Alerts" a variety of introductory video and slide presentations by alert broker teams, targeted for a general audience links to project documentation describing the alerts-related data products
Zoom Link: https://stanford.zoom.us/j/93955636098

Ranpal (she/her/hers) Thu 2020-08-13 07:05AM

https://stanford.zoom.us/j/93955636098?pwd=SGZqK3l6MmVaWVV1d3VvejZUc1VWdz09
Passcode: 257500

leanne Thu 2020-08-13 01:32PM

@parejkoj @Melissa Graham Maracon filling is buttercream :slightly_smiling_face:

parejkoj Thu 2020-08-13 01:33PM

Some of the ones in that image looked different. Does that imply they are just sandwiches, or does it make them layer cakes?

Simon Krughoff Thu 2020-08-13 01:34PM

A layer cake is a sandwich.

leanne Thu 2020-08-13 01:34PM

:exploding_head:

Matthew Graham Thu 2020-08-13 01:35PM

How about a savory macaron with Marmite as the filling

Adam Thornton Thu 2020-08-13 01:35PM

Oh no are we about to have a Taxonomy Of Sandwiches argument here too ? Why can I never escape these?

leanne Thu 2020-08-13 01:36PM

@Matthew Graham Even if that were vegemite, it would still be disgusting

Matthew Graham Thu 2020-08-13 01:36PM

Is a one piece of bread with something on it really a sandwich?

Adam Thornton Thu 2020-08-13 01:36PM

No, that's a Toast.

Matthew Graham Thu 2020-08-13 01:36PM

Or an open sandwich

Simon Krughoff Thu 2020-08-13 01:37PM

Is a pop tart a sandwich?

Adam Thornton Thu 2020-08-13 01:37PM

Look, this has been COMPREHENSIVELY DISCUSSED.

Adam Thornton Thu 2020-08-13 01:37PM

https://cuberule.com/

Simon Krughoff Thu 2020-08-13 01:38PM

I'm firmly in the structure rebel/ingredient rebel alignment.

Simon Krughoff Thu 2020-08-13 01:38PM

leanne Thu 2020-08-13 01:38PM

What did I start .....

Matthew Graham Thu 2020-08-13 01:39PM

What about the sort of thing that Shaggy and Scooby construct?

Adam Thornton Thu 2020-08-13 01:39PM

WHY? WHY did you start it?

parejkoj Thu 2020-08-13 01:39PM

Admittedly, I think it's my fault for missing seeing Melissa's macaron background every time I went into my office.

Adam Thornton Thu 2020-08-13 01:39PM

classic sandwich

Adam Thornton Thu 2020-08-13 01:40PM

structurally unstable perhaps

Adam Thornton Thu 2020-08-13 01:40PM

but a straight-up sandwich

Matthew Graham Thu 2020-08-13 01:40PM

https://en.wikipedia.org/wiki/Open_sandwich

Simon Krughoff Thu 2020-08-13 01:40PM

As long as you can pick it up, it's a sandwich.

Matthew Graham Thu 2020-08-13 01:41PM

So you don't use cutlery for a tartine?

Adam Thornton Thu 2020-08-13 01:41PM

So a SparcStation 20 is a sandwich but a PDP/11-23 isn't?

Emmanuel Gangler Thu 2020-08-13 01:41PM

I like this concept of "open sandwich" which looks like to me as a pair of tartines ...

Simon Krughoff Thu 2020-08-13 01:42PM

I'm not saying you have to pick it up.

Adam Thornton Thu 2020-08-13 01:42PM

But if you were Andre the Giant, the 11/23 would be a sandwich?

Simon Krughoff Thu 2020-08-13 01:42PM

Do you really want me to go deeper into my sandwich theory?

Simon Krughoff Thu 2020-08-13 01:42PM

E.g. buttered toast is not a sandwich but avocado toast is.

Matthew Graham Thu 2020-08-13 01:42PM

Does it include this: https://en.wikipedia.org/wiki/Ham_sandwich_theorem

Simon Krughoff Thu 2020-08-13 01:43PM

Mine isn't as mathematically rigorous.

Adam Thornton Thu 2020-08-13 01:44PM

dammit I don't want to be thinking about the Banach-Tarski paradox in the context of sandwiches when I am already hungry.

Matthew Graham Thu 2020-08-13 01:45PM

Or the Barbanel-Brams rotating-knife procedure

mrawls Thu 2020-08-13 01:39PM

Please do ask allllll your questions, I am here to demystify the acronyms and jargon :slightly_smiling_face: (and also pass along Qs to speakers!)

William O'Mullane Thu 2020-08-13 01:41PM: also https://www.lsst.org/scientists/glossary-acronyms

mrawls Thu 2020-08-13 01:40PM

Melissa's intro slide ICYMI

Tim Lister Thu 2020-08-13 01:42PM

Is there an equivalent slideor link to one for Solar System Objects (since they won't link together into DIAObjects as the coordinates change) ?

leanne Thu 2020-08-13 01:42PM: Not that I am aware of, it's a good idea to create one though.
John Swinbank Thu 2020-08-13 01:44PM: The basic principle is the same, though: we will have predicted positions for known SSObjects at the time of observations, and will associate against those and issue alerts.
mrawls Thu 2020-08-13 01:44PM: There will be SSObjects , as John describes, but I haven't seen a nice overview slide for them
John Swinbank Thu 2020-08-13 01:45PM: Sorry if I wasn't clear: the point is the slide is basically the same, substituting the word "SSObject" for "DIAObject". :slightly_smiling_face:
Tim Lister Thu 2020-08-13 01:49PM: Are they completely analogous given the that the timescales for "collapsing" DIASources into new DIA/SSObjects will be different based on the different linkage requirements?
John Swinbank Thu 2020-08-13 02:00PM: Alerts are issued for every DIASource. Within that alert packet, any association with a known SSObject or DIAObject will be included. If there is no known object to associate with, then a new DIAObject is created.

During the following day, we'll go back and do solar system processing for the previous night. At that point, we might discover that some object we created a new DIAObject for is actually a newly-discovered solar system object. That won't result in a new alert, but we'll record the corrected classification in our database and use it for future associations.
Tim Lister Thu 2020-08-13 02:30PM: Thanks @John Swinbank

Franz Bauer Thu 2020-08-13 01:43PM

Eric, can you outline what criteria/rubric will be used to select alert brokers past the second stage. I.e., what sort of basic/minimum cut will be made?

leanne Thu 2020-08-13 01:46PM: Note that the proposals will be evaluated by a panel designated by the LSST Science Advisory Committee (SAC).
leanne Thu 2020-08-13 01:47PM: DM staff will be ex-offico to this committee to provide policy and technical guidance
leanne Thu 2020-08-13 01:49PM: This committee has not yet been stood-up and the exact criteria not yet defined. Maximizing the scientific return from LSST, and reaching a wide and diverse community will be string drivers.
Eric C. Bellm Thu 2020-08-13 01:49PM: Note that http://ls.st/LDM-612 gives some evaluation criteria that the SAC committee may start from

Konstantin Malanchev Thu 2020-08-13 01:47PM

Q to @Eric C. Bellm pre-view video. Why not include light curve itself instead of its features into "minimal" packets? Half-year light curve is only ~40 observations x 4 bytes x 3 (time + flux + err) = 480 bytes of uncompressed data. And it could be even less if not include upper limits

Dave Young (QUB, UK) Thu 2020-08-13 01:47PM

Q: Is it still true that there will be 4-5 teams given access the the full alert stream? i.e. 4-5 official Rubin Alert Brokers

leanne Thu 2020-08-13 01:51PM: We are required to support at least 5 full streams to community brokers

Anais Möller Thu 2020-08-13 01:50PM

Q: will the EPO plan use APIs from brokers to get information? do you have an infrastructure already in mind?

Lauren Corlies Thu 2020-08-13 01:53PM: That is the working plan! We are actively hiring a backend developer at the moment that will ramp up this effort next year, including better defining our own infrastructure and how that could connect with the brokers.
Anais Möller Thu 2020-08-13 01:54PM: excellent, thanks!

Dave Young (QUB, UK) Thu 2020-08-13 01:50PM

Q: Which of candidate brokers are currently actively developing tools for solar-system (moving body) science?

Anais Möller Thu 2020-08-13 01:52PM

@Fink we are not yet doing this but we want to hear and collaborate with solar-system teams to make it happen!

Francisco Forster Thu 2020-08-13 01:52PM

ALeRCE is doing one stamp classification, where one of the classes is asteroid candidate. This can help get a cleaner sample for further analysis.

leanne Thu 2020-08-13 01:53PM

The https://docs.google.com/presentation/d/1Me5q46lLEJAh2YTkWjYs5amYKlaUyoHfvr2vRNgGFiA/edit#slide=id.p broker is dedicated to solar system alert processing. I believe they are actively working with ANTARES

David Trilling Thu 2020-08-13 01:54PM

@Dave Young (QUB, UK) i will talk about SNAPS in about 30 minutes. SNAPS == solar system notification alert processing system. we are 100% dedicated to solar system objects.

Rseaman Thu 2020-08-13 01:57PM

There is also a collaboration with the Minor Planet Center layered on the NEO confirmation page and other facilities.

Francisco Forster Thu 2020-08-13 04:41PM

@David Trilling , in figure 8 here: https://arxiv.org/pdf/2008.03309.pdf you can see that the unlabeled set classified as asteroid candidates based on image stamps tends to follow the ecliptic (note that this figure was produced from a random set of locations, in the final paper we will show the full distribution). This classifications can be provided as input to your classifier if needed.

Konstantin Malanchev Thu 2020-08-13 01:51PM

Q: https://dmtn-149.lsst.io/#broker-hardware says that only one host is enough to receive the alert stream._Does bandwidth budget allow a broker to use two or three independent nodes to receive a stream to increase broker's fault-tolerance?

spencer nelson Thu 2020-08-13 01:53PM: DMTN-149 isn't quite normative on how the alert stream should work in production. That section is intended to say "When designing an alert stream simulator , we should strive to make something that can run on one host." Certainly, in the real world - that is, not using a simulator - using multiple hosts to consume the stream is probably good
Konstantin Malanchev Thu 2020-08-13 01:54PM: Thank you

Ken Smith Thu 2020-08-13 01:54PM

Q: Given the cadence of the telescope, what drives the 60 second requirement that the alert is issued from close of shutter?

Colin Slater Thu 2020-08-13 02:03PM: With new images arriving every 37 seconds, if images take too much longer to process then that leads to a big pile up of images "in-flight", and consequent increase in hardware needs to avoid falling behind.
Ken Smith Thu 2020-08-13 02:39PM: Thanks @Colin Slater ! The reason I ask the question is that we have had internal discussions about requirements about turn-around time for onward progression of alerts.

Konstantin Malanchev Thu 2020-08-13 01:56PM

Q: https://dmtn-149.lsst.io/#authentication highlights a problem of a broker authentication. Can a solution be to use some stand-alone hardware to encrypt a connection at a lower layer, for example a router with IPSec support? Cisco ASR1004 class routers should be enough for 10 Gbps (no, I don't work at Cisco =). Or does this problem exist only for the simulator?

spencer nelson Thu 2020-08-13 01:57PM: IPSec is an interesting thought. I worry a bit about the logistics of implementing it versus an application-layer auth mechanism.
Adam Thornton Thu 2020-08-13 01:57PM: Is encrypting the data stream itself a good idea? I mean...what's the downside to leaving the data in the clear? It's not like it contains PHI.
spencer nelson Thu 2020-08-13 01:57PM: It's certainly a problem for the real implementation, not the simulator.
Konstantin Malanchev Thu 2020-08-13 01:58PM: I agree but if you have only 5 customers it could be a solution
Dave Morris Thu 2020-08-13 01:58PM: ZTF uses a white list of IP addresses, which works if there are a small number (~5) of authorised brokers.
spencer nelson Thu 2020-08-13 01:59PM: Is encrypting the data stream itself a good idea? Right, not even clear that it's necessary to encrypt it. That section of that document was trying to say "here's some things Spencer thinks might cause different performance in production vs simulation."
spencer nelson Thu 2020-08-13 01:59PM: Definitely agree that if it's just 5 customers, doing stuff at the IP level is pretty reasonable
spencer nelson Thu 2020-08-13 02:00PM: How often do our brokers' IP subnets change? Probably really slowly. That could be a way that IP firewalls break, but maybe that's not worth worrying too much
Adam Thornton Thu 2020-08-13 02:01PM: Have we talked about making one or more the brokers just being a subscribable reflector at a major cloud provider? That way you get the alerts a couple hundred milliseconds later if you're subscribed to the AWS Mirror or whatever, but then the bandwidth issue becomes the end-consumer's problem rather than the producer's.
Konstantin Malanchev Thu 2020-08-13 02:02PM: So the answer is that encryption and queue-level auth can be unnecessary
Russ Allbery Thu 2020-08-13 02:02PM: You do get integrity protection from TLS, which may have some non-security benefits depending on the situation.
Russ Allbery Thu 2020-08-13 02:02PM: But from a strict security standpoint, I'm not sure you need anything more than bearer token authentication.
Adam Thornton Thu 2020-08-13 02:02PM: For those people that do need to be a first-response subscribed-directly-to-the-firehose, that's still an option, but I bet a lot of subscriptions would gladly sacrifice a little latency for the convenience of a secondary broker.
Konstantin Malanchev Thu 2020-08-13 02:02PM: But TLS is not free too, isn't it?
Dave Morris Thu 2020-08-13 02:02PM: @spencer nelson Our mirror-maker and broker will be running in an Openstack cloud compute platform, so upstream you will see IP addresses from our cloud platform.
Russ Allbery Thu 2020-08-13 02:03PM: TLS is very close to free with modern hardware provided that you can maintain an open connection and don't have to redo the handshake.
Konstantin Malanchev Thu 2020-08-13 02:03PM: Ok, thank you
Russ Allbery Thu 2020-08-13 02:03PM: AES is extremely fast.
Konstantin Malanchev Thu 2020-08-13 02:03PM: Yes, I agree
Adam Thornton Thu 2020-08-13 02:03PM: I mean, if it's known that the stream is going to one or more of AWS, GCE, etc., then you could build your alert processing broker at the same provider/region and avoid egress fees.
spencer nelson Thu 2020-08-13 02:05PM: @Adam Thornton Yeah, reflection from within a cloud is probably a good idea. I think you're suggesting that that could be a good (component of a) community broker proposal, right?
Dave Morris Thu 2020-08-13 02:05PM: @Adam Thornton So Rubin would avoid egress fees, but the cost would be paid by the downstream brokers.
spencer nelson Thu 2020-08-13 02:07PM: Most cloud providers charge a lot of money if data leaves their private cloud's network, but it can be cheap or even free if you keep data within their realm. I think the idea would be that a broker might say "We provide free access to the full stream if you're consuming it from inside AWS's us-west-2 region" (for example)
Dave Morris Thu 2020-08-13 02:09PM: Yes, but a broker in the cloud would have to pay to export their outputs and results to end users
Russ Allbery Thu 2020-08-13 02:09PM: I think the theory is that the outputs may be much smaller than the alert stream?
Konstantin Malanchev Thu 2020-08-13 02:11PM: I agree, even bogus-vs-real filter will kill a large part of the stream
Dave Morris Thu 2020-08-13 02:11PM: If the outputs are 1/100 of the input, then yes, that is true.
Dave Morris Thu 2020-08-13 02:12PM: If we have 1 user
Dave Morris Thu 2020-08-13 02:12PM: If we have 100 users, then we loose the advangate.
Dave Morris Thu 2020-08-13 02:13PM: (*) not disagreeing with the idea, just that it isn't quite as simple as it seems.
spencer nelson Thu 2020-08-13 02:13PM: From one perspective we lost the advantage... but on the other hand, now you have 100 happy users, so maybe not so bad :slightly_smiling_face: But overall I agree pretty strongly that naive rebroadcast is likely to be expensive
Konstantin Malanchev Thu 2020-08-13 02:14PM: You have ~1e6 alerts per night and only 100 users, than the ratio will be 1/100 -1/1000
Russ Allbery Thu 2020-08-13 02:14PM: Yeah, I think it depends on the overall architecture heavily. I could imagine something where the alert stream was published inside a cloud provider and then the broker did the filtering inside the same cloud provider, exported the alerts of interest in one stream to somewhere else, and then served users from outside the cloud provider, but of course that's more complicated and may not be worthwhile.
Adam Thornton Thu 2020-08-13 02:16PM: The idea I have is something like NTP. There is a Tier 1 data feed, which is the direct-from-the-base-to-the-Tier-1-brokers. But then Tier 2 is provided by mirrors at the major cloud providers. If you WANT to subscribe to that in its entirety, you either locate your compute within the same cloud region, or you pay so much money that you probably should have just been Tier 1 in the first place. But it seems like if what you wanted was to filter the alerts to things of interest to YOU and rebroadcast THOSE, then you'd colocate your processing with the Tier 2 feed, to avoid egress except for the smaller amount of data you care about. This saves you a lot of money on inbound bandwidth (if it's free within a region) and your outbound costs are limited because your outbound data is much smaller (and at cloud providers you can also generally go to a consumer-pays model).
Adam Thornton Thu 2020-08-13 02:16PM: Which is what Russ said.

Massimo Dall'Ora Thu 2020-08-13 01:56PM

when a final decision will be made on the lightweight notification stream? This could impact the interest of some communities on developing a broker system

Eric C. Bellm Thu 2020-08-13 02:04PM: I don't expect us to be able to pursue a technical implementation earlier than early next calendar year
Eric C. Bellm Thu 2020-08-13 02:04PM: We are going to gauge reaction from this session :slightly_smiling_face: and use the remainder of the calendar year to discuss more formally with stakeholders if the reaction is positive
spencer nelson Thu 2020-08-13 02:08PM: @Massimo Dall'Ora I'm curious about which direction it would affect your interest. Does it make it more interesting, because it might be less technically onerous, or less interesting, because it would be possible to get most of the benefit without being a broker?
Massimo Dall'Ora Thu 2020-08-13 02:19PM: @spencer nelson very good question: as I said, our, let me say, "real-time community" is already served by Lasair (and partially by Antares and Alerce). Still there is some science that would require a prompt notification, but not necessarily in real-time. Therefore, accessing the lightweight stream could allow this science without necessarily develop a broker
spencer nelson Thu 2020-08-13 02:21PM: Cool, thanks.
Massimo Dall'Ora Thu 2020-08-13 02:30PM: stated in a different way: for example, for young stellar objects it is not fundamental to trigger an instantaneous follow-up, but at the same time they cannot wait for the catalogs. Am I right @Sara (Rosaria) Bonito ?
Sara (Rosaria) Bonito Thu 2020-08-13 02:32PM: @Massimo Dall'Ora I think it could be an application for this
Sara (Rosaria) Bonito Thu 2020-08-13 02:34PM: At least for the use of public data. I understand in 24 h all the DIA should be available through the Rubin Science Platform (for data rights holders)

Ignacio Reyes Thu 2020-08-13 01:56PM

Does the lightweight stream design include any image stamps? This has been super important in the ALeRCE broker for early SNe detection

spencer nelson Thu 2020-08-13 02:01PM: I think we'd need to leave image stamps out of the packet itself, but include a URL to go get the image stamps.

Francisco Forster Thu 2020-08-13 01:58PM

Q: given that most brokers will be simultaneously querying different external catalog services, e.g., CDS, should we aim for a unified crossmatching service?

leanne Thu 2020-08-13 02:01PM: This is a topic that was discussed at the community broker workshop in 2019 and many were in favour of it. It is outside of Rubin project scope to develop such a service, but I think it is a good idea.

Mario Juric Thu 2020-08-13 02:00PM

Q: From the point of view of a user who may be interested in receiving the full stream: are there considerations for choosing at least one broker which will commit to retransmitting full streams (even for a fee, if bandwidth costs are an issue)?

John Swinbank Thu 2020-08-13 02:02PM: LDM-612 &;4.7.2 tells us that "At least one broker that provides world-public access to alerts will be selected". That access isn't quite the same as "retransmitting", but it comes close.
Mario Juric Thu 2020-08-13 02:03PM: So that statement should be read as "access to all alerts"?
Emmanuel Gangler Thu 2020-08-13 02:03PM: I think retransmitting is a great feature, that would allow for a tiered approach where tier-2 brokers could subscribe to the tier-1 outgoing flux.
John Swinbank Thu 2020-08-13 02:03PM: @Mario Juric I believe that is the intent.
leanne Thu 2020-08-13 02:03PM: I believe that this is the model that SNAPS use working with ANTARES.
David Trilling Thu 2020-08-13 02:04PM: yes -- with SNAPS we could listen to a complete data stream but at present listen to an ANTARES re-broadcast of solar system objects (ie, a substream).
Mario Juric Thu 2020-08-13 02:05PM: @John Swinbank That would be great! If that's the intent, would it be possible to make that language a bit firmer (maybe revise the document)? So it's clear this refers to being able to connect and stream the full stream?
John Swinbank Thu 2020-08-13 02:05PM: That's a question for @Eric C. Bellm .
John Swinbank Thu 2020-08-13 02:06PM: (But as I say, I don't think you can read this as "retransmit" or "stream the full stream" -_I would read it as "access every alert")
Eric C. Bellm Thu 2020-08-13 02:06PM: I don't think we can require brokers to do it since we don't fund them directly.
Mario Juric Thu 2020-08-13 02:06PM: Hmm, so the answer to my original question seems to be "no"?
Eric C. Bellm Thu 2020-08-13 02:06PM: Data Rights holders can of course access PPDB, AlertDB, etc.
Emmanuel Gangler Thu 2020-08-13 02:07PM: But brokers may propose to do it as part of their proposal. Useful to know if this would be appreciated by the project.
John Swinbank Thu 2020-08-13 02:07PM: The answer to the original question is "it's up to the SAC".
Eric C. Bellm Thu 2020-08-13 02:07PM: @Emmanuel Gangler it would absolutely be appreciated and encouraged

Ernesto Castillo Thu 2020-08-13 02:01PM

Hi @spencer nelson , are you going to continue using Kafka for alert distribution besides the object storage API that you are planning ?, or you are going to try another technology ?

spencer nelson Thu 2020-08-13 02:02PM: I would expect to keep using Kafka. I'm interested in alternatives, but I think the community already knows how to run and work with Kafka pretty well

Kathy Vivas Thu 2020-08-13 02:05PM

@Nina Hernitschek your broker provides updates on KNOWN variable stars only? I am not sure if I understood you right

Nina Hernitschek Thu 2020-08-13 02:09PM

No, this is not only on known variable stars. The intention is to provide updates on a) known stars (e.g. to simply get updated light curves and features on specific stars), but b) first and foremost on newly found variable stars in a known region. E.g.: someone is interested in RR Lyrae in stellar streams (I am :slightly_smiling_face: ), one can specify the area on the sky and gets updates on everything that matches the desired pattern, e.g. variable, likely RR Lyrae, features such as periods/amplitudes/phases.

Kathy Vivas Thu 2020-08-13 02:09PM: Excellent! Thanks!
Monika I. Jurkovic Thu 2020-08-13 02:10PM: What is happening with long-term changing variables (e.g. RV Tauri type variables)?
Nina Hernitschek Thu 2020-08-13 02:12PM: Not implemented yet, but features should be mostly quite general, whereas if classification is reliable, also more specific (e.g.: for a periodic variable star, provide period). The most basic function should be: provide updated light curve.
Monika I. Jurkovic Thu 2020-08-13 02:13PM: Thank you. :+1:

parejkoj Thu 2020-08-13 02:09PM

Reminder: the full broker presentation videos are on the session website: https://project.lsst.org/meetings/rubin2020/agenda/session/community-alert-brokers

Matthew Graham Thu 2020-08-13 02:11PM

If people want to experiment with putting up their own broker and getting a real-time industrial-scale alert stream, talk to us at ZTF ( mailto:mjg@caltech.edu , mailto:ecbellm@uw.edu )

Emmanuel Gangler Thu 2020-08-13 02:16PM: I must say that here at the Fink team, we deeply appreciate the way ZTF is open to new brokers testing their architecture !

leanne Thu 2020-08-13 02:14PM

@Matthew Graham I'm curious about the progress/state of Babamul

andyxl Thu 2020-08-13 02:16PM: I am curious whether I can learn Akkadian on Duolingo
leanne Thu 2020-08-13 02:16PM: I'm sticking with the more conventional "Spanish" on duolingo
Matthew Graham Thu 2020-08-13 02:17PM: We've been busy securing funding for ZTF Phase II for the past six months but I've had a summer student working on some models.
Matthew Graham Thu 2020-08-13 02:18PM: @andyxl No, Duolingo does not currently support Akkadian and frankly its Latin course is awful.
Matthew Graham Thu 2020-08-13 02:18PM: So Babamul has been on the back burner a bit

Chris Lintott Thu 2020-08-13 02:16PM

Question for Pitt-Google - how does your metaclassifier handle previously unknown classes, or those which you don't have target classifiers for? Seems like it could be useful there. (Sorry, missed who presented)

spencer nelson Thu 2020-08-13 02:16PM: @Troy Raen (he/him) was the presenter there, I think
Chris Lintott Thu 2020-08-13 02:17PM: Thanks!
Troy Raen (he/him) Thu 2020-08-13 02:20PM: The meta-classifier would (obviously) need prior knowledge of a class to classify an object as such, but this does not have to come from a targeted classifier. The meta-classifier also looks directly at alert data and cross-matched data. I anticipate/hope that previously unknown classes will show up interestingly strange probability distributions.
Troy Raen (he/him) Thu 2020-08-13 02:21PM: Thanks for the question @Chris Lintott . Is this an area that you work in?
Chris Lintott Thu 2020-08-13 02:22PM: Yes, sort of - it's something we've been thinking a lot about at Zooniverse, as part of our use case is finding the weird stuff. It's also come up in using the results of a Bayesian cnn we've deployed alongside the Galaxy Zoo project (see Walmsley et al. 2020)
Troy Raen (he/him) Thu 2020-08-13 02:28PM: Thank you, I'll take a look. It's an area that I've considered a little bit, but not very in-depth.

Eric C. Bellm Thu 2020-08-13 02:17PM

Surfacing an implicit question on Zoom from @Roy Williams about how the lightweight packet contents would be selected: I expect that the project would propose a sample, and send it to the science collaborations for comment and suggestion. (A similar process is planned for timeseries features.)

spencer nelson Thu 2020-08-13 02:19PM: I think a fair goal would be to keep it under 300 bytes per alert. There's an interesting tradeoff: a really skinny packet is easy to broadcast widely... but users might be forced to fetch the big packet almost every time, so you actually could come out behind on bandwidth if you haven't included enough metadata to make a good filtering decision.
spencer nelson Thu 2020-08-13 02:20PM: The good news is that basic numbers - floats, integers, booleans - are really cheap. Today's alert design is about 90% FITS image data, 8% alert history, and 2% "plain" data.
Roy Williams Thu 2020-08-13 02:22PM: Light curve is only 3 floats per detection ...
Andy Howell Thu 2020-08-13 02:23PM: You could just have one with the LC and one without, and users can pick.
Andy Howell Thu 2020-08-13 02:26PM: I myself would never choose passing the full LC with every message. It becomes a serious accounting job to make sure you always have the latest one. I see this in MARS and it drives me crazy. But I can see how some prefer it that way.
spencer nelson Thu 2020-08-13 02:27PM: Doing it both ways requires duplicate storage if we do it with Kafka. Maybe that's fine?
spencer nelson Thu 2020-08-13 02:28PM: I don't know enough to really talk about the merits of any particular lightweight alert packet contents, but 3 bytes floats per detection does sound pretty cheap
K-T Lim Thu 2020-08-13 04:22PM: 3 floats per historic detection per current detection.

Ernesto Castillo Thu 2020-08-13 02:18PM

Hi @Eric C. Bellm , @spencer nelson , are you thinking to store LSST data in any cloud service publicly available such the Open Data Program that AWS has: https://aws.amazon.com/es/opendata/open-data-sponsorship-program/ ?, it would open new possibilities to process the data using Spark or other tools. In ALeRCE Broker we have been using AWS tools to process all ZTF historic data with good results, @JulienPeloton from Fink Broker has been using Spark on-premises as well to mention a pair of usage examples.

Eric C. Bellm Thu 2020-08-13 02:22PM: It's certainly plausible, I think the details will matter (especially where the US Data Facility is). @leanne may wish to weigh in
spencer nelson Thu 2020-08-13 02:22PM: That AWS data program is new to me and pretty intriguing. I'm not aware of any existing discussion about it, but it looks worth more investigation... do other clouds have similar things?
mwv Thu 2020-08-13 02:26PM: For the Pitt-Google Broker we're very much talking to Google about providing things as part of Google Cloud Public Datasets program.
mwv Thu 2020-08-13 02:26PM: https://cloud.google.com/public-datasets
mwv Thu 2020-08-13 02:26PM: In part, we're pursuing that option to save on our own storage costs. But we're optimistic that we can make a good case about this.
leanne Thu 2020-08-13 02:30PM: Agree with @Eric C. Bellm that it is plausible, but we have no explicit plans today to do so. Bob Blum announced in his operations QA that the Rubin interim data facility, which will be used in pre-operations until the USDF is decided, will be in the commercial cloud (which has not yet been decided). The location of the USDF will not be decided for some time. Commercial cloud companies are eligible to apply to become the USDF.
Ernesto Castillo Thu 2020-08-13 02:31PM: it would be interesting to evaluate both alternatives (Amazon and Google) at least and see what does each has to offer? Besides that it would be worth to have all the data publicly available for free and to have the option to use paid tools to perform further processing. In our case we have been able to proccess ZTF historic data ( ~4TB) in a matther of minutes using 60 instances of AWS paying less than 60 dollars peer hour.
Ernesto Castillo Thu 2020-08-13 02:33PM: maybe we can continue talking further @spencer nelson about this topic, I have been talking with Christopher Phillips related with this, may be it can be tested with ZTF data officially and see how it goes :slightly_smiling_face:
spencer nelson Thu 2020-08-13 02:34PM: I'm certainly interested, but I'd guess that this goes way above my pay grade :slightly_smiling_face: A lot of the senior management of Rubin is pretty involved in cloud vendor decisions, and I don't want to mess with that too much
spencer nelson Thu 2020-08-13 02:35PM: Definitely interested in the ZTF side
Eric C. Bellm Thu 2020-08-13 02:35PM: @Ernesto Castillo as far as discussions about ZTF, though, @spencer nelson can also chat with you there as he's working on ZTF too
Ernesto Castillo Thu 2020-08-13 02:36PM: ok, thank you, let's keep us in touch, I will send you an email to continue talking, take care!
leanne Thu 2020-08-13 02:40PM: The pre-operations IDF will be a great opportunity for us to evaluate LSST data in the cloud.
leanne Thu 2020-08-13 02:43PM: The DM team has already conducted PoC projects with both Amazon and Google. They were more limited in scope that what @Ernesto Castillo is proposing but have given us some valuable experience with working with LSST data at both Amazon and Google. See DMTN-150, DMTN-157, DMTN-125.. DMTN-137, DMTN-114 if you are interested in the details
Ernesto Castillo Thu 2020-08-13 02:47PM: thank you for the information!
Mariano Dominguez Thu 2020-08-13 04:28PM: Question for Leanne: do you known about any experience using DC2 data on GC? Thank you in advance!
leanne Fri 2020-08-14 01:11PM: Not to date no, however, the interim data facility (IDF) will be on a commercial cloud and the operations team is collaborating with the DESC to use DC2 in Data Preview 0 (DP0) in the IDF. The timescale for setting up the IDF is September 2020 so watch out for some exciting announcements from operations team in the very near future.
leanne Fri 2020-08-14 01:11PM: (p.s please @- me to get my attention)

Matthew Graham Thu 2020-08-13 02:22PM

Alerce currently reports independent discoveries on the ZTF public stream to TNS as its own effort

Francisco Forster Thu 2020-08-13 02:27PM: What do you think would be the best idea to share credits? We acknowledge ZTF as the source of the alerts. We are open to change the way we report ZTF candidates to TNS.
Matthew Graham Thu 2020-08-13 02:29PM: I think it's fine to acknowledge ZTF as the source but the identification (discovery) of the alert as interesting is something that the broker (Alerce) has done

Mario Juric Thu 2020-08-13 02:23PM

A question for the broker teams: is there a broker that plans to have a feature where a user could subscribe to a full stream (even for a fee, if bandwidth is an issue)? I'm imagining a situation where in (say) 2025 someone may come up with a good idea that requires receiving the full stream, gets a grant funded, but in the present framework it's unclear to me how they would get access to the full stream?

Troy Raen (he/him) Thu 2020-08-13 02:24PM: The Pitt-Google broker will offer the full stream. Not sure about others.
Mario Juric Thu 2020-08-13 02:25PM: Any others?
Troy Raen (he/him) Thu 2020-08-13 02:27PM: We will also keep an alert database (i.e. a public analog to the prompt products database) so users can go back and retrieve past data.
Mario Juric Thu 2020-08-13 02:29PM: That's fantastic, thanks! I presume all that will be on BigQuery/PubSub?
Troy Raen (he/him) Thu 2020-08-13 02:29PM: yes
andyxl Thu 2020-08-13 02:29PM: I would like to answer for Lasair but nervous so waiting for @Roy Williams
Roy Williams Thu 2020-08-13 02:30PM: Yes. Easier to justify if they are UK people ...
Eric C. Bellm Thu 2020-08-13 02:31PM: This doesn't really constrain the operations project, but in LDM-612 we we said that selected brokers would be evaluated throughout the survey. I could imagine a supplemental call for new full stream brokers later
Mario Juric Thu 2020-08-13 02:33PM: @Troy Raen (he/him) (I know I'm in danger of going too deep in the weeds but I'll still ask...) Assuming the stream is roughly ~1TB/night, and PubSub is ~$40/TB, then the cost to receive a nominal year (~330 days) would be k$13.2 -- is that the right ballpark?
Anais Möller Thu 2020-08-13 02:33PM: Retransmitting the full stream may be complicated technically. We do plan to forward substreams at Fink, but technically the limit may be at least after quality cuts of the stream (which may reduce ~30%). Definitely something to discuss !
Mario Juric Thu 2020-08-13 02:37PM: It's great to hear there's thinking about this! In ZTF-related experiments we've tested serving around ~40 full-scale LSST streams within the same data center (w. a 10-broker Kafka cluster) -- it seemed like the cluster could have scaled further, but I didn't try it. Happy to share some details if anyone's interested (e-mail me). [this was done on Digital Ocean, but I suspect the same would hold on other cloud providers or on-prem]
Francisco Forster Thu 2020-08-13 02:46PM: We are working closely with AWS and the Chilean Data Observatory. We will explore similar paid for solutions for full stream distribution.
Troy Raen (he/him) Thu 2020-08-13 02:46PM: @Mario Juric Yes, that's the right ballpark assuming Google does not host the data as a public dataset. If we can convince them to do so the price will drop to around $10/TB, so $3600/yr. Our https://project.lsst.org/meetings/rubin2020/sites/lsst.org.meetings.rubin2020/files/Wood-Vasey_PittGoogle.pdf include costing models.
Mario Juric Thu 2020-08-13 02:49PM: @Troy Raen (he/him) Brilliant, that's cheap (relative to a typical grant/research/additional computing cost for something that would require a full stream). You have my vote :slightly_smiling_face: !
Troy Raen (he/him) Thu 2020-08-13 02:50PM: credit to @mwv for that!
Eric C. Bellm Thu 2020-08-13 02:53PM: @Troy Raen (he/him) how are you planning to control egress costs? That always seems to be the biggest risk with cloud services
Mario Juric Thu 2020-08-13 02:54PM: (at least for my use cases), egress is not an issue if you do your computing on GCP.
Troy Raen (he/him) Thu 2020-08-13 02:55PM: Deferring to @mwv , if he has a more complete answer.
Eric C. Bellm Thu 2020-08-13 02:56PM: @Mario Juric are there inter-region charges, though? I'm just not sure how much control GCP/AWS give on the various services
Mario Juric Thu 2020-08-13 02:57PM: @Eric C. Bellm -- I don't think so, as long as computing occurs in the same region where PubSub happens (I believe, my experience is more on the AWS side). And many use case involve filtering or summarization, rather than full transforms, so data that eventually leaves the provider may be small.

In my mind this is a good rationale to have someone (the project?) ensure there's a full stream in every major cloud provider, plus all major NSF data centers -- it would be very cheap (relative to total ops cost), but would open a myriad of possibilities (and make the need to select a small number of brokers less critical).
Eric C. Bellm Thu 2020-08-13 03:00PM: Sorry, I'm asking if the data owner can restrict users to only compute on it in the region where it is stored.
Eric C. Bellm Thu 2020-08-13 03:01PM: where "users" are bringing their own credit card to to the processing, potentially, and so aren't part of the organization that owns the data from the point of view of AWS/GCP
Mario Juric Thu 2020-08-13 03:03PM: ... Reading up on how Google does PubSub pricing, but hope @Troy Raen (he/him) can beat me to the answer :slightly_smiling_face: .
Troy Raen (he/him) Thu 2020-08-13 03:06PM: I think the data owner can restrict users in that way, but I don't know the details.
Gautham Narayan Thu 2020-08-13 03:14PM: We've pretty consistently said we'd we'd re-transmit a full stream, but there's some subtleties here - what happens if LSST reprocesses diffims and updates the photometry (as they will do in the yearly releases). Do you want the originally broadcast stream as it was sent out, or the best available data? We should also be able to allow clones of the alerts database, for those who want the alerts, but not in stream format
Mario Juric Thu 2020-08-13 03:17PM: @Eric C. Bellm @Troy Raen (he/him) I think this may be up to the user -- if they need/want to run in a different region (at a higher cost), they may. But it doesn't affect the broker's costs. Regarding pricing, it looks like it's $40/TB for PubSub, no egress charges within the same region, and inter-region egress would be $10/TB (in the US), $20/TB (Europe) and $50/TB (asia). So it looks like on order of >=k$14/yr, depending on where the consumer puts their compute. This is consistent with the pricing model from the slides Troy posted. Sources: https://cloud.google.com/pubsub/pricing https://cloud.google.com/vpc/network-pricing https://davidmytton.blog/how-much-is-spotify-paying-google-cloud/ (disclaimer: IANACE -- "I am not a cloud engineer" -- so check my numbers).
Mario Juric Thu 2020-08-13 03:31PM: @Gautham Narayan I'd argue for doing exactly what Rubin does (which makes the retransmitted stream exactly interchangeable with subscribing directly to Rubin's stream). I think Rubin won't issue alerts for anything outside nightly processing (but @Eric C. Bellm is the authority here).
mwv Thu 2020-08-13 03:31PM: I think if it's a retransmitted stream, it should be exactly the same.
mwv Thu 2020-08-13 03:32PM: If the user instead wants: "What's the latest and greatest on this object?" then you can answer that question, too. But it's not the stream.
Gautham Narayan Thu 2020-08-13 03:32PM: so, e.g. no refining artifact filtering, even if we know previously sent out alerts are junk?
Gautham Narayan Thu 2020-08-13 03:32PM: this choice has consequences
Gautham Narayan Thu 2020-08-13 03:32PM: quite expensive ones
Mario Juric Thu 2020-08-13 03:35PM: I think you could (should, IMHO) offer another "enhanced" stream (as a broker). But if someone wants everything (e.g., to check their own artifact filtering algorithm, or some other purpose) they may want a full retransmitted stream. As the user pays, the onus is on them to choose the right thing for their use case. That's why I started this thread, as it wasn't clear to me if anyone was going to offer the original, unfiltered, stream.
Gautham Narayan Thu 2020-08-13 03:36PM: ahhh you are talking about the user paying - I'm not, but this really will depend on where the alert production facility itself is
mwv Thu 2020-08-13 03:39PM: To generalize a bit, I'd like to have a model where any particular capacity needs are just a simple issue of money. If the total bill is <$10k/yr, then it would be silly to individually charge users. And if the total bill to provide access to everyone who can possible want it is ~$150k/yr , then "we" should just do that.
mwv Thu 2020-08-13 03:39PM: And we can argue/compete/volunteer/delegate who "we" is.
Mario Juric Thu 2020-08-13 03:45PM: +1. IMHO, I think there's a strong case to be made for the project to commit to delivering full streams to major cloud/NSF data centers. The cost would likely be below ~100k/yr. Then full-stream-availability for O(10k/yr) is guaranteed to anyone who needs it (e.g., wishes to build/operate a broker, or do science on the full stream). The need for broker selection may also become moot.
Eric C. Bellm Thu 2020-08-13 04:10PM: I don't disagree with any of this, but the details matter (real cost, capability, restrictions, ...) Nothing would make it easier to make the case to the agencies than working examples that show the user demand and deal with the edge cases. We got LOIs from both of you and I know you have access to the ZTF stream :-)
Mario Juric Thu 2020-08-13 05:59PM: It's a good idea!! But I think it would require defining a clear pilot project with clear success criteria (i.e., what would one need to demonstrate for Rubin to switch to this model?) and also a reasonable level of commitment from Rubin to actually switch if the pilot project succeeds. Otherwise it's tough to commit one's (student's) time for something where the path to success is unclear (even if Rubin decided funded the experiment).

But if something like that could be agreed to, I'd be super interested and eager to participate!

Mario Juric Thu 2020-08-13 02:26PM

But the Rubin project could allocate one slot for at least one such broker?

Wesley Fraser Thu 2020-08-13 02:31PM

Thanks all

Mario Juric Thu 2020-08-13 02:31PM

Thanks @Melissa Graham @Eric C. Bellm et al., very useful!

Ken Smith Thu 2020-08-13 02:32PM

Thanks for the great discussion!

Jakob Nordin Thu 2020-08-13 02:32PM

Thanks!

Stephen Smartt Thu 2020-08-13 02:32PM

thanks @Melissa Graham , @Eric C. Bellm and team. all very useful :+1:

JulienPeloton Thu 2020-08-13 02:32PM

Thanks!

Eric C. Bellm Thu 2020-08-13 02:32PM

and to thanks the broker teams for sharing very impressive progress!

Bob Mann Thu 2020-08-13 02:32PM

:+1:

Amanda Bauer Thu 2020-08-13 02:32PM

how do broker results compare to each other?__(i'm thinking, for example, how does_EPO_know which one to "trust" or go to first?)

John Swinbank Thu 2020-08-13 02:34PM: I love this question. All the brokers look really impressive, but - with the exception of those which are targeting a specific scientific community -_it's hard to get a really good overview of the fundamental differences between them based on the short presentations.
John Swinbank Thu 2020-08-13 02:35PM: I guess that means I need to go and spend more time studying the material on the breakout website...
Roy Williams Thu 2020-08-13 02:37PM: For Lasair, it is not the broker making decisions, we are a platform for others to do that,
Francisco Forster Thu 2020-08-13 02:38PM: One of the results of the planned broker's workshop is to design broker benchmarks. I think it's important to think carefully about them. For example, a confusion matrix is not enough, as the training and target set are usually not representative.
Jakob Nordin Thu 2020-08-13 02:43PM: For all of the items in this thread we need some consist language to distinguish an algorithm (cited to a developer) and a broker (infrastructure that enables algorithms to run)
Monika Soraisam Thu 2020-08-13 02:44PM: for starters, the (algorithm) code can be made public [we do that in ANTARES].
Anais Möller Thu 2020-08-13 02:45PM: @Amanda Bauer very interesting question, I think developing benchmarks is a priority. In our view, brokers should "not loose objects" thus will deliver a large sample with low-purity but as complete as possible. But we can develop filters to obtain as well high-purity, reliable samples for those that need them!

Rbiswas4 Thu 2020-08-13 02:34PM

the way the lightweight alerts were phrased, it sounds like there is no downside to it, and some (hopefully large upside) . Is there a way users could be seeing a disadvantage at all?

andyxl Thu 2020-08-13 02:35PM: I think a potential disadvantage is to tempt users into thinking they can do everything themselves, and so wasting human effort....
Roy Williams Thu 2020-08-13 02:36PM: If the lightweight packet is too light, then each one will create a callback for more information, overloading the servers.
Rbiswas4 Thu 2020-08-13 02:36PM: good point on psychology... thanks!
John Swinbank Thu 2020-08-13 02:36PM: It means that when you receive an alert, you don't receive all the information the project has about it. If you want more information, you need a callback, adding latency.
Eric C. Bellm Thu 2020-08-13 02:36PM: a few off the top of my head: there's a bit more operational complexity (two services instead of just one)
spencer nelson Thu 2020-08-13 02:37PM: Latency will likely be 1 RTT, so probably about 100ms, maybe 500ms if you're around the globe, maybe 4s if you're unlucky that day
John Swinbank Thu 2020-08-13 02:37PM: It also makes the canonical form of the alert stream harder to reason about: if you have a "heavy" alert packet, you have the alert packet . If you're reliant on callbacks to some other service, then there's more scope for information to get lost, database schemas to change, etc etc.
Eric C. Bellm Thu 2020-08-13 02:37PM: the model of having only community brokers receiving the full stream (and everybody else gets nothing) makes it easier to fund community brokers
Eric C. Bellm Thu 2020-08-13 02:38PM: retrieving the full alert packet from a Rubin service creates data rights issues that are somewhat annoying to deal with
Eric C. Bellm Thu 2020-08-13 02:39PM: (even though the alert packets are still world-public in the sense of being freely sharable by anyone)
andyxl Thu 2020-08-13 02:39PM: @Eric C. Bellm it seems to me that a way to have your cake and eat it is for the community brokers to have a a switch for "give me the lightweight stream then I will think about it" rather than assuming traffic with Rubin DF.
spencer nelson Thu 2020-08-13 02:39PM: There are thorny questions around callbacks, as John said. For example, how long are alert callback URLs valid for? I don't think the answer can be "forever" just because the storage volume required would be immense
andyxl Thu 2020-08-13 02:40PM: But the project could still agree and mandate what a "lightweight packet" means
Eric C. Bellm Thu 2020-08-13 02:40PM: I like cake! And there's nothing stopping brokers from doing exactly that--indeed, the Google/Pitt slide referred to something similar
Rbiswas4 Thu 2020-08-13 02:44PM: Thanks all!
Rbiswas4 Thu 2020-08-13 02:46PM: @spencer nelson I got stuck at RTT ... what is that ?
John Swinbank Thu 2020-08-13 02:46PM: round trip time
spencer nelson Thu 2020-08-13 02:48PM: Yeah, network round trip time: how long does it take for you to send a request across the internet, and get a response? It's (2 * speed of light in the cables * distance) + (delays from network processing hardware)
Rbiswas4 Thu 2020-08-13 02:56PM: Thanks!

Rseaman Thu 2020-08-13 02:34PM

Good session! Melissa is a great chair. There was an open question in the chat at the end about FITS compression and header metadata. Suggest analyzing the requirements separately between header and data records. Subtractive dithering of aggressively lossy Rice compression can squeeze the latter quite efficiently, the FITS tiling is a separate issue.

spencer nelson Thu 2020-08-13 02:44PM: I'm interested in knowing more about this. My intuition was that we wouldn't really have a prayer at getting a 40x40 FITS image under, say, 1000 bytes. Maybe that's wrong, though?

Melissa Graham Thu 2020-08-13 02:38PM

Special thanks to moderators @parejkoj and @mrawls , and to @spencer nelson and @Chris Morrison for joining us for the session. Good job everyone! :tada:

Eric C. Bellm Thu 2020-08-13 02:43PM

@channel Quick poll: Should the Rubin Project seriously explore providing a "lightweight" alert stream to potentially more brokers/users, with the full alert packets (still publicly sharable, same contents) to be retrieved separately?

Mario Juric Thu 2020-08-13 02:45PM: Question -- Is there a downside (in terms of something else not being built if resources are diverted to explore this idea)?
Ignacio Reyes Thu 2020-08-13 02:45PM: And still having the full stream in parallel for some brokers?
spencer nelson Thu 2020-08-13 02:46PM: @Ignacio Reyes , my thought was that the full stream should be available to brokers by letting them call back for the big alert packets on every single alert - no rate limit at all. Would that be sufficient?
Eric C. Bellm Thu 2020-08-13 02:48PM: @Mario Juric there is a cost to studying it, of course--the management end just means I do this rather than spend more time looking closely at pipelines performance. In terms of technical work, it means @spencer nelson works on this instead of trying to figure out how to performance tune Kafka for enormous messages
Ignacio Reyes Thu 2020-08-13 02:48PM: I imagine that we will be interested in querying every single alert
Gautham Narayan Thu 2020-08-13 02:48PM: I don't really have a good sense for what the light weight alert would have yet. I think there's a lot of merit in just distributing the most recent detection with no history + URL to the postage stamps and no features
Rseaman Thu 2020-08-13 02:49PM: Putting aside technical / IVOA issues, you might review various discussions from the VOEvent / Hotwired workshops, slides, and papers for similar lightweight versus data-heavy packet use cases. It may be that having every lightweight packet potentially generate numerous database queries is no longer seen as an impediment.
spencer nelson Thu 2020-08-13 02:49PM: @Ignacio Reyes yes, and I think that would 100% definitely need to be supported for a specifically selected audience
Gautham Narayan Thu 2020-08-13 02:49PM: Brokers already maintain the history (for longer than is in the DIA record at that), and the postage stamps are not going to be useful for all alerts, so if you cut those out, then absolutely you'll have a small packet that many broker teams can use.
spencer nelson Thu 2020-08-13 02:50PM: @Rseaman that's a good point. My thinking was that alert packet contents are generally immutable; they only need to be queried from a database once, and then put into a disk-based cache, so DB query load should not be different
Eric C. Bellm Thu 2020-08-13 02:51PM: @Rseaman indeed, as you know the LSST heavy alert concept came from the desire to avoid users round-tripping to the databases themselves. I think that's still a problem: the database teams on DM are working very hard to enable us to get the one-year histories out for alert packets. It seems to me to continue to make sense to cache those (effectively) in an alert packet.
Rseaman Thu 2020-08-13 02:53PM: Also, don't know if Scott Barthelmy was on the call, but you might review any history GCN has between binary and ascii formats to address similar light versus heavy use cases.
Agkim Thu 2020-08-13 02:56PM: To supplement the lightweight alert stream, is it possible for the PPDB to start to be populated earlier than the nominal 24h?
Eric C. Bellm Thu 2020-08-13 02:58PM: @Agkim good question! Paging @leanne . I certainly don't think we can commit to it.
leanne Thu 2020-08-13 03:01PM: @Eric C. Bellm is right, that we cannot commit to nor promise this. We are required to make all LSST prompt data products available within 24hrs of the acquisition of science data.
leanne Thu 2020-08-13 03:05PM: We do not however intend to intentionally wait for 24 hrs to make the prompt data products available if they are available earlier, and so it is possible that they will be in the PPDB before 24hrs - but we will not commit to this.
leanne Thu 2020-08-13 03:12PM: We also have a requirement to not release prompt image and catalog data products (not alerts) for a period of time. This is currently set at 6 hrs. The scientific impacts of this delay and possible mitigation strategies are currently under discussion.
Stephen Smartt Thu 2020-08-13 03:13PM: I think this is worth exploring, if it means more teams can access the "core" data. We'd still like the full content alert stream, but it might relieve pressure on selection process. It could expand diversity of ideas.
Massimo Dall'Ora Thu 2020-08-13 04:53PM: As discussed in another thread, for several science cases a notification about what is going on is important, even if a possible follow-up could be triggered on a more relaxed time scale. Let's talk, for example, of some classes of young stellar objects, or periodic variable stars at the detection limit (so that only the maximum light curve appears, mimicking a kind of transient). This could relax the pressure on the community brokers, and allow a more extended community to develop their own tools, for specific needs.
Matthew Graham Thu 2020-08-13 05:14PM: I think that Rubin should seriously explore a lightweight alert stream/more brokers/users option
Chien-Hsiu Lee Thu 2020-08-13 06:30PM: When you say full alert packets to be retrieved separately, are they still going through the same bandwidth? There might be users want to query many (if not all) full alert packets, and that will still cause a problem to the bandwidth...
Eric C. Bellm Thu 2020-08-13 07:05PM: @Chien-Hsiu Lee the current model assumes that all of the alert stream has to get out in ~5 seconds to meet the 60 second latency budget. That's a very bursty use of the bandwidth. Callback retrieval could be more spread out in time.
Chien-Hsiu Lee Thu 2020-08-13 07:49PM: Thanks @Eric C. Bellm !
Francisco Forster Fri 2020-08-14 11:05AM: @spencer nelson , am I right to think that some of the problems associated to full alert stream distribution are due to the very large image stamps which are sometimes produced (discussed by @JulienPeloton in community)? It seems to me that they are associated to a few niche science cases. Would it make sense to have only those out of the full stream and make them retrievable?
spencer nelson Fri 2020-08-14 11:32AM: Thats some of the problem, but not all. Moderate alerts with stamps that arent huge are about 80KB, and we release 10,000 of them in 5 seconds. That's 128Mbps per consumer outbound bandwidth
spencer nelson Fri 2020-08-14 11:32AM: The anomalous 4MB alerts are definitely bad but they arent the whole story. Smoothing out the load over 60s with rate limited callbacks is what really saves us
Ignacio Reyes Fri 2020-08-14 12:07PM: Could we spread the stream distribution window using kafka? Is that critical to be under 60s latency? Maybe using extra 20s or 30s to send the data to the brokers can avoid the bandwidth spike issue.

Ranpal (she/her/hers) Fri 2020-08-14 07:11AM

The recording of the live session is here: https://youtu.be/Bg4dv9qvFyU