[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20231014160744.GA18959@openwall.com>
Date: Sat, 14 Oct 2023 18:07:44 +0200
From: Solar Designer <solar@...nwall.com>
To: oss-security@...ts.openwall.com
Subject: Re: linux-distros list membership application - CIQ Rocky Linux Security Team
Hi Neal,
Thank you for bringing up your objections and rationale.
I find some of your arguments valid (but insufficient to block this
application), some others not. Anyhow, this is an opportunity for us to
discuss how the various projects (distros, SIGs) handle things and
whether they're represented or eligible for (linux-)distros membership.
A recurring theme in both of your messages is whether an "open project
or community project" can be represented on (linux-)distros. Your
answer is no because "there is no mechanism in the project to hide
anything from the community." My answer is yes if there is such a
mechanism despite of the project being open/community in other ways.
For example, Debian is an open community project, yet Debian is
explicitly listed as a linux-distros member and indeed representatives
from Debian's security team are subscribed and the team manages to
prepare security updates without breaking embargoes. How exactly they
do it I don't know, but I assume they have a mechanism.
You say that "Fedora is not a member". While we do not specifically
list Fedora as a member, we do list Red Hat, and my understanding is
that this includes representation for Fedora - not from the open
community side of it, but from Red Hat's side of it. Using the glibc
CVE-2023-4911 example and Red Hat Bugzilla entries:
https://bugzilla.redhat.com/show_bug.cgi?id=2238352
https://bugzilla.redhat.com/show_bug.cgi?id=2241966
we can see that Zack Miele at Red Hat both participated in handling of
the incoming embargoed issue report (in this case, presumably Red Hat
was directly notified by Qualys a couple of weeks before the issue was
brought to linux-distros, which is fine) and created the public entry
for Fedora at 2023-10-03 17:12 UTC, which is almost exactly at the
pre-agreed public disclosure date/time of 2023-10-03 17:00 UTC. The
first Fedora glibc package update listed in there is glibc-2.38-6.fc39
at 20:05 UTC, a mere 3 hours later, or a mere 2 hours after Qualys'
actual public disclosure of the issue (the oss-security posting was at
17:50 UTC). That's great. Looking at the package change log:
https://packages.fedoraproject.org/pkgs/glibc/glibc/fedora-39.html
We see the relevant change was made by "Arjun Shankar <arjun at redhat
dot com>", so also by someone at Red Hat. I doubt we'd see such quick
handling of the issue without preparedness from Red Hat's side, and thus
without Red Hat having advance knowledge of the issue. While for this
specific issue they had knowledge before linux-distros, I think it's the
same in cases where the issue first gets to Red Hat via linux-distros -
they don't forget about preparedness also for Fedora.
What I am proposing for CIQ and Rocky Linux is similar - only "CIQ Rocky
Linux Security Team" would receive the embargoed information, and (at
least for issues above an overall severity threshold, like this glibc
one was) would help ensure preparedness for both CIQ LTS branches and
Rocky Linux, without leaking the information to CIQ customers nor to the
entire Rocky Linux community prematurely.
Some further comments inline:
On Fri, Oct 13, 2023 at 03:50:13AM -0700, Neal Gompa wrote:
> On Wed, Oct 11, 2023 at 10:00 AM Solar Designer <solar@...nwall.com> wrote:
> > > The publicly verifiable track record currently consists of timely
> > > rebuild and re-release of RHEL security update packages and security
> > > advisories, as published here:
> > >
> > > https://errata.rockylinux.org
> > >
> > > Not currently verifiable publicly, but Gregory further tells me:
> > >
> > > "We've been doing LTS privately to our customers for over a year now.
> > > This means we maintain security fixes for customers who need long term
> > > support for point releases."
>
> From my point of view, this does not count. Rocky's public track record
> of rebuilding RHEL updates and shipping them in a timely fashion does
> not indicate that Rocky/CIQ can respond effectively when you have a craft
> updates from scratch.
Fair enough. Ideally, we'd also have public track record showing CIQ's
LTS branch updates, but unfortunately this is not currently public. So
we have this combination of publicly verifiable track record of rebuilds
and republishing (which shows that the project cares and is long-term),
statement that own updates were also being made for LTS branches, and
public information on recent own updates via the SIG (no track record,
but demonstrates capability, infrastructure setup, and intent).
The timely rebuilds alone satisfy the criterion's current wording. Not
being a rebuild-only distro or having additional justification is a
separate criterion, which does not require a long-term track record.
I think this combination (barely) clears the bar for the two criteria.
> Furthermore, there are public posts and articles
> indicating that Rocky Linux/CIQ has trouble with shipping updates in a
> timely fashion at all.
>
> Examples on updates:
> https://forums.rockylinux.org/t/some-errata-missing-in-comparison-with-rhel-and-almalinux/3843
> https://forums.rockylinux.org/t/rocky-linux-9-errata-missing-late-8-errata/6890
> https://forums.rockylinux.org/t/errata-rockylinux-org-not-updated-since-sep-02-2022/7676
There are occasional hiccups with receiving the upstream distro's errata
publications. In fact, I am aware of a missing recent security
advisory, even though the actual update packages are there - I'm told
this one Red Hat advisory is mysteriously missing from the specific
upstream API we use, which will hopefully be corrected by switching to
another available API for these. So yes, there are such examples.
However, the criterion isn't that 100% of updates and publications must
be quick. Things do go wrong sometimes, and updates for lower severity
issues are often reasonably delayed, including by current linux-distros
members and especially for issues that were not even handled via the
list. Rather, the criterion should be that updates are typically quick,
especially for high severity issues handled via linux-distros, so that
membership could make these even quicker.
I see that the current wording mentions specific delays, but does not
mention issue severity - perhaps that's something to add, as it's
unreasonable to insist on quick fixes for low severity issues (they're
nice to have and provide extra justification, but not a requirement).
> Example on releases: https://www.theregister.com/2022/07/18/rocky_linux_9/
Rocky Linux 8 remained fully supported (and still is), so the delay in
releasing Rocky Linux 9 is of no direct relevance to this application.
It's great that AlmaLinux was much quicker, and this may (or may not)
indirectly compare the teams' capabilities (or maybe focus areas), but
for the purpose of this membership application it's not a competition.
> > > > Not be (only) downstream or a rebuild of another distro (or else we need convincing additional justification of how the list membership would enable you to release fixes sooner, presumably not relying on the upstream distro having released their fixes first?)
> > >
> > > Besides being a "downstream or a rebuild of another distro", CIQ has its
> > > LTS branches and Rocky Linux has its additional and replacement packages
> > > via the SIGs. Security maintenance for these should be provided by CIQ
> > > and Rocky Linux.
>
> Special interest groups cannot count because they are intended to be
> public community projects. Unless you're saying that all Rocky Linux
> SIGs are shadows of CIQ work that can be held back for public consumption,
> that is effectively out of scope for consideration.
I note that you're not arguing against CIQ LTS branches being relevant.
Great. As to the SIGs, no, I am not "saying that all Rocky Linux SIGs
are shadows of CIQ work", but there is overlap in people involved and
occasionally CIQ can help prepare important security updates "that can
be held back for public consumption" until the coordinated release date.
> Otherwise, Fedora and CentOS SIGs would be eligible for linux-distros@
> (and my understanding is that they are not).
Current membership criteria start with "Be an actively maintained
Unix-like operating system distro with substantial use of Open Source
components", so a SIG like these isn't eligible because of not being a
complete "operating system distro". However, if Red Hat would manage to
contribute to their related SIGs' preparedness without breaking list
rules, that would be allowed. Specifically, the rules allow to share
information "with others within your distro's team based on their
need-to-know" as long as they also accepted the rules. So if a person
directly with the distro takes a SIG's package, prepares an update, and
only makes it available to the SIG on the CRD, that's fine. Ditto if
it's the same person wearing two hats.
Similarly, Rocky Linux SIGs are not eligible on their own, but the
distro's security team can contribute to them as the rules permit.
> I will also note that CIQ/RESF/Rocky have made public statements about
> maintaining the pure-rebuild nature of the distribution, which I
> believe summarily disqualifies it.
>
> https://ciq.com/blog/rhel-changes-what-it-means-for-ciq/
> https://rockylinux.org/news/2023-06-22-press-release/
> https://rockylinux.org/news/brave-new-world-path-forward/
> https://rockylinux.org/news/keeping-open-source-open/
This applies to the main Rocky Linux distribution. Yes, with only that
one distribution we'd not have "convincing additional justification of
how the list membership would enable you to release fixes sooner" and
thus be disqualified. However, the existence of CIQ LTS branches and of
Rocky Linux SIGs changes that, as the team to be subscribed(*) is to
provide security maintenance for these, and via the Security SIG also
optional mitigations and early fixes for Rocky Linux.
(*) or who I'd initially be relaying specific bits of info to, based on
their need-to-know and indeed understanding and acceptance of the terms
> CloudLinux's membership was based on the fact that they replaced and
> maintained a very large chunk of the distribution for their own
> purpose. They used a RHEL compatible userland, but most of the server
> software stacks and the kernel were replaced with their own builds.
> They wanted access for the maintenance of that stuff, which is very
> reasonable.
>
> Rocky/CIQ has not demonstrated a similar need from my point of view.
Fair enough. I wish more about CIQ's offerings were available publicly.
However, I feel that what I described above is sufficient for the
purpose of linux-distros membership.
> > > Also, CentOS was once a member.
>
> CentOS was a very strange project in that it operated in a very closed
> fashion and it was difficult for volunteers to join the effort. I do
> not pretend to know if the current rules existed when CentOS was a
> member, but I would not accept them today on the basis that it's
> effectively a RHEL build.
Yes, CentOS' membership pre-dates the current specific criteria, and I
don't know if CentOS would be accepted today. As a rebuild only, it
would not be, but if they offered to provide security maintenance for
extras from their SIGs, maybe.
Anyway, the current CentOS Stream is a project of Red Hat, and it's up
to Red Hat to provide security maintenance for it or not, including
using information obtained via linux-distros as the rules permit.
> Fedora is not a member because there is no mechanism in the project to
> hide anything from the community. For this reason, I have not
> considered joining as a representative of CentOS Hyperscale, Mageia,
> or Fedora (all distributions that I do participate in security
> response for).
Thank you for sharing your perspective on this. Makes sense. From my
perspective, Fedora isn't explicitly a member because it does not need
to be, with that kind of preparedness provided by Red Hat.
We could go into hair-splitting and require that RHEL, Fedora, and
CentOS Stream be individually listed as member distros. Maybe this
would actually help some issue reporters understand which distros
they're notifying, so it isn't necessarily unreasonable. However, in
terms of people subscribed I think it'd be just Red Hat folks wearing a
variety of project hats anyway, so is easier to manage as one member.
> While I certainly recognize you and value your contributions
> over the years, I do not feel that you alone is sufficient for
> Rocky/CIQ to be accepted onto linux-distros@.
Of course not - the new member also needs to meet the criteria, and I
think it does (even if barely so for the not-only-rebuild one).
Alexander
Powered by blists - more mailing lists
Please check out the
Open Source Software Security Wiki, which is counterpart to this
mailing list.
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
