Factory -> Rolling releases: What does it mean for the kernel package?

[Jeff Mahoney]

Hi folks -

The openSUSE team recently announced[1] that openSUSE Factory is
moving to a rolling development model, similar to how Tumbleweed has
functioned for some time. I think this will do wonders for the
stability of Factory and hopefully make it interesting again for those
of us who used to sync nightly and zypper dup first thing every morning.

It does bring a few questions with it, though. How do we handle the
kernel with a rolling release? Having multiple outstanding versions
isn't an issue unique to the kernel, but few other projects are as
large and have as much churn between releases. The folks trying to
support the kernel as best we can are short on time as it is, so I'm
concerned about establishing parameters for which bug reports will be
considered valid[2]. I'd like to define what those are so that anyone
going through bug reports (even someone not actively involved in
kernel development or maintenance) and see which ones can be closed
or, at least, put on hold until they've been reproduced with the
latest version.

1) How do we handle releases of the Factory kernel?
2) What releases will be considered "supported" simultaneously?

For 1), I propose we make it official that Kernel:stable is now the
external repository for openSUSE Factory. Even though Factory has
already been pulling from Kernel:stable, making it an official thing
means that security fixes and other backports are actively added to it
rather than it being Jiri's side project. Factory would only contain
point releases moving forward and release candidates would be confined
to Kernel:HEAD. As a result of the change, I'd start adding -rc1
updates to Kernel:HEAD rather than waiting until -rc2 as I have in the
past. It also means that there would no longer be a delay for things
like Xen and ARM in Factory.

As for 2), I'd like to hear suggestions. With only so much time to
devote to supporting the Factory kernel, we'll need to find a balance
between convenience for users and bugs actually getting fixed. We have
options between supporting every release (which I would actively
fight) and dropping support for any previous kernel as soon as a new
one is added to the repository. I've added some interested parties to
the CC list, but I'd like to hear from anyone with an opinion.

Related to 2, I'd like to put more effort into trimming down the
number of patches we carry in the openSUSE kernel. Over the years,
we've done a pretty good job of that (compare the number of patches in
a SLES release vs openSUSE), but we can do more. SUSE has already
hired several engineers to work on getting our divergent Xen
implementation into something that is mergeable upstream. Once that
happens, the openSUSE kernel patchset should only contain
security/stability backports.

- -Jeff

[1]
https://news.opensuse.org/2014/07/29/factory-rolling-release/#more-18251
[2] I'll be the first to admit we're open to criticism in how fast
openSUSE bugs are handled already

- --
Jeff Mahoney
SUSE Labs

[Takashi Iwai]

At Thu, 31 Jul 2014 10:32:06 -0400,

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi folks -
The openSUSE team recently announced[1] that openSUSE Factory is
moving to a rolling development model, similar to how Tumbleweed has
functioned for some time. I think this will do wonders for the
stability of Factory and hopefully make it interesting again for those
of us who used to sync nightly and zypper dup first thing every morning.
It does bring a few questions with it, though. How do we handle the
kernel with a rolling release? Having multiple outstanding versions
isn't an issue unique to the kernel, but few other projects are as
large and have as much churn between releases. The folks trying to
support the kernel as best we can are short on time as it is, so I'm
concerned about establishing parameters for which bug reports will be
considered valid[2]. I'd like to define what those are so that anyone
going through bug reports (even someone not actively involved in
kernel development or maintenance) and see which ones can be closed
or, at least, put on hold until they've been reproduced with the
latest version.
1) How do we handle releases of the Factory kernel?
2) What releases will be considered "supported" simultaneously?
For 1), I propose we make it official that Kernel:stable is now the
external repository for openSUSE Factory. Even though Factory has
already been pulling from Kernel:stable, making it an official thing
means that security fixes and other backports are actively added to it
rather than it being Jiri's side project. Factory would only contain
point releases moving forward and release candidates would be confined
to Kernel:HEAD. As a result of the change, I'd start adding -rc1
updates to Kernel:HEAD rather than waiting until -rc2 as I have in the
past. It also means that there would no longer be a delay for things
like Xen and ARM in Factory.

I'm for Kernel:stable, too. Taking rc kernel doesn't fit with
"releases".

As for 2), I'd like to hear suggestions. With only so much time to
devote to supporting the Factory kernel, we'll need to find a balance
between convenience for users and bugs actually getting fixed. We have
options between supporting every release (which I would actively
fight) and dropping support for any previous kernel as soon as a new
one is added to the repository. I've added some interested parties to
the CC list, but I'd like to hear from anyone with an opinion.

We can follow the rule others use, e.g. defining the expiration period
when reporter doesn't reproduce the issue with the updated kernel,
holding it opened for a certain period (say one month) or after three
strikes. After the expiration, close bugs as NORESPONSE
(automatically if possible).

Related to 2, I'd like to put more effort into trimming down the
number of patches we carry in the openSUSE kernel. Over the years,
we've done a pretty good job of that (compare the number of patches in
a SLES release vs openSUSE), but we can do more. SUSE has already
hired several engineers to work on getting our divergent Xen
implementation into something that is mergeable upstream. Once that
happens, the openSUSE kernel patchset should only contain
security/stability backports.

I'm pretty optimistic about this. The current amount of patches is
already manageable, IMO. But, of course, I'd love to see more
reduction there, too.

Where we're at it: what about ABI, e.g. for infamous graphics drivers?


thanks,

Takashi

[Jeff Mahoney]

Post by Takashi Iwai

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi folks -
The openSUSE team recently announced[1] that openSUSE Factory is
moving to a rolling development model, similar to how Tumbleweed
has functioned for some time. I think this will do wonders for
the stability of Factory and hopefully make it interesting again
for those of us who used to sync nightly and zypper dup first
thing every morning.
It does bring a few questions with it, though. How do we handle
the kernel with a rolling release? Having multiple outstanding
versions isn't an issue unique to the kernel, but few other
projects are as large and have as much churn between releases.
The folks trying to support the kernel as best we can are short
on time as it is, so I'm concerned about establishing parameters
for which bug reports will be considered valid[2]. I'd like to
define what those are so that anyone going through bug reports
(even someone not actively involved in kernel development or
maintenance) and see which ones can be closed or, at least, put
on hold until they've been reproduced with the latest version.
1) How do we handle releases of the Factory kernel? 2) What
releases will be considered "supported" simultaneously?
For 1), I propose we make it official that Kernel:stable is now
the external repository for openSUSE Factory. Even though Factory
has already been pulling from Kernel:stable, making it an
official thing means that security fixes and other backports are
actively added to it rather than it being Jiri's side project.
Factory would only contain point releases moving forward and
release candidates would be confined to Kernel:HEAD. As a result
of the change, I'd start adding -rc1 updates to Kernel:HEAD
rather than waiting until -rc2 as I have in the past. It also
means that there would no longer be a delay for things like Xen
and ARM in Factory.

I'm for Kernel:stable, too. Taking rc kernel doesn't fit with
"releases".

As for 2), I'd like to hear suggestions. With only so much time
to devote to supporting the Factory kernel, we'll need to find a
balance between convenience for users and bugs actually getting
fixed. We have options between supporting every release (which I
would actively fight) and dropping support for any previous
kernel as soon as a new one is added to the repository. I've
added some interested parties to the CC list, but I'd like to
hear from anyone with an opinion.

We can follow the rule others use, e.g. defining the expiration
period when reporter doesn't reproduce the issue with the updated
kernel, holding it opened for a certain period (say one month) or
after three strikes. After the expiration, close bugs as
NORESPONSE (automatically if possible).

Could you elaborate here? When it is it ok to say, "sorry, just use
the newer kernel?"

Post by Takashi Iwai

Related to 2, I'd like to put more effort into trimming down the
number of patches we carry in the openSUSE kernel. Over the
years, we've done a pretty good job of that (compare the number
of patches in a SLES release vs openSUSE), but we can do more.
SUSE has already hired several engineers to work on getting our
divergent Xen implementation into something that is mergeable
upstream. Once that happens, the openSUSE kernel patchset should
only contain security/stability backports.

I'm pretty optimistic about this. The current amount of patches
is already manageable, IMO. But, of course, I'd love to see more
reduction there, too.
Where we're at it: what about ABI, e.g. for infamous graphics
drivers?

We'd probably freeze it when pulling from the master to the stable branch.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[Takashi Iwai]

At Thu, 31 Jul 2014 10:55:49 -0400,

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Takashi Iwai

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi folks -
The openSUSE team recently announced[1] that openSUSE Factory is
moving to a rolling development model, similar to how Tumbleweed
has functioned for some time. I think this will do wonders for
the stability of Factory and hopefully make it interesting again
for those of us who used to sync nightly and zypper dup first
thing every morning.
It does bring a few questions with it, though. How do we handle
the kernel with a rolling release? Having multiple outstanding
versions isn't an issue unique to the kernel, but few other
projects are as large and have as much churn between releases.
The folks trying to support the kernel as best we can are short
on time as it is, so I'm concerned about establishing parameters
for which bug reports will be considered valid[2]. I'd like to
define what those are so that anyone going through bug reports
(even someone not actively involved in kernel development or
maintenance) and see which ones can be closed or, at least, put
on hold until they've been reproduced with the latest version.
1) How do we handle releases of the Factory kernel? 2) What
releases will be considered "supported" simultaneously?
For 1), I propose we make it official that Kernel:stable is now
the external repository for openSUSE Factory. Even though Factory
has already been pulling from Kernel:stable, making it an
official thing means that security fixes and other backports are
actively added to it rather than it being Jiri's side project.
Factory would only contain point releases moving forward and
release candidates would be confined to Kernel:HEAD. As a result
of the change, I'd start adding -rc1 updates to Kernel:HEAD
rather than waiting until -rc2 as I have in the past. It also
means that there would no longer be a delay for things like Xen
and ARM in Factory.

I'm for Kernel:stable, too. Taking rc kernel doesn't fit with
"releases".

As for 2), I'd like to hear suggestions. With only so much time
to devote to supporting the Factory kernel, we'll need to find a
balance between convenience for users and bugs actually getting
fixed. We have options between supporting every release (which I
would actively fight) and dropping support for any previous
kernel as soon as a new one is added to the repository. I've
added some interested parties to the CC list, but I'd like to
hear from anyone with an opinion.

We can follow the rule others use, e.g. defining the expiration
period when reporter doesn't reproduce the issue with the updated
kernel, holding it opened for a certain period (say one month) or
after three strikes. After the expiration, close bugs as
NORESPONSE (automatically if possible).

Could you elaborate here? When it is it ok to say, "sorry, just use
the newer kernel?"

IMO, one month would be seen as long enough. Reporter can reopen the
bug at any time if it's reproduced with the updated kernel.

Or, three strikes, i.e. keep opened for two releases and then close.

Post by Takashi Iwai

Related to 2, I'd like to put more effort into trimming down the
number of patches we carry in the openSUSE kernel. Over the
years, we've done a pretty good job of that (compare the number
of patches in a SLES release vs openSUSE), but we can do more.
SUSE has already hired several engineers to work on getting our
divergent Xen implementation into something that is mergeable
upstream. Once that happens, the openSUSE kernel patchset should
only contain security/stability backports.

I'm pretty optimistic about this. The current amount of patches
is already manageable, IMO. But, of course, I'd love to see more
reduction there, too.
Where we're at it: what about ABI, e.g. for infamous graphics drivers?

We'd probably freeze it when pulling from the master to the stable branch.

kABI is often PITA, and I'm not sure how solid we should keep it.


Takashi

[Jeff Mahoney]

Post by Takashi Iwai

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Post by Takashi Iwai

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi folks -
The openSUSE team recently announced[1] that openSUSE Factory
is moving to a rolling development model, similar to how
Tumbleweed has functioned for some time. I think this will do
wonders for the stability of Factory and hopefully make it
interesting again for those of us who used to sync nightly
and zypper dup first thing every morning.
It does bring a few questions with it, though. How do we
handle the kernel with a rolling release? Having multiple
outstanding versions isn't an issue unique to the kernel, but
few other projects are as large and have as much churn
between releases. The folks trying to support the kernel as
best we can are short on time as it is, so I'm concerned
about establishing parameters for which bug reports will be
considered valid[2]. I'd like to define what those are so
that anyone going through bug reports (even someone not
actively involved in kernel development or maintenance) and
see which ones can be closed or, at least, put on hold until
they've been reproduced with the latest version.
1) How do we handle releases of the Factory kernel? 2) What
releases will be considered "supported" simultaneously?
For 1), I propose we make it official that Kernel:stable is
now the external repository for openSUSE Factory. Even though
Factory has already been pulling from Kernel:stable, making
it an official thing means that security fixes and other
backports are actively added to it rather than it being
Jiri's side project. Factory would only contain point
releases moving forward and release candidates would be
confined to Kernel:HEAD. As a result of the change, I'd start
adding -rc1 updates to Kernel:HEAD rather than waiting until
-rc2 as I have in the past. It also means that there would no
longer be a delay for things like Xen and ARM in Factory.

I'm for Kernel:stable, too. Taking rc kernel doesn't fit with
"releases".

As for 2), I'd like to hear suggestions. With only so much
time to devote to supporting the Factory kernel, we'll need
to find a balance between convenience for users and bugs
actually getting fixed. We have options between supporting
every release (which I would actively fight) and dropping
support for any previous kernel as soon as a new one is added
to the repository. I've added some interested parties to the
CC list, but I'd like to hear from anyone with an opinion.

We can follow the rule others use, e.g. defining the
expiration period when reporter doesn't reproduce the issue
with the updated kernel, holding it opened for a certain period
(say one month) or after three strikes. After the expiration,
close bugs as NORESPONSE (automatically if possible).

Could you elaborate here? When it is it ok to say, "sorry, just
use the newer kernel?"

IMO, one month would be seen as long enough. Reporter can reopen
the bug at any time if it's reproduced with the updated kernel.
Or, three strikes, i.e. keep opened for two releases and then
close.

Post by Takashi Iwai

Related to 2, I'd like to put more effort into trimming down
the number of patches we carry in the openSUSE kernel. Over
the years, we've done a pretty good job of that (compare the
number of patches in a SLES release vs openSUSE), but we can
do more. SUSE has already hired several engineers to work on
getting our divergent Xen implementation into something that
is mergeable upstream. Once that happens, the openSUSE kernel
patchset should only contain security/stability backports.

I'm pretty optimistic about this. The current amount of
patches is already manageable, IMO. But, of course, I'd love
to see more reduction there, too.
Where we're at it: what about ABI, e.g. for infamous graphics drivers?

We'd probably freeze it when pulling from the master to the
stable branch.

kABI is often PITA, and I'm not sure how solid we should keep it.

If our only concern is truly just the graphics drivers, we could
probably collect the ABI dependencies from them and limit the kABI
checker to those interfaces. Otherwise, yeah, the full kABI check is
overkill.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[Takashi Iwai]

At Thu, 31 Jul 2014 11:21:17 -0400,

Post by Jeff Mahoney

Post by Takashi Iwai

Post by Jeff Mahoney

Post by Takashi Iwai
Where we're at it: what about ABI, e.g. for infamous graphics drivers?

We'd probably freeze it when pulling from the master to the
stable branch.

kABI is often PITA, and I'm not sure how solid we should keep it.

If our only concern is truly just the graphics drivers, we could
probably collect the ABI dependencies from them and limit the kABI
checker to those interfaces. Otherwise, yeah, the full kABI check is
overkill.

The reduced kABI set sounds like a good idea, indeed.


Takashi

[Michal Marek]

Post by Jeff Mahoney

Post by Takashi Iwai

Post by Jeff Mahoney
We'd probably freeze it when pulling from the master to the
stable branch.

kABI is often PITA, and I'm not sure how solid we should keep it.

If our only concern is truly just the graphics drivers, we could
probably collect the ABI dependencies from them and limit the kABI
checker to those interfaces. Otherwise, yeah, the full kABI check is
overkill.

Agreed. The other KMPs are part of Factory and rebuilt automatically. We
just need to keep Kernel:stable and openSUSE:Factory as close as possible.

Michal

[Michal Marek]

Post by Jeff Mahoney
1) How do we handle releases of the Factory kernel?
2) What releases will be considered "supported" simultaneously?
For 1), I propose we make it official that Kernel:stable is now the
external repository for openSUSE Factory. Even though Factory has
already been pulling from Kernel:stable, making it an official thing
means that security fixes and other backports are actively added to it
rather than it being Jiri's side project.

OK. I added factory-maintainers to the Kernel:stable project, so that
Coolo can pull from there. Or shall I automatically submit new kernels
to devel:openSUSE:Factory:kernel?

Post by Jeff Mahoney
As for 2), I'd like to hear suggestions. With only so much time to
devote to supporting the Factory kernel, we'll need to find a balance
between convenience for users and bugs actually getting fixed. We have
options between supporting every release (which I would actively
fight) and dropping support for any previous kernel as soon as a new
one is added to the repository. I've added some interested parties to
the CC list, but I'd like to hear from anyone with an opinion.

IMO having bugreportes reproduce with the latest stable kernel is not
asking too much. The packages are conveniently available.

Michal

[Takashi Iwai]

At Thu, 31 Jul 2014 17:29:28 +0200,

Post by Michal Marek

Post by Jeff Mahoney
As for 2), I'd like to hear suggestions. With only so much time to
devote to supporting the Factory kernel, we'll need to find a balance
between convenience for users and bugs actually getting fixed. We have
options between supporting every release (which I would actively
fight) and dropping support for any previous kernel as soon as a new
one is added to the repository. I've added some interested parties to
the CC list, but I'd like to hear from anyone with an opinion.

IMO having bugreportes reproduce with the latest stable kernel is not
asking too much. The packages are conveniently available.

Right. But now one thing came to my mind: is there any backup of the
old released (kernel) packages? For the released products in the
past, we keep every updated package. But with a rolling release...?

Having the old packages is often really helpful for debugging, it can
narrow down the range.


Takashi

[Jeff Mahoney]

Post by Takashi Iwai

Post by Michal Marek

Post by Jeff Mahoney
As for 2), I'd like to hear suggestions. With only so much time
to devote to supporting the Factory kernel, we'll need to find
a balance between convenience for users and bugs actually
getting fixed. We have options between supporting every release
(which I would actively fight) and dropping support for any
previous kernel as soon as a new one is added to the
repository. I've added some interested parties to the CC list,
but I'd like to hear from anyone with an opinion.

IMO having bugreportes reproduce with the latest stable kernel is
not asking too much. The packages are conveniently available.

Right. But now one thing came to my mind: is there any backup of
the old released (kernel) packages? For the released products in
the past, we keep every updated package. But with a rolling
release...?
Having the old packages is often really helpful for debugging, it
can narrow down the range.

Even more importantly, having old packages is where we get the
debuginfo. We'll need to sort that out too.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[Michal Marek]

Post by Takashi Iwai
At Thu, 31 Jul 2014 17:29:28 +0200,

Post by Michal Marek

Post by Jeff Mahoney
As for 2), I'd like to hear suggestions. With only so much time to
devote to supporting the Factory kernel, we'll need to find a balance
between convenience for users and bugs actually getting fixed. We have
options between supporting every release (which I would actively
fight) and dropping support for any previous kernel as soon as a new
one is added to the repository. I've added some interested parties to
the CC list, but I'd like to hear from anyone with an opinion.

IMO having bugreportes reproduce with the latest stable kernel is not
asking too much. The packages are conveniently available.

Right. But now one thing came to my mind: is there any backup of the
old released (kernel) packages? For the released products in the
past, we keep every updated package. But with a rolling release...?
Having the old packages is often really helpful for debugging, it can
narrow down the range.

I've been running a script to keep an archive of the kernels built in
the internal buildservice for some time. I just need to export it
somehow at least internally. But if the buildservice could do it for the
Factory packages, that would be great as well.

Michal

[Stephan Kulow]

Post by Michal Marek
OK. I added factory-maintainers to the Kernel:stable project, so that
Coolo can pull from there. Or shall I automatically submit new kernels
to devel:openSUSE:Factory:kernel?

devel:openSUSE:Factory:kernel is just a workaround for the switch
between HEAD and stable.

Once stable is settle, we can remove the devel: prj again.

Greetings, Stephan

[Michal Marek]

Post by Stephan Kulow

Post by Michal Marek
OK. I added factory-maintainers to the Kernel:stable project, so that
Coolo can pull from there. Or shall I automatically submit new kernels
to devel:openSUSE:Factory:kernel?

devel:openSUSE:Factory:kernel is just a workaround for the switch
between HEAD and stable.
Once stable is settle, we can remove the devel: prj again.

OK, and shall I then autosubmit kernels from Kernel:stable to
openSUSE:Factory?

Thanks,
Michal

[Jeff Mahoney]

Post by Michal Marek

Post by Stephan Kulow

Post by Michal Marek
OK. I added factory-maintainers to the Kernel:stable project,
so that Coolo can pull from there. Or shall I automatically
submit new kernels to devel:openSUSE:Factory:kernel?

devel:openSUSE:Factory:kernel is just a workaround for the
switch between HEAD and stable.
Once stable is settle, we can remove the devel: prj again.

OK, and shall I then autosubmit kernels from Kernel:stable to
openSUSE:Factory?

That's what I had in mind.

We still need to sort out when we pull from master into stable. David
Sterba suggested waiting until the first -stable update for the next
point release. So, e.g. Factory uses 3.15.X now and we wouldn't move
to 3.16 until 3.16.1 was released.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[Stephan Kulow]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Jeff Mahoney
That's what I had in mind.
We still need to sort out when we pull from master into stable.
David Sterba suggested waiting until the first -stable update for
the next point release. So, e.g. Factory uses 3.15.X now and we
wouldn't move to 3.16 until 3.16.1 was released.

I wouldn't do that. Factory is actually using 3.16.rc7 and beside the
mkfs.btrfs OOPS I have no problem with it. I can live well without the
RCs in Factory, but I would like to see problems as early as possible
- - with the hope someone is actually fixing them too.

Greetings, Stephan

- --
Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die
ganze Welt an Arsch offen hat, oder grad deswegn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlPbH4YACgkQwFSBhlBjoJYRywCggJJlEgaHwWsTjoY3Xbu4S3P4
HR4AnA8qnTC2cT5L8kvWViBDsHbDx4oJ
=aQs2
-----END PGP SIGNATURE-----

[Jeff Mahoney]

Post by Stephan Kulow

Post by Jeff Mahoney
That's what I had in mind.
We still need to sort out when we pull from master into stable.
David Sterba suggested waiting until the first -stable update
for the next point release. So, e.g. Factory uses 3.15.X now and
we wouldn't move to 3.16 until 3.16.1 was released.

I wouldn't do that. Factory is actually using 3.16.rc7 and beside
the mkfs.btrfs OOPS I have no problem with it. I can live well
without the RCs in Factory, but I would like to see problems as
early as possible - with the hope someone is actually fixing them
too.

I must be missing the goal of the new rolling update Factory then. My
impression was that it was effectively going to be what Tumbleweed has
been in the past and that it shouldn't be a dumping ground for
prerelease software anymore.

Users who want to test an -rcX kernel will still be able to via
Kernel:HEAD.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[Stephan Kulow]

Post by Stephan Kulow
I wouldn't do that. Factory is actually using 3.16.rc7 and beside
the mkfs.btrfs OOPS I have no problem with it. I can live well
without the RCs in Factory, but I would like to see problems as
early as possible - with the hope someone is actually fixing them
too.
I must be missing the goal of the new rolling update Factory then. My
impression was that it was effectively going to be what Tumbleweed has
been in the past and that it shouldn't be a dumping ground for
prerelease software anymore.
Users who want to test an -rcX kernel will still be able to via
Kernel:HEAD.

Yes. I was arguing in favor of .0 releases - saying that so far rc7
never caused a problem,
.0 releases can't be worse.

Greetigns, Stephan

[Jeff Mahoney]

Post by Stephan Kulow

Post by Stephan Kulow
I wouldn't do that. Factory is actually using 3.16.rc7 and
beside the mkfs.btrfs OOPS I have no problem with it. I can live
well without the RCs in Factory, but I would like to see problems
as early as possible - with the hope someone is actually fixing
them too. I must be missing the goal of the new rolling update
Factory then. My impression was that it was effectively going to
be what Tumbleweed has been in the past and that it shouldn't be
a dumping ground for prerelease software anymore.
Users who want to test an -rcX kernel will still be able to via
Kernel:HEAD.

Yes. I was arguing in favor of .0 releases - saying that so far
rc7 never caused a problem, .0 releases can't be worse.

Ok, thanks. Sounds like we're in agreement.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[David Sterba]

Post by Stephan Kulow

Post by Jeff Mahoney
That's what I had in mind.
We still need to sort out when we pull from master into stable.
David Sterba suggested waiting until the first -stable update for
the next point release. So, e.g. Factory uses 3.15.X now and we
wouldn't move to 3.16 until 3.16.1 was released.

I wouldn't do that. Factory is actually using 3.16.rc7 and beside the
mkfs.btrfs OOPS I have no problem with it. I can live well without the
RCs in Factory, but I would like to see problems as early as possible
- - with the hope someone is actually fixing them too.

My idea is to be more conservative when the kernel is shipped to large
number of users who do just 'zypper dup' and expect it to ~work. I hope
that this will improve usability of a rolling distro in the long term

People who want to test the .0 release could install from the
Kernel:stable repository directly (I'll do that). When I'm aware of
potential problems with new kernel, I'm also prepared to do a rollback
to the last working kernel: setting up multiversion and keeping multiple
kernels installed, bootloader points to the working kernel and I
boot-once from the new one should any problems arise during boot.

This excercise is natural to me because I work on kernel but I'm not
expecting an average Factory user to set this up by default. The bad
scenario I'd like to avoid is "I've updated Factory/kernel and it does
not boot and I don't have a fallback". After such experience the users
may leave Factory or just avoid updating kernel when there is a new
release.

[Stephan Kulow]

Post by David Sterba
My idea is to be more conservative when the kernel is shipped to large
number of users who do just 'zypper dup' and expect it to ~work. I
hope that this will improve usability of a rolling distro in the long
term People who want to test the .0 release could install from the
Kernel:stable repository directly (I'll do that). When I'm aware of
potential problems with new kernel, I'm also prepared to do a rollback
to the last working kernel: setting up multiversion and keeping
multiple kernels installed, bootloader points to the working kernel
and I boot-once from the new one should any problems arise during
boot. This excercise is natural to me because I work on kernel but I'm
not expecting an average Factory user to set this up by default. The
bad scenario I'd like to avoid is "I've updated Factory/kernel and it
does not boot and I don't have a fallback". After such experience the
users may leave Factory or just avoid updating kernel when there is a
new release.

To avoid that we run > 80 tests on that kernel in openQA before factory
is published. And we will
add more over time.

Greetings, Stephan

[Andreas Schwab]

Post by David Sterba
This excercise is natural to me because I work on kernel but I'm not
expecting an average Factory user to set this up by default.

This is already the default, since 12.3.

Andreas.

[Stefan Seyfried]

Post by Stephan Kulow

Post by Jeff Mahoney
That's what I had in mind.
We still need to sort out when we pull from master into stable.
David Sterba suggested waiting until the first -stable update for
the next point release. So, e.g. Factory uses 3.15.X now and we
wouldn't move to 3.16 until 3.16.1 was released.

I wouldn't do that. Factory is actually using 3.16.rc7 and beside the
mkfs.btrfs OOPS I have no problem with it. I can live well without the
RCs in Factory, but I would like to see problems as early as possible
- with the hope someone is actually fixing them too.

Actually if Factory is supposed to be the "rolling release for grandma
and grandpa" then the RCs are not supposed to go there.

People who want to test them can easily add Kernel:HEAD (like I have
done for ages) and get the RCs from there, but still have the option to
easily go back to a working kernel.

I actually lost that possibility with Factory switching to the unstable
kernels lately

(my secret trick is that I have kernel-default from Factory and
kernel-desktop from Kernel:HEAD, and since I never use "zypper dup" but
only "zypper up", they always get updated from their origin repository,
so kernel-default was the "stable, working" kernel before)

[Michal Kubecek]

Post by Stefan Seyfried

Post by Stephan Kulow

Post by Jeff Mahoney
That's what I had in mind.
We still need to sort out when we pull from master into stable.
David Sterba suggested waiting until the first -stable update for
the next point release. So, e.g. Factory uses 3.15.X now and we
wouldn't move to 3.16 until 3.16.1 was released.

I wouldn't do that. Factory is actually using 3.16.rc7 and beside the
mkfs.btrfs OOPS I have no problem with it. I can live well without
the RCs in Factory, but I would like to see problems as early as
possible - with the hope someone is actually fixing them too.

Actually if Factory is supposed to be the "rolling release for grandma
and grandpa" then the RCs are not supposed to go there.

As Coolo already explained, this was not meant as a proposal to use RC
in Factory but as an argument for moving to next version with 3.x.0
rather than waiting for 3.x.1

Michal Kubeček

[Jean Delvare]

Hi Jeff,

Post by Jeff Mahoney

Post by Michal Marek
OK, and shall I then autosubmit kernels from Kernel:stable to
openSUSE:Factory?

That's what I had in mind.
We still need to sort out when we pull from master into stable. David
Sterba suggested waiting until the first -stable update for the next
point release. So, e.g. Factory uses 3.15.X now and we wouldn't move
to 3.16 until 3.16.1 was released.

I would start with 3.16.0. Avoiding the .0 because it might not be good
enough does not work, if everybody does that then .1 won't be good
enough either.

My only concern is 3rd party kernel drivers, in my experience it can
take some time before they are adjusted to build and work with the most
recent kernel.

[Stephan Kulow]

Post by Michal Marek

Post by Stephan Kulow

Post by Michal Marek
OK. I added factory-maintainers to the Kernel:stable project, so that
Coolo can pull from there. Or shall I automatically submit new kernels
to devel:openSUSE:Factory:kernel?

devel:openSUSE:Factory:kernel is just a workaround for the switch
between HEAD and stable.
Once stable is settle, we can remove the devel: prj again.

OK, and shall I then autosubmit kernels from Kernel:stable to
openSUSE:Factory?

I SRed 3.16.0 from Kernel:HEAD now and also filed a changedevel request
to set Kernel:stable as devel prj - once 3.16.0 is either in factory or
Kernel:stable, we can approve that and then Kernel:HEAD is purely
experimental ;)

Greetings, Stephan

[Jeff Mahoney]

Post by Stephan Kulow

Post by Michal Marek

Post by Stephan Kulow

Post by Michal Marek
OK. I added factory-maintainers to the Kernel:stable project,
so that Coolo can pull from there. Or shall I automatically
submit new kernels to devel:openSUSE:Factory:kernel?

devel:openSUSE:Factory:kernel is just a workaround for the
switch between HEAD and stable.
Once stable is settle, we can remove the devel: prj again.

OK, and shall I then autosubmit kernels from Kernel:stable to
openSUSE:Factory?

I SRed 3.16.0 from Kernel:HEAD now and also filed a changedevel
request to set Kernel:stable as devel prj - once 3.16.0 is either
in factory or Kernel:stable, we can approve that and then
Kernel:HEAD is purely experimental ;)

Thanks, Coolo.

Jiri, how do you want to handle the syncing between master and stable
in the future? Once -rc1 is released, I'll start merging it into master.

- -Jeff

- --
Jeff Mahoney
SUSE Labs

[Jiri Slaby]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Jeff Mahoney

Post by Stephan Kulow
I SRed 3.16.0 from Kernel:HEAD now and also filed a changedevel
request to set Kernel:stable as devel prj - once 3.16.0 is
either in factory or Kernel:stable, we can approve that and then
Kernel:HEAD is purely experimental ;)

Kernel:stable is now at 3.16.1. Do you want me to create SRs when we
have something new and (hopefully) working?

Post by Jeff Mahoney
Jiri, how do you want to handle the syncing between master and
stable in the future? Once -rc1 is released, I'll start merging it
into master.

I would continue with the current scenario: we start merging master to
stable when final version X is put into master, until (X+1)-rc1 is put
there.

- --
js
suse labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJT8xXFAAoJEL0lsQQGtHBJ0mAQAJ5nRecNZPRKVzFq6SX5pnac
+hhmDzt8vpqoPai0H5VBwDxkrdI04si1yfe7ho1wjKPu6lNkxSIGEtWnxuudcxS/
mirWFByxs1y0M2pYPb1fDeCj4h0Fq1wPCz0RqJKdlvPRmdW9HZ1UGFNJWFi+wIB0
zdUK83teVGNP/hFawdQclU0uUwbThCQbCRFgiFJinkyry5dNC0UfQhuEoGAyy6QI
WI19FZqDs8+f7qHXJBWtzvcnD5peDpHvjJIB4qdbJ1bMNBtA0fJNum3ckeSFXsRt
51AlXzBGvkU5GHz11cyWt0kVRWVdtx6cle3/L1H7NWiH6Br8HxFDQ7hwmo+aEVL/
DAY9rRsnqcK9/gULfXUmjrDvekEo5soYMpubPg7rIhNnBi6/igr1kxGtrKz79uxs
JUbioWfrvs9tTdkT+ePeXMq8mMYM97pz989X5cd+VvlP1MK2WG4Spt6uNgplEOVX
RGrh9+jS6i3eq0nM2M5S92iCIXBikkrD4LspIEPJR/2Bj1Mh2uvqG52700JjW9f5
G9UsmqmMFYSl5sqNY3OfxGo1xC2UNvqmEYdt9Mj50qnR41BfkgYXLTfh2TAl6pMN
f/+oUOIxAHa9J1LGAs5COnKhNTmG4YoKXvSjrGndE0T/4Z+uYzf3h8MaC5RF0tyj
Iq/uWBV5amnClBT2eA0Y
=tIem
-----END PGP SIGNATURE-----

[Stephan Kulow]

Post by Jiri Slaby

Post by Stephan Kulow
I SRed 3.16.0 from Kernel:HEAD now and also filed a changedevel
request to set Kernel:stable as devel prj - once 3.16.0 is
either in factory or Kernel:stable, we can approve that and
then Kernel:HEAD is purely experimental ;)

Kernel:stable is now at 3.16.1. Do you want me to create SRs when
we have something new and (hopefully) working?

Can this be done please? Right now factory-maintainer script submits
the packages and factory-repo-checker script denies them because they
don't come from a valid devel project (i.e. one that builds against
factory).

So either create a new repo or a new project or whatever - but please
fix this situation.

Greetings, Stephan

[Jiri Slaby]

Post by Stephan Kulow

Post by Jiri Slaby

Post by Stephan Kulow
I SRed 3.16.0 from Kernel:HEAD now and also filed a changedevel
request to set Kernel:stable as devel prj - once 3.16.0 is
either in factory or Kernel:stable, we can approve that and
then Kernel:HEAD is purely experimental ;)

Kernel:stable is now at 3.16.1. Do you want me to create SRs when
we have something new and (hopefully) working?

Can this be done please? Right now factory-maintainer script submits
the packages and factory-repo-checker script denies them because they
don't come from a valid devel project (i.e. one that builds against
factory).

Provided, we decided the kernel to be the base for the factory, I
inclined in the end to switch to build against factory. So I committed
that change to the repo and changed that manually in the repo for today.

thanks,

[Luis R. Rodriguez]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi folks -
The openSUSE team recently announced[1] that openSUSE Factory is
moving to a rolling development model, similar to how Tumbleweed has
functioned for some time. I think this will do wonders for the
stability of Factory and hopefully make it interesting again for those
of us who used to sync nightly and zypper dup first thing every morning.
It does bring a few questions with it, though. How do we handle the
kernel with a rolling release? Having multiple outstanding versions
isn't an issue unique to the kernel, but few other projects are as
large and have as much churn between releases. The folks trying to
support the kernel as best we can are short on time as it is, so I'm
concerned about establishing parameters for which bug reports will be
considered valid[2]. I'd like to define what those are so that anyone
going through bug reports (even someone not actively involved in
kernel development or maintenance) and see which ones can be closed
or, at least, put on hold until they've been reproduced with the
latest version.
1) How do we handle releases of the Factory kernel?
2) What releases will be considered "supported" simultaneously?
For 1), I propose we make it official that Kernel:stable is now the
external repository for openSUSE Factory. Even though Factory has
already been pulling from Kernel:stable,

Does Kernel:stable always follow Linus' tree? Or does it wait and skip?

making it an official thing
means that security fixes and other backports are actively added to it
rather than it being Jiri's side project. Factory would only contain
point releases moving forward and release candidates would be confined
to Kernel:HEAD.

Would we have packages say on Kernel:HEAD to help folks on Kernel:stable
looking for a new driver / fix? By definition Kernel:stable will have to wait
until Linus' tree gets a fix merged before it trickles down unless we cherry
pick. If we want to reduce work on Kernel:stable having a pointer to
Kernel:HEAD rpms could be a way to get issues resolved a bit quicker once
submitted. I'm saying we could do that if we wanted to try to reduce work load
on factory and take advantage of existing development models which can help
users. Otherwise its a bit of work, not a lot, but it is some work.

As a result of the change, I'd start adding -rc1
updates to Kernel:HEAD rather than waiting until -rc2 as I have in the
past. It also means that there would no longer be a delay for things
like Xen and ARM in Factory.

Can you elaborate on the existing delay for Xen and ARM in existing
Factory? Would like to make sure I understand that.

As for 2), I'd like to hear suggestions. With only so much time to
devote to supporting the Factory kernel, we'll need to find a balance
between convenience for users and bugs actually getting fixed. We have
options between supporting every release (which I would actively
fight) and dropping support for any previous kernel as soon as a new
one is added to the repository. I've added some interested parties to
the CC list, but I'd like to hear from anyone with an opinion.

To be clear you are against supporting every release contiguously,
so say we 3.17 is released but we were on 3.16.2, we'd immediately
phase out 3.16.2 as soon as we release a package for 3.17.

This makes perfect sense to me. I'd be nice if we had an option to let users
select if they want to opt-in to rc release of packages as that would allow for
a streamlined way for users to get test packages being baked, which I suspect
will be useful for getting users to test before reporting / following up on a
fix. A kind of *self help service*. I wonder if Gentoo has something similar,
Vlastimil? This could be a generally useful thing, not just for the kernel.
Furthermore if we had an option to use a "vanilla" kernel then we could use the
latest and greatest kernel:HEAD with no delta and give users the ability to
report directly to kernel.org. That could be another way of *self helping*, and
getting folks more engaged upstream.

Related to 2, I'd like to put more effort into trimming down the
number of patches we carry in the openSUSE kernel. Over the years,
we've done a pretty good job of that (compare the number of patches in
a SLES release vs openSUSE), but we can do more. SUSE has already
hired several engineers to work on getting our divergent Xen
implementation into something that is mergeable upstream. Once that
happens, the openSUSE kernel patchset should only contain
security/stability backports.

My goal with xen is to kill any delta other than required backported
stuff, and that also should have a respective upstream commit. We
already have gobs of stuff we can nuke, so I wonder if testing it
out on Kernel:stable would be a good starting point?

- -Jeff
[1]
https://news.opensuse.org/2014/07/29/factory-rolling-release/#more-18251
[2] I'll be the first to admit we're open to criticism in how fast
openSUSE bugs are handled already

IMHO a rolling release should try to get users more in line with the
direct core communities involved to help with considerations on
resources on reports. For the kernel I would hope that this
would enable a similar approach, we should strive to keep folks
engaged upstream and the closer that is to the latest release the
easier I think it will be on both us and users.

A few points not yet addressed:

3) The kernel is now tied to systemd, so how are we going to pair those up
on OpenSUSE Factory?

4) Backports: if a user lacks support for a device driver they typically should
be uprading their kernel and if we're on the latest point release that should
suffice for many users. For the latest and greatest hardware though, those
components might only still be being baked. This may also be true if a series
of fixes or enhancements have been made to a device driver but not yet
released. This is where the backports project comes in. In terms of reducing
effort to deal with issues if a user is using a linux-next backport snapshot
and an issue is found they can be sure they can report it upstream to the
respective subsystems mailing lists or file a report on kernel.org. The chance
of the issue being a backport specific bug if on a recent kernel is extremely
low given that backporting consists about less than or = to about 1% of the
code used.

As I see it moving to a rolling release is a good way to get folks more
engaged with their respective upstreams, so any chance at that I think
is something we should take advantage of to help reduce our own work load.
Enabling use of rc releases of the kernel, specially if they are vanilla,
or backports should help with this.

Luis

[Jeff Mahoney]