From Fedora Project Wiki

(mention fedpkg --release is needed)
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
Frequently asked questions about the [[Kernel_Vanilla_Repositories|kernel vanilla repositories for Fedora]].
Frequently asked questions about the [[Kernel_Vanilla_Repositories_Copr|kernel vanilla repositories for Fedora]].


= FAQ for users =
= FAQ for users =


== What's the goal of these repositories? ==
== What is the goal of these repositories? ==


The main ideas is to help upstream development, which in the end will benefit Fedora as well.
The main idea is to help upstream development, which in the end will benefit Fedora as well:


* With the packages from the _kernel-vanilla-mainline_ repository it for example is quite easy to test kernels that are still under development. Bugs thus can be found early and reported and fixed upstream way before they enter a Linux kernel version shipped as a proper update in Fedora.
* With the packages from the @kernel-vanilla/mainline repository it for example is quite easy to test code early that at some point will land in Fedora Linux. In other words: the repository thus allows people to find, report, and eliminate bugs so that they never even hit Fedora. The packages also make it easy to check if a particular bug you encountered is already fixed upstream, as all bug-fixes have to land in mainline before they are considered for backporting to stable and longterm kernels.
* The kernels from the _kernel-vanilla-fedora_ repository make it easy to check if a bug that happens with a Fedora kernel is present upstream or caused by the patches the Fedora developers apply.  
* The situation with @kernel-vanilla/stable repository is similar to the @kernel-vanilla/mainline, you are just not that much ahead of Fedora’s kernels, as they often follow stable quite closely.  
* The kernels from the _kernel-vanilla-stable_ repository make it easy to check if a particular bug still happens with the latest stable kernel.
* The kernels from the @kernel-vanilla/fedora repository allow you to check if a bug happening with Fedora’s kernel is present upstream or caused by one of the patches Fedora applied.


In general, these kernels make it easier to directly interact with upstream developers. That way users don't have to go through the sometimes overworked and quite busy developers that maintain the Linux kernel packages in Fedora.
In general, these kernels make it simple to directly interact with upstream developers. That way users don't have to bother Fedora’s kernel maintainers, who often are buried in work and lack the manpower to look into issues that only happen in certain, not that common environments (for example only on a single Laptop).


== Are the kernels from the kernel-vanilla repositories as good as those Fedora provides? ==
== Are the kernels from the kernel vanilla repositories as good as those Fedora provides? ==


No. There are several reasons for why not; the most important ones:
For many people they just as fine, but in the end they are not as good:


* the kernels shipped in the official Fedora repositories get a lot of testing. The kernels from the kernel vanilla repositories nearly no testing before uploaded to the repositories.
* The kernels shipped in the official Fedora repositories are tested by many people before they reach regular users; the kernels from the kernel vanilla repositories see no testing before being uploaded to the repositories.
* the official Fedora kernels sometimes contain changes that fix security problems or other crucial bugs before a fix gets merged in a official Linux kernel release.
* The official Fedora kernels don’t require you to turn off UEFI Secure Boot.
* the developers that take care of the kernel package in Fedora are far more experienced developers than those that take care of the kernel-vanilla repositories.
* The developers that take care of the kernel package in Fedora are far more experienced developers than the maintainers of the kernel vanilla repositories.
On the other hand using the kernels from the kernel-vanilla repositories should not be any more dangerous than downloading a Linux version from kernel.org and compiling and installing it source yourself.
* The official Fedora kernels sometimes contain fixes for security vulnerabilities or other crucial bugs before the problem is fixed upstream; on the other hand, you often will receive fixes a bit quicker by using these repositories if they are applied upstream first.
 
In the end though, using kernels from the kernel vanilla repositories is not any more dangerous than installing a Linux version using the sources from kernel.org yourself.


== Why so many repos? This looks stupid and over-engineered! ==
== Why so many repos? This looks stupid and over-engineered! ==


Yes, it might look over-engineered on the first sight, but allows us to serve different users bases. And once you look closer you'll notice that most of the time those four repositories contain only two Linux kernel versions per Fedora release.
Yes, it might look over-engineered at the first sight, but allows us to serve multiple use-cases with nearly no overhead. And once you look closer you'll notice that most of the time those four repositories contain only two Linux kernel versions per Fedora release.


[[File:Kernel.org-20180419.png|thumb|Kernel.org frontpage on 2018-04-09]] To explain this a bit more lets for example take a look April 9th, 2018. Back then…
[[File:Kernel.org-20180419.png|thumb|Kernel.org frontpage on 2018-04-09]] To explain this a bit more lets for example take a look April 9th, 2018. Back then…


* Linux 4.17 had one week into development, thus Linux 4.17-rc1 was still one week away.
* Linux 4.16 was one week old and the first stable release 4.16.1 had just been released.
* Linux 4.16 was one week old and the first stable release 4.16.1 had just been released.
* Linux 4.17 had one week into development, thus Linux 4.17-rc1 was still one week away.
* Linux 4.15.''y'' was still supported upstream and 4.15.16 had just been released.
* Linux 4.15.x was still supported upstream and 4.15.16 had just been released.


At the same time…
At the same time…


* Fedora 29 was prepared in Rawhide, which contained a mainline snapshot (4.17-pre-rc1).
* Fedora 29 was prepared in Rawhide, which contained a mainline snapshot (4.17-pre-rc1).
* Fedora 28 was in Beta and contained 4.16._x_.
* Fedora 28 was in Beta and contained 4.16.''y''.
* Fedora 27 and 26 were the current releases and contained a Kernel based on Linux 4.15._x_.
* Fedora 27 and 26 were the current releases and contained a kernel based on Linux 4.15.''y''.


In this particular point in time there were…
In this particular point in time there were…


* users that want the latest mainline snapshots (4.17-pre-rc1)
* users that wanted the latest mainline snapshots (4.17-pre-rc1)
* users that normally want the latest mainline snapshots, but want to avoid them during the busy merge window when most of the changes happen, as the risk of bad bugs is a bit bigger in those two weeks. Those users thus wanted the latest stable release (4.16._x_)
* users that normally wanted the latest mainline snapshots, except during the busy merge window when most of the changes happen, as the risk of bad bugs is a bit bigger in those two weeks; those users thus wanted the latest stable release (4.16.1)
* users that just wanted to use the latest Linux stable release (4.16._x_)
* users that just wanted to use the latest Linux stable release (4.16.1)
* users that want to check if a problem they face with a Fedora kernel might be due to a patch that Fedora applied to their kernels (4.15._x_ for Fedora 27 and 26)
* users that wanted to check if a problem they face with a Fedora kernel might be due to a patch that Fedora applied to their kernels (4.15.16 for Fedora 27 and 26)


[[File:Repos-20180419.png|thumb|Kernel vanilla repositories on 20180409]] And that's why there are four repositories, to serve each of those users what they want:
[[File:Repos-20180419.png|thumb|Kernel vanilla repositories on 20180409]] And that's why there are four repositories, to serve each of those users what they wanted:


* the _kernel-vanilla-mainline_ repo shipped a mainline snapshot (4.17-pre-rc1).
* @kernel-vanilla/mainline shipped a mainline snapshot (4.17-pre-rc1).
* the _kernel-vanilla-mainline-wo-mergew_ repo shipped the latest stable release (4.16.1, but will jump to 4.17-rc1 once it's released).
* @kernel-vanilla/mainline-wo-mergew shipped the latest stable release (4.16.1, but will jump to 4.17-rc1 once it's released).
* the _kernel-vanilla-stable_ repo also shipped the latest stable release (4.16.1) (side note: the RPM packages are hardlinked to save space).
* @kernel-vanilla/stable also shipped the latest stable release (4.16.1)  
* the _kernel-vanilla-fedora_ repo shipped a vanilla build of the kernel version that release is using; hence F29/rawhide had a mainline snapshot  (4.17-pre-rc1), F28 had the lastest stable release (4.16.1) and F27 and F26 had the stable release from the previous version line (4.15.16).
* @kernel-vanilla/fedora shipped a vanilla build of the kernel version that release is using; hence F29/rawhide had a mainline snapshot  (4.17-pre-rc1), F28 had the lastest stable release (4.16.1) and F27 and F26 had the stable release from the previous version line (4.15.16).


Two or three weeks later F26 and F27 had made the jump to 4.16 and the merge window was over. Then the mainline and mainline-wo-mergew repo both shipped a 4.17 prerelease; the stable and the fedora repos both ship a 4.16.x kernel.
Two or three weeks later F26 and F27 had made the jump to 4.16 and the merge window was over. Then @kernel-vanilla/mainline and @kernel-vanilla/mainline-wo-mergew both provided a 4.17 prerelease; @kernel-vanilla/stable, @kernel-vanilla/stable-rc, and @kernel-vanilla/fedora all provided the same 4.16.x kernel.


== Can we trust the people behind it? ==
== Can we trust the people behind the kernel vanilla repositories? ==


You have to decide yourself. If it is any help: Some of those that use or contribute to Fedora since a while will know that Thorsten has quite a history of Fedora contributions, even if he is very active in the past few years. You can assume he has no interest in ruin his reputation quickly by providing crappy packages in these repositories. On the other hand you should know that Thorsten started these repositories to dig deeper into the kernel and kernel development; so expect he'll make some mistakes along the way. And be reminded that using vanilla kernels has some known downsides and risks (see below).
You have to decide for yourself.


== Thorsten, do you use the vanilla kernels yourself? ==
If it is any help: some people that have used or contributed to Fedora regularly will know that Thorsten (the main maintainer for these repositories) has quite a history of Fedora contributions, even if he is not very active in Fedora these days. You can assume he has no interest in ruining his reputation quickly by providing crappy packages in these repositories. On the other hand you should know that Thorsten is not a real kernel developer, so expect an occasional mistake along the way. And be reminded that using vanilla kernels has some known downsides and risks (see below).


Yes, I normally use the x86-64 vanilla kernels from the mainline-wo-mergew repository, either on the current Fedora release or the beta of the next one. But I don't reboot every day, so I might not boot each of the kernels in that repo.
== Are the vanilla kernels tested before publication? ==
 
No, as they are built with Fedora’s Copr infrastructure, which doesn’t offer a simple way to test newly built packages before their publication. But all kernels shortly afterwards are booted in a virtual machine to rule out any packaging issues (but those are extremely rare anyway). Thorsten also regularly uses the x86_64 builds from the @kernel-vanilla/mainline-wo-mergew repository, either on the latest Fedora release or a pre-release of the next one; but he doesn't reboot every day and hence won’t give each of the builds a try run.


== Are the kernels safe to use? ==
== Are the kernels safe to use? ==
Line 66: Line 70:
That depends on your definition of 'safe'.
That depends on your definition of 'safe'.


The Linux kernel is a complex piece of software and thus contains bugs. Those bugs lead to data loss a few times in the past; in very rare situations they even damaged hardware. Bugs like that often only show up under specific circumstances, as they otherwise likely would have been found and fixed already. Specific circumstances for example can be a specific mix of hardware used in combinations with a specific kernel version built with a particular set of configuration options. Nevertheless, in the end it unlikely that such a bug makes it into one of the non-development kernels from the kernel vanilla repositories, but there is still a very small chance fot that to happen.
The Linux kernel is a complex piece of software and thus contains bugs. Those bugs in the past led to data loss a few times; in extremely rare situations they even damaged hardware. Bugs like that often only show up under specific circumstances, as they otherwise likely would have been found and fixed earlier already. Specific circumstances for example can be a specific mix of hardware used in combinations with a specific kernel version built with a particular set of configuration options. Nevertheless, in the end it is unlikely that such a bug makes it into one of the non-development kernels from the kernel vanilla repositories, but there is still a very small chance for that to happen.


Note that self compiled kernels bear exactly the same risk; chances of hitting serious bugs are lower for kernels that have undergone widespread testing already, as those found in the official Fedora repositories.
Note that self compiled kernels bear exactly the same risk; chances of hitting serious bugs are lower for kernels that have undergone widespread testing already, as those found in the official Fedora repositories.


In other words: The kernels from the kernel-vanilla repositories will work just fine for most people. But use them on your own risk and have current backups at hand, as you always should.
In other words: The kernels from the kernel vanilla repositories will work just fine for most people. But use them at your own risk and have current backups at hand, as you always should.


== Will everything work with the vanilla kernels that works with the official Fedora kernels? ==
== Will everything continue to work with these kernels as it does with the official Fedora kernels? ==


No. Linux vanilla kernels are not that different from the kernels Fedora provides, but the latter include a few enhancements. Each was added for a good reason to make Fedora better, hence these improvements are missing when you use Linux vanilla kernels.
Most of the time yes, but in a few not that common situations it might not. That’s for example the case if Fedora’s kernels include a fix for an issue not yet fixed upstream. This is not that common, but within the cards; you on the other hand sometimes get bugfixed earlier with the kernels from these repositories.  
 
When this text was written in the spring of 2012, Fedora for example included utrace in their Linux kernels to support userspace tracing with Systemtap; hence this feature didn't work with the kernels from the kernel-vanilla repositories (but it should work these days, as Systemtap now uses a solution from the upstream kernel for its work).
 
Another example: The kernels from Fedora sometimes include fresher drivers which some systems will require to work properly.
 
Furthermore, in rare cases there are inter-dependencies between drivers in kernel and userland. The nouveau driver for graphics hardware from Nvidia was one such driver, as it had no stable API yet when this text was written; that's why the DRM/KMS driver in the kernel was marked as 'staging' back then. The Mesa 3D or X.org drivers included in a particular Fedora release therefore might depend exactly on the nouveau DRM/KMS driver shipped in the official Fedora kernel. In such cases the nouveau drivers for Mesa 3D and X.org of a certain Fedora release might not work properly with kernels found in the kernel-vanilla repositories, as their nouveau DRM/KMS driver might be newer or older and thus incompatible.
 
The non-development kernels found in the kernel-vanilla repositories therefore should work on a lot of systems, but on a few systems they might work slightly worse than the kernels Fedora provides.


== Where to report bugs ==
== Where to report bugs ==
Line 92: Line 88:
== How can I avoid switching back and forth between vanilla kernels and Fedora kernels ? ==
== How can I avoid switching back and forth between vanilla kernels and Fedora kernels ? ==


Add 'exclude=kernel*' to the first section of these files in /etc/yum.repos.d/
This normally shouldn’t happen, as the kernel vanilla repositories typically contain packages package managers will consider as newer than the Fedora kernel. And if they nevertheless lag behind that’s likely because the maintainers of the kernel vanilla repositories are inactive for some good or bad reason – hence it might be wise to temporarily switch back to the Fedora kernel. If you nevertheless want to avoid that, add 'exclude=kernel*' to the first section of the following files in /etc/yum.repos.d/: fedora.repo, fedora-updates.repo, fedora-updates-testing.repo


fedora.repo
== Will this repository also ship updated userland components like drivers or udev that match the kernels in the repositories? ==
fedora-updates.repo
fedora-updates-testing.repo
 
== Will this repository also ship updates userland components like drivers or udev that match the kernels in the repositories? ==


No, as there should be no need to, as the interfaces between the kernel and userland software should never change in incompatible ways; Linus Torvalds makes this pretty clear every so often.
No, as there should be no need to, as the interfaces between the kernel and userland software should never change in incompatible ways; Linus Torvalds makes this pretty clear every so often.


That is the long story short. There are situations where the world is more complicated:
Nevertheless in very rare situations there might be a strong reason to include such a package to avoid breakage for many users; in that case these repositories might include a package to prevent those issues.
 
* above mentioned rule does not apply to staging drivers, so situations might arise where the vanilla kernels are not usable for people that need staging drivers for their system.
* Fedora sometimes might contain software that depends on bits that are not upstream.
 
And even with this rule sometimes a new mainline kernel versions brings changes that require updates userland software. Three examples:
 
* the version number jump from 2.6.39 to 3.0 confused some software.
* in rare cases fixing security problems was only possible my changing the interfaces in incompatible ways.
* sometimes nobody notices early enough that interfaces have changed and reverting the change might break systems that already rely on the new behavior.
 
How we deal with them will be decided on a case by case basis. In some cases we might have to other solution then to add new versions of other software to the repositories. But the plan is to avoid this if possible.
 
== Do you plan to provide packages for Longterm kernels ==
 
Unlikely. The main goal of the kernel vanilla repositories for Fedora is to help upstream kernel development; but Longterm kernels are a dead end and quite a bit away from mainline development, so they would not fit that well.


Using a Linux kernel version that is from an older version line than the one used by Fedora also increases the risk that something works worse, so it's possible to argue this idea most of the time doesn't make sense.
== Do you plan to provide packages for longterm (aka LTS) kernels ==


Additionally, the RPM packages shipped in these repositories normally have a higher version number then the kernel packages that Fedora ships with, thus Dnf automatically will prefer the kernel packages from this repo. Shipping older versions would only work for users that change the repositories definitions Fedora provides (fedora.repo; fedora-updates.repo, … – see above) to ignore all the kernels that Fedora ships in its repositories. That is not hard, but makes things more complicated.
That’s very unlikely, as using a Linux kernel version from a series older than the one used by the particular Fedora release can lead to issues. Providing such kernels hence would not provide a good user-experience, unless Fedora itself starts to ship longterm kernels.  


== Why are there no stable and mainline-wo-mergew repos for rawhide? ==
Additionally, the main goal of the kernel vanilla repositories for Fedora is to help upstream kernel development – longterm kernels don’t line up well with that, as they are quite a bit away from mainline development and a dead end.


Those versions would be older than the ones used in Fedora rawhide. That is something we try to avoid for now (see above answer to "Do you plan to provide packages for longterm kernels" for details).
== Are debug features enabled in the vanilla builds? ==


== Do you plan to provide packages for "linux-next" or "linux-rt" as well? ==
Only those that Fedora enables in their stock kernels – hence, there are none enabled that are known to have a significant performance overhead.


For now: No. I know there is some interest in packages for them, but maintaining those will consume a lot of time regularly, and we have not enough resources to do it properly right now.
== Do you plan to provide packages for "linux-rt" as well? ==


[[user:thl|Get in contact]] if you think investing time in these areas makes sense.
Maybe. It depends on the interest and how hard setup and maintenance would be. [[user:thl|Get in contact]] if you think investing time in these areas makes sense – or might want to help with the work.


== Do you plan to provide vanilla kernels for RHEL and its derivatives? ==
== Do you plan to provide vanilla kernels for RHEL and its derivatives? ==


Sounds like a good addition. But there are people more familiar with these distributions [http://elrepo.org/tiki/kernel-ml|that provide such packages already]. It would mean additional work for us, too; and we currently have no one that would regularly run such kernels. So for now we won't get our feet wet in that area. But if you want to step up and help, [[User:thl|get in contact]].
For now it was decided to stay out of that business, even if it sounds like a good addition. There are several reasons for that. There for example are people more familiar with these distributions [http://elrepo.org/tiki/kernel-ml|who provide such packages already]. It would mean additional work for the maintainers of this repository, too.  


== What configuration do those kernels use? ==
== What configuration do those kernels use? ==


The mainline kernels use basically the same configuration as the Fedora rawhide kernels. Maybe a few staging drivers might get turned on to help their development, but apart from it the plan is to stick closely to what Fedora does.
The configuration is pretty close to Fedora’s kernels. Which config exactly depends on the repo and branch; the kernels in @kernel-vanilla/fedora use a config that is basically identical to the kernel in the latest stable Fedora release, while those in @kernel-vanilla/mainline will be based on the config from Fedora’s rawhide kernel.
 
== Why don't you put these kernels in Fedoras main repositories ==


The consensus in the Fedora project as far as we know is: That's not a good idea, as it divides the user base. It also would make the vanilla Linux kernels more 'official' and people might simply use those kernels without knowing what their downsides are.
== Why don't you put these kernels in Fedora’s main repositories? ==


That's the long story rough and short. And sure, there are reasons why having vanilla kernels in the main repositories would make sense. Feel free to start a discussion on [http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org the Fedora devel mailing list], we'll watch and might jump in.
The idea is not new, but the consensus in the Fedora project as far as we can see is this: that's not a good idea, as it divides the user base; it also would make the vanilla Linux kernels more 'official' and people might simply use them without knowing their downsides. Putting the kernels in a well known and widely used external add-on repository for Fedora might make sense, but some problems would be similar.


Putting the kernels in a well know 3rd party add-on repository for Fedora might make sense, but some problems would be similar. It would also lead to more problems, as then users might ask said 3rd party repos to build add-on modules packages for those kernels, too.
That's the long story, rough and short. And sure, there are reasons why having vanilla kernels in the main repositories would make sense. Feel free to start a discussion on the [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/ Fedora devel] and the [https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org/ Fedora kernel] mailing list, we'll watch and might jump in.


The best approach would be to reduce the number of patches the Fedora kernel developers include for one reason or another down to zero or something very close to that. That would require changes not only in Fedora, but in the upstream workflow as well.
But in the end the best approach would be to reduce the number of patches the Fedora kernel developers include to zero or something very close to that, as then some of the repos this effort provides wouldn’t be necessary at all.


== Are those kernels really unpatched? ==
== Are those kernels really unpatched? ==


Normally yes.
Yes, apart from a handful of very small changes that are needed for packaging.  


From time to time the packages  but  need to contain a handful of very small changes that are needed for packaging. Sometimes we also might be forced to temporary use patches to make the kernel build or usable for typical systems; this happens very rarely and fixes like these will normally head upstream quickly and hence vanish from the vanilla packages pretty soon again. Furthermore, this normally should only happen with mainline development kernels, not with stable kernels.
That being said: in very rare situations we might include a patch to fix build problems. That normally only happens for mainline builds; those fixes often head upstream quickly and hence vanish from the vanilla packages pretty soon again.  


== How up2date will those repositories be? ==
== How up2date will those repositories be? ==


Most of the time they are quite up2date and only a day or two behind at max. But we do the work in our spare time. Sometimes the day job and this strange thing called 'real life' leave not much time to work on these kernels, which will lead to a bigger lag.
Most of the time they are quite up to date. But the maintainers of the kernel vanilla repositories do the work in their spare time. Sometimes the day job and this strange thing called 'real life' leaves not much time to work on these repositories, which will lead to a bigger lag.


= FAQ for contributors and developers =
= FAQ for contributors and developers =
Line 165: Line 140:
== Can you please include the patch found at <URL>? ==
== Can you please include the patch found at <URL>? ==


No. Get your patch merged upstream, then the change you are interested in will automatically show up in these packages. And even better: it will automatically get into Fedora and other distributions, too!
No. Convince upstream to merge that patch, then the change you are interested in will automatically show up in these packages. And even better: it will automatically get into Fedora and other distributions, too!


== Is there a Git tree with the stuff used to build the SRPM somewhere? ==
== Where is the code from which the packages are built? ==


Sure: [http://fedorapeople.org/cgit/thl/public_git/kernel.git/ http://fedorapeople.org/cgit/thl/public_git/kernel.git/]. Kernels in the kernel-vanilla-mainline repository normally get build from the rawhide branch. Kernels in the kernel-vanilla-fedora repository are always built from the appropriate Fedora branch. Most of the time the kernels in the kernel-vanilla-stable repository come from here to, but sometimes they are build from the stabilization or transition branch.
The packages in the kernel vanilla repositories for Fedora Linux are built using the [https://gitlab.com/cki-project/kernel-ark kernel-ark infrastructure], just like the regular kernel SRPMs in Fedora Linux are.


Let us know if we should do modifications to allow others to contribute to or benefit from this git tree better.
The code can be found in various branches [https://gitlab.com/knurd42/linux/ at gitlab.com/knurd42/linux/]:


Note: when using `fedpkg` with this dist.git repo you need to specify the Fedora release you are aiming for, for example like this: `fedpkg --release f33`. Additionally, the command sadly won't be able to download the files specified in the sources file if you clone the repo directly from the above URL. That's because fedpkg derives the download configuration for those files from the URL for the origin branch (specified in `.git/config`) and the reason why commands like `fedpkg sources`, `fedpkg srpm`, `fedpkg prep` or `fedpkg local` will fail.  
* branches like `ark-infra-os-build` and `ark-infra-fedora-6.2` are forks of the [https://gitlab.com/cki-project/kernel-ark kernel-ark] branches like os-build and fedora-6.2, but with modifications for vanilla builds. These branches are normally not rebased, unless kernel-ark does rebase their branches.


To avoid this, you better add the repo as a remote to Fedora's official dist.git repo for the kernel, which the vanilla git repo is based on anyway:
* branches like `ark-patches-mainline` or `ark-patches-stable-6.2` are tracking the Linux upstream trees; they when needed hold patches to avoid known build-problems that otherwise would prevent vanilla builds. Such situations are rare, these branches thus most of the time are identical to upstream. These branches are rebased before each build.  


<pre>
* the vanilla packages are build from branches like `ark-vanilla-mainline` or `ark-vanilla-stable-6.2`. They are rebased for every build, as they get generated on-the-fly by taking a patches branch (say `ark-patches-mainline`) as new base and importing only the ark-infra (e.g. the redhat/ directory and a few other bits) from one of the ark-infra branches (`ark-infra-os-build` for next and mainline; branches like `ark-infra-fedora-6.2` for stable-rc, stable, and fedora).
git clone https://src.fedoraproject.org/rpms/kernel.git
git remote add vanilla git://fedorapeople.org/home/fedora/thl/public_git/kernel.git
git fetch vanilla
git checkout vanilla/rawhide-user-thl-vanilla-fedora
</pre>


If you are struggling with this, feel free to get the vanilla dist.git repo like this (it uses a tricky workaround to make things work):
With this set up git can easily merge the latest ark-infra changes from the kernel-ark repo into branches that hold changes for vanilla builds – at the same time any patches kernel-ark applied are avoided.


<pre>
All these branches are maintained with the help of [https://gitlab.com/knurd42/helpers-ark-vanilla some scripts]. They watch out for changes to the kernel-ark infra, new Linux releases, and kick of new builds automatically or on request.
git clone https://thl.fedorapeople.org/rpms/kernel.git
</pre>
 
== Why are there no debug kernels and not even debuginfo packages ==
 
The space on repos.fedoraprople.org is limited, hence we need to limit the number of packages we can provide. The debuginfo packages are also quite big and thus downloading the koji results (for testing) and uploading the final repo would take a lot longer.
 
If you need the debuginfo packags consider rebuilding the SRPM with "rpmbuild -bb --with debuginfo" and installing the results. You can also rebuild the SRPM using "rpmbuild -bb --with dbgonly" in case you need a kernel image where all sorts of debug options are enabled in the configuration.
 
Let us know if there is interest in these packages, then maybe a solution can be found to provide these packages sooner or later.
 
== Why don't you commit your changes to Fedora's kernel git repo on pkgs.fedoraproject.org? ==
 
That might make sense. But it bears the risk that a commit is done to a wrong branch and disturbs the work of the Fedora kernels maintainer. Further: Not all of those that contribute to Fedora can commit there. That's similar with the fedorapeople git repository, but the docs indicate others can be given access with the help of ACLs.
 
But whatever: Git is made for distributed development, so simply clone it and send pull requests if you have any additions.


== Can I help? ==
== Can I help? ==
Line 210: Line 164:
== Do you work together with the developers that maintain Fedora's kernel packages? ==
== Do you work together with the developers that maintain Fedora's kernel packages? ==


There is cooperation already. If you think more is needed in some areas let us know.
We know about each other and talk occasionally..


== Please stop providing alternative kernel packages, they take attention away from the kernel packages Fedora provide and thus harm Fedora! ==
== Please stop providing alternative kernel packages, they take attention away from the kernel packages Fedora provides and thus harm Fedora! ==


That's a valid concern, but we think the benefits outweigh the downsides.
That's a valid concern, but we think the benefits outweigh the downsides.


That again that is the long story short. Just to get a little deeper into it and show a different view on the matter at hand: Similar arguments could be used to argue that Fedora should stop shipping patched kernels, as they take attention away from the upstream kernel. Up to a point such an argument is valid, too, but there are good reasons why Fedora patches its kernels.
That again is the long story short. Just to get a little deeper into it and show a different view on the matter at hand: similar arguments could be used to argue that Fedora should stop shipping patched kernels, as they take attention away from the upstream kernel. Up to a point such an argument is valid, too, but there are good reasons why Fedora patches its kernels.
 
== Why did you drop the '-vanilla' postfix that normally gets added to the 'name' macro when you build Fedora's kernel RPM without patches locally? ==
 
I've thought about dropping or leaving it for a while, as both schemes have various benefits and downsides. In the end I went for dropping it due to reasons like this:
 
* nearly every other repository in Fedoraland that ships variants of packages that are included in Fedora do not change the name.
* the postfix in the name breaks some tools – for example things like 'fedpkg srpm' on the git checkout.
* external solutions that heavily depend on the naming scheme Fedora uses (like the akmod/kmod stuff used in some external repositories) would break with the -vanilla postfix in the name.
* DNF might not recognize kernel packages with a '-vanilla' postfix as 'installonly' and thus would perform a regular update for vanilla packages instead of installing them parallel to the current one.

Revision as of 06:23, 23 March 2023

Frequently asked questions about the kernel vanilla repositories for Fedora.

FAQ for users

What is the goal of these repositories?

The main idea is to help upstream development, which in the end will benefit Fedora as well:

  • With the packages from the @kernel-vanilla/mainline repository it for example is quite easy to test code early that at some point will land in Fedora Linux. In other words: the repository thus allows people to find, report, and eliminate bugs so that they never even hit Fedora. The packages also make it easy to check if a particular bug you encountered is already fixed upstream, as all bug-fixes have to land in mainline before they are considered for backporting to stable and longterm kernels.
  • The situation with @kernel-vanilla/stable repository is similar to the @kernel-vanilla/mainline, you are just not that much ahead of Fedora’s kernels, as they often follow stable quite closely.
  • The kernels from the @kernel-vanilla/fedora repository allow you to check if a bug happening with Fedora’s kernel is present upstream or caused by one of the patches Fedora applied.

In general, these kernels make it simple to directly interact with upstream developers. That way users don't have to bother Fedora’s kernel maintainers, who often are buried in work and lack the manpower to look into issues that only happen in certain, not that common environments (for example only on a single Laptop).

Are the kernels from the kernel vanilla repositories as good as those Fedora provides?

For many people they just as fine, but in the end they are not as good:

  • The kernels shipped in the official Fedora repositories are tested by many people before they reach regular users; the kernels from the kernel vanilla repositories see no testing before being uploaded to the repositories.
  • The official Fedora kernels don’t require you to turn off UEFI Secure Boot.
  • The developers that take care of the kernel package in Fedora are far more experienced developers than the maintainers of the kernel vanilla repositories.
  • The official Fedora kernels sometimes contain fixes for security vulnerabilities or other crucial bugs before the problem is fixed upstream; on the other hand, you often will receive fixes a bit quicker by using these repositories if they are applied upstream first.

In the end though, using kernels from the kernel vanilla repositories is not any more dangerous than installing a Linux version using the sources from kernel.org yourself.

Why so many repos? This looks stupid and over-engineered!

Yes, it might look over-engineered at the first sight, but allows us to serve multiple use-cases with nearly no overhead. And once you look closer you'll notice that most of the time those four repositories contain only two Linux kernel versions per Fedora release.

Kernel.org frontpage on 2018-04-09

To explain this a bit more lets for example take a look April 9th, 2018. Back then…

  • Linux 4.17 had one week into development, thus Linux 4.17-rc1 was still one week away.
  • Linux 4.16 was one week old and the first stable release 4.16.1 had just been released.
  • Linux 4.15.y was still supported upstream and 4.15.16 had just been released.

At the same time…

  • Fedora 29 was prepared in Rawhide, which contained a mainline snapshot (4.17-pre-rc1).
  • Fedora 28 was in Beta and contained 4.16.y.
  • Fedora 27 and 26 were the current releases and contained a kernel based on Linux 4.15.y.

In this particular point in time there were…

  • users that wanted the latest mainline snapshots (4.17-pre-rc1)
  • users that normally wanted the latest mainline snapshots, except during the busy merge window when most of the changes happen, as the risk of bad bugs is a bit bigger in those two weeks; those users thus wanted the latest stable release (4.16.1)
  • users that just wanted to use the latest Linux stable release (4.16.1)
  • users that wanted to check if a problem they face with a Fedora kernel might be due to a patch that Fedora applied to their kernels (4.15.16 for Fedora 27 and 26)
Kernel vanilla repositories on 20180409

And that's why there are four repositories, to serve each of those users what they wanted:

  • @kernel-vanilla/mainline shipped a mainline snapshot (4.17-pre-rc1).
  • @kernel-vanilla/mainline-wo-mergew shipped the latest stable release (4.16.1, but will jump to 4.17-rc1 once it's released).
  • @kernel-vanilla/stable also shipped the latest stable release (4.16.1)
  • @kernel-vanilla/fedora shipped a vanilla build of the kernel version that release is using; hence F29/rawhide had a mainline snapshot (4.17-pre-rc1), F28 had the lastest stable release (4.16.1) and F27 and F26 had the stable release from the previous version line (4.15.16).

Two or three weeks later F26 and F27 had made the jump to 4.16 and the merge window was over. Then @kernel-vanilla/mainline and @kernel-vanilla/mainline-wo-mergew both provided a 4.17 prerelease; @kernel-vanilla/stable, @kernel-vanilla/stable-rc, and @kernel-vanilla/fedora all provided the same 4.16.x kernel.

Can we trust the people behind the kernel vanilla repositories?

You have to decide for yourself.

If it is any help: some people that have used or contributed to Fedora regularly will know that Thorsten (the main maintainer for these repositories) has quite a history of Fedora contributions, even if he is not very active in Fedora these days. You can assume he has no interest in ruining his reputation quickly by providing crappy packages in these repositories. On the other hand you should know that Thorsten is not a real kernel developer, so expect an occasional mistake along the way. And be reminded that using vanilla kernels has some known downsides and risks (see below).

Are the vanilla kernels tested before publication?

No, as they are built with Fedora’s Copr infrastructure, which doesn’t offer a simple way to test newly built packages before their publication. But all kernels shortly afterwards are booted in a virtual machine to rule out any packaging issues (but those are extremely rare anyway). Thorsten also regularly uses the x86_64 builds from the @kernel-vanilla/mainline-wo-mergew repository, either on the latest Fedora release or a pre-release of the next one; but he doesn't reboot every day and hence won’t give each of the builds a try run.

Are the kernels safe to use?

That depends on your definition of 'safe'.

The Linux kernel is a complex piece of software and thus contains bugs. Those bugs in the past led to data loss a few times; in extremely rare situations they even damaged hardware. Bugs like that often only show up under specific circumstances, as they otherwise likely would have been found and fixed earlier already. Specific circumstances for example can be a specific mix of hardware used in combinations with a specific kernel version built with a particular set of configuration options. Nevertheless, in the end it is unlikely that such a bug makes it into one of the non-development kernels from the kernel vanilla repositories, but there is still a very small chance for that to happen.

Note that self compiled kernels bear exactly the same risk; chances of hitting serious bugs are lower for kernels that have undergone widespread testing already, as those found in the official Fedora repositories.

In other words: The kernels from the kernel vanilla repositories will work just fine for most people. But use them at your own risk and have current backups at hand, as you always should.

Will everything continue to work with these kernels as it does with the official Fedora kernels?

Most of the time yes, but in a few not that common situations it might not. That’s for example the case if Fedora’s kernels include a fix for an issue not yet fixed upstream. This is not that common, but within the cards; you on the other hand sometimes get bugfixed earlier with the kernels from these repositories.

Where to report bugs

If the Linux kernels in the packages from these repositories show any bugs please report them upstream to the Linux kernel developers, just as you would after installing a Linux kernel yourself using the sources available at kernel.org; that way all the bug reports go to the place where the people hang out that know how to fix them.

In case there are bugs in the packaging sent a mail to Thorsten Leemhuis (aka "knurd").

How can I avoid switching back and forth between vanilla kernels and Fedora kernels ?

This normally shouldn’t happen, as the kernel vanilla repositories typically contain packages package managers will consider as newer than the Fedora kernel. And if they nevertheless lag behind that’s likely because the maintainers of the kernel vanilla repositories are inactive for some good or bad reason – hence it might be wise to temporarily switch back to the Fedora kernel. If you nevertheless want to avoid that, add 'exclude=kernel*' to the first section of the following files in /etc/yum.repos.d/: fedora.repo, fedora-updates.repo, fedora-updates-testing.repo

Will this repository also ship updated userland components like drivers or udev that match the kernels in the repositories?

No, as there should be no need to, as the interfaces between the kernel and userland software should never change in incompatible ways; Linus Torvalds makes this pretty clear every so often.

Nevertheless in very rare situations there might be a strong reason to include such a package to avoid breakage for many users; in that case these repositories might include a package to prevent those issues.

Do you plan to provide packages for longterm (aka LTS) kernels

That’s very unlikely, as using a Linux kernel version from a series older than the one used by the particular Fedora release can lead to issues. Providing such kernels hence would not provide a good user-experience, unless Fedora itself starts to ship longterm kernels.

Additionally, the main goal of the kernel vanilla repositories for Fedora is to help upstream kernel development – longterm kernels don’t line up well with that, as they are quite a bit away from mainline development and a dead end.

Are debug features enabled in the vanilla builds?

Only those that Fedora enables in their stock kernels – hence, there are none enabled that are known to have a significant performance overhead.

Do you plan to provide packages for "linux-rt" as well?

Maybe. It depends on the interest and how hard setup and maintenance would be. Get in contact if you think investing time in these areas makes sense – or might want to help with the work.

Do you plan to provide vanilla kernels for RHEL and its derivatives?

For now it was decided to stay out of that business, even if it sounds like a good addition. There are several reasons for that. There for example are people more familiar with these distributions provide such packages already. It would mean additional work for the maintainers of this repository, too.

What configuration do those kernels use?

The configuration is pretty close to Fedora’s kernels. Which config exactly depends on the repo and branch; the kernels in @kernel-vanilla/fedora use a config that is basically identical to the kernel in the latest stable Fedora release, while those in @kernel-vanilla/mainline will be based on the config from Fedora’s rawhide kernel.

Why don't you put these kernels in Fedora’s main repositories?

The idea is not new, but the consensus in the Fedora project as far as we can see is this: that's not a good idea, as it divides the user base; it also would make the vanilla Linux kernels more 'official' and people might simply use them without knowing their downsides. Putting the kernels in a well known and widely used external add-on repository for Fedora might make sense, but some problems would be similar.

That's the long story, rough and short. And sure, there are reasons why having vanilla kernels in the main repositories would make sense. Feel free to start a discussion on the Fedora devel and the Fedora kernel mailing list, we'll watch and might jump in.

But in the end the best approach would be to reduce the number of patches the Fedora kernel developers include to zero or something very close to that, as then some of the repos this effort provides wouldn’t be necessary at all.

Are those kernels really unpatched?

Yes, apart from a handful of very small changes that are needed for packaging.

That being said: in very rare situations we might include a patch to fix build problems. That normally only happens for mainline builds; those fixes often head upstream quickly and hence vanish from the vanilla packages pretty soon again.

How up2date will those repositories be?

Most of the time they are quite up to date. But the maintainers of the kernel vanilla repositories do the work in their spare time. Sometimes the day job and this strange thing called 'real life' leaves not much time to work on these repositories, which will lead to a bigger lag.

FAQ for contributors and developers

Can you please include the patch found at <URL>?

No. Convince upstream to merge that patch, then the change you are interested in will automatically show up in these packages. And even better: it will automatically get into Fedora and other distributions, too!

Where is the code from which the packages are built?

The packages in the kernel vanilla repositories for Fedora Linux are built using the kernel-ark infrastructure, just like the regular kernel SRPMs in Fedora Linux are.

The code can be found in various branches at gitlab.com/knurd42/linux/:

  • branches like ark-infra-os-build and ark-infra-fedora-6.2 are forks of the kernel-ark branches like os-build and fedora-6.2, but with modifications for vanilla builds. These branches are normally not rebased, unless kernel-ark does rebase their branches.
  • branches like ark-patches-mainline or ark-patches-stable-6.2 are tracking the Linux upstream trees; they when needed hold patches to avoid known build-problems that otherwise would prevent vanilla builds. Such situations are rare, these branches thus most of the time are identical to upstream. These branches are rebased before each build.
  • the vanilla packages are build from branches like ark-vanilla-mainline or ark-vanilla-stable-6.2. They are rebased for every build, as they get generated on-the-fly by taking a patches branch (say ark-patches-mainline) as new base and importing only the ark-infra (e.g. the redhat/ directory and a few other bits) from one of the ark-infra branches (ark-infra-os-build for next and mainline; branches like ark-infra-fedora-6.2 for stable-rc, stable, and fedora).

With this set up git can easily merge the latest ark-infra changes from the kernel-ark repo into branches that hold changes for vanilla builds – at the same time any patches kernel-ark applied are avoided.

All these branches are maintained with the help of some scripts. They watch out for changes to the kernel-ark infra, new Linux releases, and kick of new builds automatically or on request.

Can I help?

Of course. Talk to Thorsten; best if you come with some ideas what you can and want to do.

Do you work together with the developers that maintain Fedora's kernel packages?

We know about each other and talk occasionally..

Please stop providing alternative kernel packages, they take attention away from the kernel packages Fedora provides and thus harm Fedora!

That's a valid concern, but we think the benefits outweigh the downsides.

That again is the long story short. Just to get a little deeper into it and show a different view on the matter at hand: similar arguments could be used to argue that Fedora should stop shipping patched kernels, as they take attention away from the upstream kernel. Up to a point such an argument is valid, too, but there are good reasons why Fedora patches its kernels.