From Fedora Project Wiki
< Changes
(draft) |
(Change rejected by FESCo) |
||
| (29 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name. This keeps all change proposals in the same namespace --> | <!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name. This keeps all change proposals in the same namespace --> | ||
<!-- The name of your change proposal --> | |||
= Debug Info Standardization (from DWZ to -fdebug-types-section) = | |||
== Summary == | == Summary == | ||
Fedora 18 implemented [[Features/DwarfCompressor]]. As the format | Fedora 18 implemented [[Features/DwarfCompressor]] called DWZ. As the format remains specific to Linux distributions it became a burden to make existing debugging tools compatible with DWZ. | ||
== Owner == | == Owner == | ||
| Line 14: | Line 15: | ||
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. --> | <!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. --> | ||
* Email: jan.kratochvil@redhat.com | * Email: jan.kratochvil@redhat.com | ||
This Change does not represent Red Hat's opinion, see [[#Feedback]]. | |||
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo) | <!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo) | ||
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address> | * FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address> | ||
| Line 42: | Line 44: | ||
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development | CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development | ||
--> | --> | ||
* FESCo issue: | * FESCo issue: [https://pagure.io/fesco/issue/2482 #2482] | ||
* Tracker bug: <will be assigned by the Wrangler> | * Tracker bug: <will be assigned by the Wrangler> | ||
* Release notes tracker: <will be assigned by the Wrangler> | * Release notes tracker: <will be assigned by the Wrangler> | ||
| Line 48: | Line 50: | ||
== Detailed Description == | == Detailed Description == | ||
Debug info files *.debug contained in *-debuginfo.rpm are very big in general (x86_64 Fedora 32 distribution has debug/ directory of 82GB while all its other files are only 75GB). There exist several methods how to make the *-debuginfo.rpms at least a bit smaller. Fedora 18 started using DWZ tool (from [[Features/DwarfCompressor]]) while [https://gcc.gnu.org/pipermail/gcc-patches/2008-August/246281.html Google implemented] the same goal in a different way called -fdebug-types-section. | |||
Linux distributions use existing DWZ (Fedora/CentOS/RHEL, SuSE OSes, OpenMandriva, Debian, Ubuntu maybe others?). DWZ support is missing in crossplatform tools like [https://lldb.llvm.org/ LLDB] or [https://llvm.org/docs/CommandGuide/llvm-dwarfdump.html llvm-dwarfdump]. -fdebug-types-section is used internally by Google (produced by clang). | |||
* DWZ advantage: On the whole Fedora distro it saves 3.3% (5GB of the 157GB distribution size) | |||
** If the 3.3% size increase is a concern I can implement a different optimization as a GCC post-processing phase which would require no changes in any DWARF consumers. DWZ would still makes the Fedora distribution smaller but only by 2.4%. | |||
** With [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/LIFG55TW5RBSBENQIYY773FGJGYHA3RQ/ another optimization] (reintroducing a common file but a very simple one just containing DW_UT_type units) DWZ still makes the Fedora distribution smaller but only by 1.8%. | |||
* DWZ disadvantage: DWZ has currently less support across consumers (LLDB, llvm-dwarfdump) | |||
* DWZ disadvantage: DWZ requires 8x times more complicated (LoC count) support in consumers than -fdebug-types-section | |||
* DWZ disadvantage: Changes/DebugInfoLldbIndex (LLDB acceleration) is disabled by DWZ; unless DWZ gets excluded from clang-built binaries | |||
* DWZ disadvantage: -fdebug-types-section unifies tools compatibility with format of binaries from rpms vs. binaries from edit-compile-debug cycle. DWZ is an additional time overhead for edit-compile-debug cycle while -fdebug-types-section even speeds up edit-compile-debug cycle (its linking phase). | |||
* DWZ disadvantage: DWZ DWARF-5 support is a work-in-progress. DWZ has been blocking DWARF-5 for Fedora for 3.5 years and only after I have now proposed to drop DWZ Mark Wielaard has started porting DWZ to DWARF-5. It can be expected next DWARF extensions will remain unsupported again. Even currently there is no plan to support DWARF-5 features used by clang which may need -fdebug-types-section for clang-built binaries or no size optimization of clang-built debug info at all. | |||
* DWZ disadvantage: Compilation (linking) requires for C++ up to 2x as big disk space (as DWZ is processing files after linker and DWZ is incompatible with -fdebug-types-section) | |||
* DWZ disadvantage: Compilation (linking) is slower | |||
This proposed DWARF format was originally submitted already for Fedora 18 as [[Features/DebugTypesSections]]. | |||
== Feedback == | == Feedback == | ||
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. --> | <!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. --> | ||
[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/HFMALNUAFJVEEEAIBLA45R5MFVXHVUUG/ Fedora development mailing list thread] | |||
Jakub Jelinek (and at least some people from GCC+GDB Red Hat teams) [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/D542XJTMEQDO3PVLMVZVZWI3GE5IACNW/ disagrees with this Change]. LLDB/clang/LLVM Red Hat team (incl. me) in general supports this Change. | |||
== Benefit to Fedora == | == Benefit to Fedora == | ||
| Line 90: | Line 104: | ||
https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack) | https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack) | ||
--> | --> | ||
* Better compatibility with existing debugging and tracing tools, primarily [https://lldb.llvm.org/ LLDB]. | |||
* Less resource-intensive rebuilds of C++ packages (in disk space, memory requirements and compilation time). | |||
== Scope == | == Scope == | ||
* Proposal owners: | * Proposal owners: It affects all packages generating *-debuginfo.rpm, that is compiled (not scripted) languages. | ||
<!-- What work do the feature owners have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | <!-- What work do the feature owners have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | ||
* Other developers: | * Other developers: Report any possible debuginfo incompatibility (unexpected). | ||
<!-- What work do other developers have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | <!-- What work do other developers have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | ||
* Release engineering: [https://pagure.io/releng/ | * Release engineering: [https://pagure.io/releng/issue/9769 #9769] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required? include a link to the releng issue. | <!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required? include a link to the releng issue. | ||
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | ||
* Policies and guidelines: | * Policies and guidelines: All the needed changes should be done in [https://src.fedoraproject.org/rpms/redhat-rpm-config redhat-rpm-config]. The [https://src.fedoraproject.org/rpms/dwz dwz package] can be then retired. | ||
<!-- Do the packaging guidelines or other documents need to be updated for this feature? If so, does it need to happen before or after the implementation is done? If a FPC ticket exists, add a link here. --> | <!-- Do the packaging guidelines or other documents need to be updated for this feature? If so, does it need to happen before or after the implementation is done? If a FPC ticket exists, add a link here. --> | ||
| Line 108: | Line 124: | ||
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. --> | <!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. --> | ||
* Alignment with Objectives: | * Alignment with Objectives: The size differences are only for *-debuginfo.rpm which is outside of scope of the listed objectives. | ||
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider --> | <!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider --> | ||
| Line 115: | Line 131: | ||
<!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
As each *-debuginfo.rpm has to exactly match NVRA of its binary package the Fedora change compatibility is not applicable. Existing tools supporting DWZ will still support the DWZ file format in packages which have not been rebuilt. | |||
== How To Test == | == How To Test == | ||
| Line 132: | Line 148: | ||
--> | --> | ||
The change will update [https://src.fedoraproject.org/rpms/redhat-rpm-config redhat-rpm-config] by [https://people.redhat.com/jkratoch/redhat-rpm-config-fdebug-types-section.patch an -fdebug-types-section patch]. | |||
Then one can use rpmbuild to rebuild a package. For mock use -a|--addrepo with modified redhat-rpm-config.rpm (with increased NVRA). For packages already rebuilt in Koji nothing is needed. | |||
Test programs like lldb and gdb if they still can print source code, function parameters, variables etc. | |||
One should also verify integrated testsuites of tools like clang, lldb, gcc, binutils, gdb, elfutils or rpm are not regressing with the -fdebug-types-section option. | |||
One can also compare *.debug files built with/without DWZ and/or -fdebug-types-section using [https://src.fedoraproject.org/rpms/libabigail libabigail] utility dwdiff but that will be rather done by the change owner. | |||
== User Experience == | == User Experience == | ||
| Line 146: | Line 169: | ||
- Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system. | - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system. | ||
--> | --> | ||
No user visible change. This affects what tools can developers use. | |||
== Dependencies == | == Dependencies == | ||
<!-- What other packages (RPMs) depend on this package? Are there changes outside the developers' control on which completion of this change depends? In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate? Other upstream projects like the kernel (if this is not a kernel change)? --> | <!-- What other packages (RPMs) depend on this package? Are there changes outside the developers' control on which completion of this change depends? In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate? Other upstream projects like the kernel (if this is not a kernel change)? --> | ||
Mass rebuild is not required. Packages inherited from F-33 will just keep the DWZ format which stays incompatible with some tools but that has been so since Fedora 18. | |||
This change affects/enables: [[Changes/DebugInfoLldbIndex]] | |||
== Contingency Plan == | == Contingency Plan == | ||
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan? This might be as simple as "Revert the shipped configuration". Or it might not (e.g. rebuilding a number of dependent packages). If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. --> | <!-- If you cannot complete your feature by the final development freeze, what is the backup plan? This might be as simple as "Revert the shipped configuration". Or it might not (e.g. rebuilding a number of dependent packages). If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. --> | ||
* Contingency mechanism: | * Contingency mechanism: Revert the change in [https://src.fedoraproject.org/rpms/redhat-rpm-config redhat-rpm-config]. Fedora can continue using DWZ, just some debugging/tracing tools will stay incompatible. | ||
<!-- When is the last time the contingency mechanism can be put in place? This will typically be the beta freeze. --> | <!-- When is the last time the contingency mechanism can be put in place? This will typically be the beta freeze. --> | ||
* Contingency deadline: | * Contingency deadline: beta freeze | ||
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? --> | <!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? --> | ||
* Blocks release? | * Blocks release? No | ||
* Blocks product? | * Blocks product? N/A | ||
== Documentation == | == Documentation == | ||
<!-- Is there upstream documentation on this change, or notes you have written yourself? Link to that material here so other interested developers can get involved. --> | <!-- Is there upstream documentation on this change, or notes you have written yourself? Link to that material here so other interested developers can get involved. --> | ||
* [http://www.dwarfstd.org/doc/DWARF5.pdf DWARF-5] E.2 Using Type Units | |||
* [https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fdebug-types-section GCC -fdebug-types-section] | |||
== Release Notes == | == Release Notes == | ||
Latest revision as of 19:52, 12 October 2020
Debug Info Standardization (from DWZ to -fdebug-types-section)
Summary
Fedora 18 implemented Features/DwarfCompressor called DWZ. As the format remains specific to Linux distributions it became a burden to make existing debugging tools compatible with DWZ.
Owner
- Name: Jan Kratochvil
- Email: jan.kratochvil@redhat.com
This Change does not represent Red Hat's opinion, see #Feedback.
Current status
- Targeted release: Fedora 34
- Last updated: 2020-10-12
- FESCo issue: #2482
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
Debug info files *.debug contained in *-debuginfo.rpm are very big in general (x86_64 Fedora 32 distribution has debug/ directory of 82GB while all its other files are only 75GB). There exist several methods how to make the *-debuginfo.rpms at least a bit smaller. Fedora 18 started using DWZ tool (from Features/DwarfCompressor) while Google implemented the same goal in a different way called -fdebug-types-section.
Linux distributions use existing DWZ (Fedora/CentOS/RHEL, SuSE OSes, OpenMandriva, Debian, Ubuntu maybe others?). DWZ support is missing in crossplatform tools like LLDB or llvm-dwarfdump. -fdebug-types-section is used internally by Google (produced by clang).
- DWZ advantage: On the whole Fedora distro it saves 3.3% (5GB of the 157GB distribution size)
- If the 3.3% size increase is a concern I can implement a different optimization as a GCC post-processing phase which would require no changes in any DWARF consumers. DWZ would still makes the Fedora distribution smaller but only by 2.4%.
- With another optimization (reintroducing a common file but a very simple one just containing DW_UT_type units) DWZ still makes the Fedora distribution smaller but only by 1.8%.
- DWZ disadvantage: DWZ has currently less support across consumers (LLDB, llvm-dwarfdump)
- DWZ disadvantage: DWZ requires 8x times more complicated (LoC count) support in consumers than -fdebug-types-section
- DWZ disadvantage: Changes/DebugInfoLldbIndex (LLDB acceleration) is disabled by DWZ; unless DWZ gets excluded from clang-built binaries
- DWZ disadvantage: -fdebug-types-section unifies tools compatibility with format of binaries from rpms vs. binaries from edit-compile-debug cycle. DWZ is an additional time overhead for edit-compile-debug cycle while -fdebug-types-section even speeds up edit-compile-debug cycle (its linking phase).
- DWZ disadvantage: DWZ DWARF-5 support is a work-in-progress. DWZ has been blocking DWARF-5 for Fedora for 3.5 years and only after I have now proposed to drop DWZ Mark Wielaard has started porting DWZ to DWARF-5. It can be expected next DWARF extensions will remain unsupported again. Even currently there is no plan to support DWARF-5 features used by clang which may need -fdebug-types-section for clang-built binaries or no size optimization of clang-built debug info at all.
- DWZ disadvantage: Compilation (linking) requires for C++ up to 2x as big disk space (as DWZ is processing files after linker and DWZ is incompatible with -fdebug-types-section)
- DWZ disadvantage: Compilation (linking) is slower
This proposed DWARF format was originally submitted already for Fedora 18 as Features/DebugTypesSections.
Feedback
Fedora development mailing list thread
Jakub Jelinek (and at least some people from GCC+GDB Red Hat teams) disagrees with this Change. LLDB/clang/LLVM Red Hat team (incl. me) in general supports this Change.
Benefit to Fedora
- Better compatibility with existing debugging and tracing tools, primarily LLDB.
- Less resource-intensive rebuilds of C++ packages (in disk space, memory requirements and compilation time).
Scope
- Proposal owners: It affects all packages generating *-debuginfo.rpm, that is compiled (not scripted) languages.
- Other developers: Report any possible debuginfo incompatibility (unexpected).
- Release engineering: #9769 (a check of an impact with Release Engineering is needed)
- Policies and guidelines: All the needed changes should be done in redhat-rpm-config. The dwz package can be then retired.
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: The size differences are only for *-debuginfo.rpm which is outside of scope of the listed objectives.
Upgrade/compatibility impact
As each *-debuginfo.rpm has to exactly match NVRA of its binary package the Fedora change compatibility is not applicable. Existing tools supporting DWZ will still support the DWZ file format in packages which have not been rebuilt.
How To Test
The change will update redhat-rpm-config by an -fdebug-types-section patch.
Then one can use rpmbuild to rebuild a package. For mock use -a|--addrepo with modified redhat-rpm-config.rpm (with increased NVRA). For packages already rebuilt in Koji nothing is needed.
Test programs like lldb and gdb if they still can print source code, function parameters, variables etc.
One should also verify integrated testsuites of tools like clang, lldb, gcc, binutils, gdb, elfutils or rpm are not regressing with the -fdebug-types-section option.
One can also compare *.debug files built with/without DWZ and/or -fdebug-types-section using libabigail utility dwdiff but that will be rather done by the change owner.
User Experience
No user visible change. This affects what tools can developers use.
Dependencies
Mass rebuild is not required. Packages inherited from F-33 will just keep the DWZ format which stays incompatible with some tools but that has been so since Fedora 18.
This change affects/enables: Changes/DebugInfoLldbIndex
Contingency Plan
- Contingency mechanism: Revert the change in redhat-rpm-config. Fedora can continue using DWZ, just some debugging/tracing tools will stay incompatible.
- Contingency deadline: beta freeze
- Blocks release? No
- Blocks product? N/A
Documentation
- DWARF-5 E.2 Using Type Units
- GCC -fdebug-types-section