From Fedora Project Wiki
 
(25 intermediate revisions by 2 users not shown)
Line 24: Line 24:


== Current status ==
== Current status ==
[[Category:ChangeAnnounced]]
[[Category:ChangeAcceptedF35]]
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->  
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->  
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
Line 39: Line 39:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* FESCo issue: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/2597 #2597]
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1956952 #1956952]
* Release notes tracker: <will be assigned by the Wrangler>
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/694 #694]
* fedora-devel thread: [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NWCFXZXZW4IIHMXE3MDIJH3PWGEJWVJB/ 2021-04-07]


== Detailed Description ==
== Detailed Description ==
Line 60: Line 61:
See also: [https://www.mankier.com/8/debuginfod#Security] and [https://www.mankier.com/1/debuginfod-find#Security].
See also: [https://www.mankier.com/8/debuginfod#Security] and [https://www.mankier.com/1/debuginfod-find#Security].


* What kinds of attacks might be possible with malicious debuginfo files?
'''What kinds of attacks might be possible with malicious debuginfo files?'''


It depends on the consumer tool - on their robustness, the privileges under which they operate, and the privilege of a resulting artifact if any.
It depends on the consumer tool - on their robustness, the privileges under which they operate, and the privilege of a resulting artifact if any.


=== How is it verified that files received from debuginfo servers have not been tampered with? ===
'''How is it verified that files received from debuginfo servers have not been tampered with?'''


Debuginfod servers provide the verbatim contents of the verbatim distro archives, and transmit them securely across HTTPS.  There is no per-file signing infrastructure in Fedora, and debuginfod doesn't add one.  Thus there is no mechanism to manually verify these files, beyond downloading a corresponding signed archive out-of-band and comparing.  The client side code will be taking some rudimentary measures with file permissions to reduce risk of accidental change.  In principle, if the received files were tampered with, then the same tamperers could mess with the user's consumer tools and/or take over the account.
Debuginfod servers provide the verbatim contents of the verbatim distro archives, and transmit them securely across HTTPS.  There is no per-file signing infrastructure in Fedora, and debuginfod doesn't add one. (But see also [[Changes/Signed_RPM_Contents]].) Thus there is no mechanism to manually verify these files, beyond downloading a corresponding signed archive out-of-band and comparing.  The client side code will be taking some rudimentary measures with file permissions to reduce risk of accidental change.  In principle, if the received files were tampered with, then the same tamperers could mess with the user's consumer tools and/or take over the account.


=== Is there any end-to-end authentication from the Fedora build system to my workstation – which there is with signed debuginfo packages – or do the tools blindly trust a whole network of federated debuginfo servers? ===
OTOH, [https://sourceware.org/bugzilla/show_bug.cgi?id=27758 there is an RFE] to provide optional client-side cache content verification.  That is, we may be able to help detect previous ''client-side'' tampering or data corruption.  We may be able to implement this in time for F35, if this capability is deemed useful & necessary.
 
'''Is there any end-to-end authentication from the Fedora build system to my workstation – which there is with signed debuginfo packages – or do the tools blindly trust a whole network of federated debuginfo servers?'''


This change proposal does not relate to the whole network of federated debuginfo servers, but only to the Fedora official ones, which take content from the official Fedora build system only.
This change proposal does not relate to the whole network of federated debuginfo servers, but only to the Fedora official ones, which take content from the official Fedora build system only.


=== Does (calling "home" to debuginfod) happen every time, or are downloaded files cached locally? ===
'''Privacy: what information is sent to the server?'''
 
Whenever a debugger type tool needs information, it would send an query across HTTPS, passing:
* inherently, the machine's IP address
* the hex buildid of the binary it is interested in
* if requesting a source file (usually if the debuginfo has already been found), then that source file's name
* a User-Agent: string identifying its version of fedora and elfutils, and the architecture name
 
This could disclose the existence of debugging activity to the servers.  It would be stored temporarily in the general system-wide logs in fedora-infra, along with usual web server and other logs.
 
'''Does (calling "home" to debuginfod) happen every time, or are downloaded files cached locally?'''


Downloaded files are indeed cached locally.  A cache hit means no calling home.  See the [https://www.mankier.com/3/debuginfod_find_debuginfo debuginfod_find_debuginfo] man page.
Downloaded files are indeed cached locally.  A cache hit means no calling home.  See the [https://www.mankier.com/3/debuginfod_find_debuginfo debuginfod_find_debuginfo] man page.  If we add the client-side verification option and it is activated, then there'd be more "calling home".


=== If there is a cache, when are old files purged from the cache? ===
'''If there is a cache, when are old files purged from the cache?'''


The same man page identifies the config file parameters that govern this cleaning.  The defaults are to retain files accessed during the last week.
The same man page identifies the config file parameters that govern this cleaning.  The defaults are to retain files accessed during the last week.
'''Does this make the debuginfod server a juicy target for attackers?
'''
Yes.  If it were possible to break into the server and feed crafted DWARF data to debugger programs, bad things could happen.  We would welcome assistance with security auditing or other advice.  To ameliorate this concern, consider:
* The code is C++, so normal C str* goo is not done on user input.
* It is regularly run under valgrind for testing.
* In current deployment, it is already doing detailed logging to centralized fedora-infra `log01` servers.
* The server is already under constant monitoring by prometheus and collectd, so unusual performance characteristics are logged & noticeable.
* We can imagine some automated integrity monitoring, such as periodically testing random or key RPMs for a match between locally signature-checked content and debuginfod-supplied content, though this might not be hard to defeat.
* In the worst case, we can pull the server offline within a minute.  This leaves clients in the same state as before this feature existed: relying on manual downloading.


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 94: Line 119:


The work could consist one extra parameter in the `elfutils.spec` `%configure`. Its effect is to arrange for the `elfutils-debuginfod-client` RPM
The work could consist one extra parameter in the `elfutils.spec` `%configure`. Its effect is to arrange for the `elfutils-debuginfod-client` RPM
to install an `/etc/profile.d` file that sets the `DEBUGINFOD_URLS` environment variable automatically to `https://debuginfod.fedoraproject.org/`.  (At the time of this writing, the _staging_ server is getting ready for testing: `https://debuginfod.stg.fedoraproject.org/`.)
to install an `/etc/profile.d` file that sets the `DEBUGINFOD_URLS` environment variable automatically to `https://debuginfod.fedoraproject.org/`.  (The _staging_ test server is also available for use: `https://debuginfod.stg.fedoraproject.org/`.)


* Other developers: None - relevant code has been previously upstreamed! <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: None - relevant code has been previously upstreamed! <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Line 115: Line 140:
None.
None.


Note that these servers will index all active Fedora releases (32-), all architectures, so users of those versions can already set `DEBUGINFOD_URLS` manually to take advantage.
Note that these servers index all active Fedora releases (32-), all architectures, so users of those versions can already set `DEBUGINFOD_URLS` manually to take advantage.


== How To Test ==
== How To Test ==


* Install `elfutils-debuginfod-client`
* Install `elfutils-debuginfod-client` - but it is likely already installed.
* Open arbitrary fedora binary via gdb.
* Open arbitrary fedora binary via gdb.
** Admire the immediate downloading of debuginfo and source code.
** Admire the immediate downloading of debuginfo and source code.
Line 129: Line 154:


Primarily: users running debuggers, profilers, tracing tools on internet-capable machines can work immediately, without switching to privileged users and fragile manual `dnf` commands to install this data.
Primarily: users running debuggers, profilers, tracing tools on internet-capable machines can work immediately, without switching to privileged users and fragile manual `dnf` commands to install this data.
Users running debuggers on programs with large shared library dependencies (such as GUIs) may experience a [https://sourceware.org/bugzilla/show_bug.cgi?id=17547 long first-time download], as gdb eagerly fetches all debuginfo at startup.
In case of some rare types of outages, a request may take up to $DEBUGINFOD_TIMEOUT (default of 90 seconds) to fail.  A user experiencing this should consider turning off the facility temporarily, or to use the backup staging server.


== Dependencies ==
== Dependencies ==
Line 145: Line 174:


There is upstream documentation in the debugging tools as well as associated with the client code / cli tooling.  What our Release Notes would focus on however is the _automatic activation_ of this facility via the environment variable.
There is upstream documentation in the debugging tools as well as associated with the client code / cli tooling.  What our Release Notes would focus on however is the _automatic activation_ of this facility via the environment variable.
Accessing the main debuginfod.fedoraproject.org URL directly redirects people to [[Debuginfod|this page]], with further usage information.


== Release Notes ==
== Release Notes ==


TBD.
Many debugging-related tools (gdb, systemtap, perf, valgrind, bpftrace, dwarves, ...) now use the Fedora Project's `debuginfod` service by default.  This lets them automatically download debugging data and source code from the https://debuginfod.fedoraproject.org/ server for Fedora packages.  This makes debugging and tracing compiled distro programs super easy.  If you're a build engineer or ISV, consider running a server for your team or for users.
 
While performance is generally fast, you may notice first-time download times for programs with numerous or large shared libraries.  This content is saved under $HOME/.cache, and is periodically cleaned.  You may opt out of the service by clearing your $DEBUGINFOD_URLS environment variable in your .bashrc or equivalent.  From elfutils version 0.186 or above, you can also set the $DEBUGINFOD_MAXTIME=$SECONDS and/or $DEBUGINFOD_MAXSIZE=$BYTES environment variables to allow only fast and/or small downloads.


<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->

Latest revision as of 17:22, 21 September 2021


Debuginfod By Default

Summary

Fedora users / developers who need to debug/trace distro binaries can make use of the recently activated elfutils-debuginfod servers to automatically fetch debugging data and source code, instead of having to use # sudo dnf commands.

Owner

Current status

Detailed Description

Numerous fedora debugging-type tools have built-in capabilities to use the debuginfod protocol to fetch debuginfo/source code automatically. We would like to activate a setting so that Fedora's debuginfod servers are automatically used, rather than requiring hand-editing individual users' dot files.

Feedback

There has existed broad interest in a Fedora debuginfod server since the project was proposed / announced in 2020, and several distros already operate public servers of this nature. Some of the distros configure their installations by default to talk to those servers, some do not.

Turning on this by default has some limited privacy implications. Some Debian users have expressed concerns that this facility "calls home" during debugging, so it may expose a limited amount of information about what a user is debugging. The information is limited to the build-id and source code file names of programs being debugged, and is only sent to the servers if their machine lacks locally installed debuginfo. Whether this should be opt-in or opt-out and how has been resolved there via an install-time query to the sysadmin. In contrast, on OpenSUSE Tumbleweed, it is simply defaulted-on, and we have heard of no controversy.

We anticipate discussing this topic on the mailing list, and noting it in the release notes either way.

Security FAQ

See also: [1] and [2].

What kinds of attacks might be possible with malicious debuginfo files?

It depends on the consumer tool - on their robustness, the privileges under which they operate, and the privilege of a resulting artifact if any.

How is it verified that files received from debuginfo servers have not been tampered with?

Debuginfod servers provide the verbatim contents of the verbatim distro archives, and transmit them securely across HTTPS. There is no per-file signing infrastructure in Fedora, and debuginfod doesn't add one. (But see also Changes/Signed_RPM_Contents.) Thus there is no mechanism to manually verify these files, beyond downloading a corresponding signed archive out-of-band and comparing. The client side code will be taking some rudimentary measures with file permissions to reduce risk of accidental change. In principle, if the received files were tampered with, then the same tamperers could mess with the user's consumer tools and/or take over the account.

OTOH, there is an RFE to provide optional client-side cache content verification. That is, we may be able to help detect previous client-side tampering or data corruption. We may be able to implement this in time for F35, if this capability is deemed useful & necessary.

Is there any end-to-end authentication from the Fedora build system to my workstation – which there is with signed debuginfo packages – or do the tools blindly trust a whole network of federated debuginfo servers?

This change proposal does not relate to the whole network of federated debuginfo servers, but only to the Fedora official ones, which take content from the official Fedora build system only.

Privacy: what information is sent to the server?

Whenever a debugger type tool needs information, it would send an query across HTTPS, passing:

  • inherently, the machine's IP address
  • the hex buildid of the binary it is interested in
  • if requesting a source file (usually if the debuginfo has already been found), then that source file's name
  • a User-Agent: string identifying its version of fedora and elfutils, and the architecture name

This could disclose the existence of debugging activity to the servers. It would be stored temporarily in the general system-wide logs in fedora-infra, along with usual web server and other logs.

Does (calling "home" to debuginfod) happen every time, or are downloaded files cached locally?

Downloaded files are indeed cached locally. A cache hit means no calling home. See the debuginfod_find_debuginfo man page. If we add the client-side verification option and it is activated, then there'd be more "calling home".

If there is a cache, when are old files purged from the cache?

The same man page identifies the config file parameters that govern this cleaning. The defaults are to retain files accessed during the last week.

Does this make the debuginfod server a juicy target for attackers?

Yes. If it were possible to break into the server and feed crafted DWARF data to debugger programs, bad things could happen. We would welcome assistance with security auditing or other advice. To ameliorate this concern, consider:

  • The code is C++, so normal C str* goo is not done on user input.
  • It is regularly run under valgrind for testing.
  • In current deployment, it is already doing detailed logging to centralized fedora-infra log01 servers.
  • The server is already under constant monitoring by prometheus and collectd, so unusual performance characteristics are logged & noticeable.
  • We can imagine some automated integrity monitoring, such as periodically testing random or key RPMs for a match between locally signature-checked content and debuginfod-supplied content, though this might not be hard to defeat.
  • In the worst case, we can pull the server offline within a minute. This leaves clients in the same state as before this feature existed: relying on manual downloading.

Benefit to Fedora

This will improve developers' experience.

It may reduce download server burden, as only individual ELF/DWARF/source files are downloaded rather than entire -debuginfo and -debugsource RPMs.

It would help Fedora catch up to other distros who put up debuginfod servers already. :-)

Scope

  • Proposal owners:

The work could consist one extra parameter in the elfutils.spec %configure. Its effect is to arrange for the elfutils-debuginfod-client RPM to install an /etc/profile.d file that sets the DEBUGINFOD_URLS environment variable automatically to https://debuginfod.fedoraproject.org/. (The _staging_ test server is also available for use: https://debuginfod.stg.fedoraproject.org/.)

  • Other developers: None - relevant code has been previously upstreamed!
  • Release engineering: None - our team is operating the debuginfod[.stg].fedoraproject.org VMs.
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Objectives: N/A


Upgrade/compatibility impact

None.

Note that these servers index all active Fedora releases (32-), all architectures, so users of those versions can already set DEBUGINFOD_URLS manually to take advantage.

How To Test

  • Install elfutils-debuginfod-client - but it is likely already installed.
  • Open arbitrary fedora binary via gdb.
    • Admire the immediate downloading of debuginfo and source code.
  • Run eu-stack -v -p $pid for an arbitrary process.
    • Admire the immediate downloading of debuginfo to give precise file:line data.

User Experience

Primarily: users running debuggers, profilers, tracing tools on internet-capable machines can work immediately, without switching to privileged users and fragile manual dnf commands to install this data.

Users running debuggers on programs with large shared library dependencies (such as GUIs) may experience a long first-time download, as gdb eagerly fetches all debuginfo at startup.

In case of some rare types of outages, a request may take up to $DEBUGINFOD_TIMEOUT (default of 90 seconds) to fail. A user experiencing this should consider turning off the facility temporarily, or to use the backup staging server.

Dependencies

The debuginfod servers at fedora-infra need to be up.

Contingency Plan

  • Contingency mechanism: change the elfutils-debuginfod-client subrpm to not set the default DEBUGINFOD_URLS environment variable for all users. In the case of a server outage, the debugger tools revert to debuginfo-less operation, prior to this feature.
  • Contingency deadline: shortly before freeze
  • Blocks release? No

Documentation

There is upstream documentation in the debugging tools as well as associated with the client code / cli tooling. What our Release Notes would focus on however is the _automatic activation_ of this facility via the environment variable.

Accessing the main debuginfod.fedoraproject.org URL directly redirects people to this page, with further usage information.

Release Notes

Many debugging-related tools (gdb, systemtap, perf, valgrind, bpftrace, dwarves, ...) now use the Fedora Project's debuginfod service by default. This lets them automatically download debugging data and source code from the https://debuginfod.fedoraproject.org/ server for Fedora packages. This makes debugging and tracing compiled distro programs super easy. If you're a build engineer or ISV, consider running a server for your team or for users.

While performance is generally fast, you may notice first-time download times for programs with numerous or large shared libraries. This content is saved under $HOME/.cache, and is periodically cleaned. You may opt out of the service by clearing your $DEBUGINFOD_URLS environment variable in your .bashrc or equivalent. From elfutils version 0.186 or above, you can also set the $DEBUGINFOD_MAXTIME=$SECONDS and/or $DEBUGINFOD_MAXSIZE=$BYTES environment variables to allow only fast and/or small downloads.