How to debug Wayland problems

Wayland is intended as a simpler replacement for X11. It changes the design of a Linux desktop architecture considerably. Unlike X11, there is no dedicated standalone server in Wayland. What was previously done between the app, its toolkit, the Xserver and the window manager is now shared between the app, its toolkit and the Wayland compositor which manages the compositing, input, windows management, etc. The apps and toolkits are now in charge of their own rendering and decorations (client side decorations), so any issues usually sit between the toolkit (e.g. GTK+) and the Wayland compositor (e.g. mutter).

You can read more about Wayland on the GNOME Wayland initiative wiki page. You can read more about the current state of Wayland features on Wayland features page.

Identifying Wayland problems

Are you running a Wayland session?

In GNOME, there's a gear button at the login screen which can be used to either log into standard GNOME desktop (X11 session), or GNOME on Wayland session (if you have a password-less user account, you won't see the gear icon, it is displayed only when the password prompt appears). Use the gear button to determine type of session you're logging into. If you want to start your session in a different way, read the advanced techniques for trying Wayland.

In KDE, there is support for running a nested Wayland session inside your X11 session. You'll need to install kwin-wayland package and then follow up with this howto. There doesn't seem to be out-of-the-box support for running a full Wayland session at the moment.

Other desktop environments are not currently capable of running a Wayland session.

If you want to figure out which type of session you're running right now, without logging out and in again, you can use several ways to figure it out:

• Wayland session should have WAYLAND_DISPLAY variable set, X11 session should not have it:

  $ echo $WAYLAND_DISPLAY 
  wayland-0

• loginctl can give you this information. First run loginctl and find your session number (if should be an integer number, with your username and seat assigned). Then look at the session type (x11 or wayland):

  $ loginctl show-session <YOUR_NUMBER> | grep Type
  Type=x11

If you're not running X11 session, not a Wayland session, your problems are not related to Wayland. It's a bug either in that particular application, or X11 itself, see How to debug Xorg problems.

Does your application run on Wayland natively, or uses XWayland (X11 compatibility layer)?

It is important to know whether the problematic application is a native Wayland application, or runs through XWayland, which allows legacy applications to still run on top of Xorg server, but display in a Wayland session.

There are several ways how to identify whether an application is using Wayland or XWayland:

• Select the window using xwininfo or xprop. Run:

  $ xwininfo

  Your mouse pointer should change to a cross under X11, it doesn't seem to change under Wayland. Now click anywhere inside the app window you want to test. If the xwininfo command finishes (it should print window properties into the terminal), the app under test is running under XWayland. If nothing happens (the xwininfo command is still waiting until you select a window), the app under test is running under Wayland (you can close the command with Ctrl+C).
  You can also use xprop command using the same instructions.

• XWayland applications are listed in xlsclients output. Run:

  $ xlsclients

  However, this list of not always entirely reliable, some apps might be missing.

• You can try to run the app while unsetting DISPLAY environment variable:

  $ DISPLAY='' command

  If the application runs OK, it should be using Wayland natively.

• You can run the app with WAYLAND_DEBUG=1 environment variable:

  $ WAYLAND_DEBUG=1 command

  If you see loads of output (when compared to a standard run), the app is using Wayland natively.

• Under GNOME, you can determine this using integrated Looking Glass tool. Hit Alt+F2, run:

  lg

  click on Windows in the upper right corner of the tool and select desired window by clicking on its name. If you see MetaWindowWayland in the first line, this app is running under Wayland. If you see MetaWindowX11 in the first line, this app is running under X11.

If you have identified the problem to be in a XWayland application, try to reproduce the issue in a standard X11 session. If it happens as well, it is not related to Wayland, it's a bug either in that particular application, or Xorg server, see How to debug Xorg problems. If the problem happens only under XWayland but not in an X11 session, it should still be reported against Xorg server (xorg-x11-server package), because XWayland is included in it (as xorg-x11-server-Xwayland subpackage).

Identifying problem component

Wayland itself is a protocol and the problem is rarely in the protocol itself. Rather, the problem is likely to be in the app or its toolkit, or in the compositor.

The most notable Wayland-ready toolkits are:

• GTK+ 3 - default apps in GNOME environment use almost exclusively this toolkit. Please note that apps using older GTK+ 2 are not Wayland-ready.
• QT 5 - many apps in KDE environment use this toolkit. Please note that apps using older QT 4 are not Wayland-ready.

The most notable Wayland compositors are:

• Weston - the reference implementation of a Wayland compositor, maintained directly by the Wayland project
• Mutter - compositor in GNOME. If you run GNOME, it is using this compositor.
• Kwin - compositor in KDE. If you run KDE, it is using this compositor.

Testing under different compositors

If you experience a problem with a Wayland app, it is very useful to know whether the problem is present under just a single compositor (in that case it's likely a compositor bug) or under multiple compositors (in that case it's likely an app/toolkit bug).

Please run your session with the reference Weston compositor and try to reproduce the issue. You can either run Weston as a nested window, or as a full session. First, install weston package (you can read many useful information in its man page):

$ sudo dnf install weston

Then create a config file which will specify that you want to have XWayland support enabled in your weston sessions. Create ~/.config/weston.ini with this contents:

[core]
modules=xwayland.so

Now you can start weston either as nested window or as a full session.

• To start a nested Weston window, run this from a terminal:

  $ weston

  A Weston window should open and you should see and a terminal icon in its top left corner. Use that icon to launch a terminal and from that you can run apps and other commands using Weston. Exit the compositor by simply closing the window or killing the weston process.

• To start a full Weston session (not nested inside another X11 or Wayland session), switch to a free VT (Ctrl+Alt+Fx) and run:

  $ weston-launch

  You can exit the session by pressing Control+Alt+Backspace shortcut.

If you can reproduce the issue with Weston, file an issue against the app or its toolkit (gtk+, qt, etc). Otherwise file the issue against the compositor your environment uses (mutter, kwin, etc). If the problem occurs only with XWayland apps but not native Wayland apps, report a bug against Xorg server.

Reporting the issue

Using up-to-date software

Before reporting the bug, please make sure you use the latest available software. You need to run on Fedora 23 or later, older Fedora versions are not going to get updated with latest Wayland fixes. Make sure there are no system updates waiting:

$ sudo dnf update

If there are (and the available updates look plausibly related to the components you're seeing issues with), please update the system and verify whether the issue is still present or has been fixed.

Looking for similar reports

In order to avoid duplicate reports and also wasting your time debugging something someone has maybe already debugged, please search through the existing reports first. The most visible issues or concerns are listed in Known issues, frequent complaints, fundamental changes. If you don't see it there, you need to search deeper. You can find Wayland related issues most likely in here:

• mutter/wayland and GTK+/wayland in GNOME Bugzilla
• Wayland-related issues tracker across GNOME Bugzilla
• Wayland-related issues tracker across Red Hat Bugzilla
• Wayland in Red Hat Bugzilla
• Wayland in Freedesktop Bugzilla
• Google search

Filing a bug

After you've identified against which component to (most probably) report the issue and found no existing report of it, there are several places where to report it:

• Red Hat Bugzilla - recommended for issues related to wayland itself, weston compositor, non-GNOME apps, KDE project, QT toolkit
• GNOME Bugzilla - recommended for issues related to mutter compositor, GTK+ toolkit, applications under the GNOME project (most of default apps in Fedora Workstation)

When reporting the issue, please make your report block our tracker, so that we have a good overall picture of what is broken across many different components. In your bug report, set Blocks: WaylandRelated (one of the fields available after you've reported the bug, you might need to toggle showing advanced fields). That will make it block one of these trackers, depending where you reported the bug:

• Wayland Tracker in Red Hat Bugzilla
• Wayland Tracker in GNOME Bugzilla

Information to include in your bug report

1. System journal. Since there is no unique server like the X11 server, most of the important information will come from the the Wayland compositor and the apps. All of that should be in system journal nowadays. You can save a full journal since last boot like this:

   $ journalctl -ab > journal.log

   You can also edit the file and according to the timestamps remove everything long prior to when the issue occurred, in order to make the log more succinct.

   • If your system crashed or became unresponsive so that you had to reboot it, you can see the journal from the previous boot using journalctl -a -b -1 instead.
2. Wayland debug output. If you can reproduce the issue, please run the problematic app like this:

   $ WAYLAND_DEBUG=1 command |& tee debug.out

   You should see loads of output being printed out. It will involve all communication between the app and the compositor.

3. Information whether the same problem occurs when you run the app using XWayland instead of Wayland. For GTK+ 3 apps, you can force a native Wayland app to run using XWayland like this:

   $ GDK_BACKEND=x11 command

   Vice versa, you can also force a XWayland app to run using Wayland (in case it has just experimental support):

   $ GDK_BACKEND=wayland command

   All of this applies to just GTK+ 3 apps. FIXME: what about other toolkits, like QT 5?

4. Hardware description is useful for some hardware-related bugs:

   $ lspci -nn > lspci.out

5. Package versions. You can collect the list and versions of all your packages installed using:

   $ rpm -qa | sort > packages.out

6. The usual information that every bug report should have.

Debugging gnome-shell

If gnome-shell gets stuck and unresponsive, it's very helpful to obtain a backtrace from its process and attach it to the report. If this happens, switch to a different VT if possible (Ctrl+Alt+F3 through F7), or log in using ssh. First install debug symbols:

$ sudo dnf debuginfo-install `rpm -q gnome-shell`

Then attach gdb debugger to your gnome-shell process:

$ gdb -p `pgrep -U $(id -un) -x gnome-shell`
...
(gdb) set logging on
(gdb) thread apply all backtrace full
... press Enter until the whole backtrace is displayed ...
(gdb) quit

You should have the backtrace saved in gdb.txt file.

Debugging mutter

You can debug mutter (used in gnome-shell) by setting its environment variables. These need to be set prior to run gnome-shell, so if you want to log into GNOME from GDM, you need to create a wrapper script called from a desktop file in /usr/share/wayland-sessions.

FIXME: Putting the wrapper script and desktop file here would be helpful.

Known issues, frequent complaints, fundamental changes

Here we will list high-profile issues which are known to be broken, not yet implemented, or intentionally behaving differently from regular X11 apps. Also please look at Wayland features which lists all current missing or in-progress features and their details.

To see all known issues, look at Bugzilla reports as mentioned in Looking for similar reports.

Windows appear in top-left corner or on a wrong monitor

There was a race condition when determining window location. As a result, most of the time your apps opened in the top left corner on the left-most monitor (if you have more than one). That should hopefully be already fixed, we just need to wait until a new build including the fix is released as an update.

Some windows resize themselves when you switch focus

When have a certain window in focus, and then switch focus to a different window, the former window might resize itself. This often happens with virtual machines like virt-manager, or Evince. This problem is fixed already, we're just waiting for a fixed build to be released as an update.

There is no middle-click clipboard selection

Under X11, you have one clipboard accessible through well-known ctrl+c, ctrl+v shortcuts, and another clipboard (called primary selection) that copies everything you highlight using the cursor and pastes text by pressing the middle mouse button. The former clipboard is still accessible in Wayland programs, but the middle-click clipboard is no longer supported. That is a design choice, not a bug, though quite contentious. Read Primary Selection design page to learn about some ideas how to mitigate this change.

Drag and drop always makes copies

The current implementation of drag&drop only supports the copying operation. So if you drag and drop files in Nautilus or emails in Evolution or use some other app, the behavior will likely be different than what you're used to in X11. Other modes of operation are in development. See bug 755625.

Drag and drop is not supported in certain applications

In some cases, drag and drop is still not supported. This is mainly visible when trying to drop a file from Nautilus to Firefox - nothing happens. At the moment, you need to use file dialogs to upload files to websites in Firefox. See bug 758395.

Moving a maximized window between monitors freezes gnome-shell

There is a nasty bug which freezes your whole graphical environment if you use the GNOME's overview mode to move a maximized window from one monitor to a different one. See bug 757676. The only solution then is to hard-reboot your computer or connect from a remote computer and force-restart your GNOME session. Until this bug is fixed, either unmaximize your window before moving it between monitors, or use keyboard shortcuts to do that.

Alt+Tab does not always switch focus correctly between XWayland and Wayland apps

There is a rare issue when you switch windows using Alt+Tab ⇆ (or ⊞ Win+Tab ⇆) and the requested window is displayed on top, but keyboard focus is still in the previous window. So if you start typing, it will go to the background window, not the foreground one. You'll need to switch between them again, or use mouse pointer and click inside the foreground window (maybe even several times) to correctly switch the focus. See bug 756272.

Graphical applications can't be run as root from terminal

It is not possible to start graphical apps under the root account from terminal when using su or sudo. Apps which use polkit to request administrator permissions for just certain operations and only when needed are not affected (they are not started as root right away). The discussion is ongoing about the best approach to take, see bug 1274451 and "On running gui applications as root" thread in fedora-devel mailing list.

Many well-known X11 utilities don't work

Power users are familiar with a large range of X11-related utilities, like xkill, xrandr, xdotool, xsel. These tools won't work under Wayland session, or will only work with XWayland applications but not Wayland applications. Some tools might have a replacement which allows to perform similar tasks.

FIXME: add some Wayland-ready replacements for popular X11 tools

Mouse locking and relative mouse movement is not implemented yet, affecting games

Games often need to lock your mouse pointer at a single place, and receive relative movement events instead of absolute coordinates. This is not yet implemented in Wayland, so your cursor might move even when it is not supposed to or it might leave game window boundaries even when it should stay locked inside the window. See bug 744104.

Games and other apps can't change monitor resolution

It is no longer possible for an app to change monitor resolution. Usually this was done by games to increase performance. Wayland-based games will use a different approach - scaling its output. But for X11 games (running through XWayland) this solution is not available. This results in a number of different types of behavior, based on how the game is written - the game might be fixed in the desktop resolution, or rendered as a small centered image with black bars around it, or crash on startup, or something different. See bug 1289714.

For some games, a possible workaround is to manually set custom monitor resolution before running the game, if you really need it. It will not help always, though.

Some games crash on start due to missing XFree86-VidModeExtension

XWayland does not implement XFree86-VidModeExtension, which some games require in order to start. See bug 87806.

Screen capture is not available with usual apps

One of the features of Wayland is its security design, which helps to guard the user against malicious apps. Apps can no longer see everything on the screen and spy on you. But that also means you cannot run a common application (like shutter or gtk-recordmydesktop) and use it to make a screenshot or a screencast of your desktop - it will see only its own window, but nothing else (or it might crash right away). System (trusted) apps need to be used to perform these actions.

In GNOME, you can use Screenshot tool (available in overview or as Printscreen hotkey or as gnome-screenshot command) to capture a screenshot of the full desktop or a particular window. You can press Ctrl+Alt+Shift+R keyboard shortcut to start video recording of the whole desktop (stop it by pressing the same shortcut again, there's an indicator in the upper right corner) and find the screencast in ~/Videos. For screencast, you can also use EasyScreenCast gnome-shell extension.

Desktop icons not showing in GNOME

If you configured GNOME to show icons on your desktop, this doesn't currently work under Wayland. See bug 1280195.

~/.bash_profile and ~/.profile are not processed

If you have some code in your ~/.bash_profile or ~/.profile files (e.g. adjusting your $PATH), it is executed when logging in into an X11 session, but not into a Wayland session. See bug 736660.

Display rotation is not yet supported in GNOME

Wayland itself is capable of having rotated displays, but support needs to be present in the compositor. GNOME does not yet allow to rotate your display in its control center. See bug 745079.

Mouse pointer is lagging/stuttering under load

If your computer is under load, your mouse pointer movement might stop being fluent, but start lagging (get stuck in a place for a short time, then jump to a different place instantly). This is probably more noticeable on slow systems/systems with fewer CPU cores. See bug 745032.

Mouse pointer sometimes disappears in XWayland applications

There is a rare issue when mouse pointer disappears in XWayland applications. See bug 1135661.

Keyboard events are sometimes quickly repeated

There is a rare issue when you press a key to type a letter and you'll see multiple copies of the letter typed in. See bug 757942.

Not all keys can be sent to a remote desktop or a virtual machine

Some applications forward all input, including system-specific keys/shortcuts like ⊞ Win or Alt+Tab ⇆, to a remote system. This is mostly remote desktop viewers like vncviewer or virtual machine managers like virt-manager or boxes. Under Wayland, some of these shortcuts can't be intercepted, and therefore are used in the host system, not the remote/guest system. See bug 1285770.

NumLock is always disabled on login

If you tend to use the numerical keypad on external keyboards, you'll notice that NumLock is currently disabled every time you log in. You need to enable it every time before using the numerical keys to input numbers. See bug 757943.