File U_0003-Fix-rendering-on-big-endian-system.patch of Package tigervnc

Overview Repositories Revisions Requests Users Attributes Meta

File U_0003-Fix-rendering-on-big-endian-system.patch of Package tigervnc

From cc647dbd40398e5d56a5b5d649f3bb20d90608e6 Mon Sep 17 00:00:00 2001
From: Pierre Ossman <ossman@cendio.se>
Date: Thu, 25 Oct 2018 10:36:21 +0200
Subject: [PATCH] Fix rendering on big endian system

Our XRender code assumes a certain pixel layout which was not
guaranteed on big endian systems. The previous workaround only worked
for some cases, so fix this properly now.
---
 vncviewer/Surface_X11.cxx | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/vncviewer/Surface_X11.cxx b/vncviewer/Surface_X11.cxx
index 3523da3d..6562634d 100644
--- a/vncviewer/Surface_X11.cxx
+++ b/vncviewer/Surface_X11.cxx
@@ -109,6 +109,7 @@ void Surface::blend(Surface* dst, int src_x, int src_y, int x, int y, int w, int
 
 void Surface::alloc()
 {
+  XRenderPictFormat templ;
   XRenderPictFormat* format;
 
   // Might not be open at this point
@@ -117,7 +118,37 @@ void Surface::alloc()
   pixmap = XCreatePixmap(fl_display, XDefaultRootWindow(fl_display),
                          width(), height(), 32);
 
-  format = XRenderFindStandardFormat(fl_display, PictStandardARGB32);
+  // Our code assumes a BGRA byte order, regardless of what the endian
+  // of the machine is or the native byte order of XImage, so make sure
+  // we find such a format
+  templ.type = PictTypeDirect;
+  templ.depth = 32;
+  if (XImageByteOrder(fl_display) == MSBFirst) {
+    templ.direct.alpha = 0;
+    templ.direct.red   = 8;
+    templ.direct.green = 16;
+    templ.direct.blue  = 24;
+  } else {
+    templ.direct.alpha = 24;
+    templ.direct.red   = 16;
+    templ.direct.green = 8;
+    templ.direct.blue  = 0;
+  }
+  templ.direct.alphaMask = 0xff;
+  templ.direct.redMask = 0xff;
+  templ.direct.greenMask = 0xff;
+  templ.direct.blueMask = 0xff;
+
+  format = XRenderFindFormat(fl_display, PictFormatType | PictFormatDepth |
+                             PictFormatRed | PictFormatRedMask |
+                             PictFormatGreen | PictFormatGreenMask |
+                             PictFormatBlue | PictFormatBlueMask |
+                             PictFormatAlpha | PictFormatAlphaMask,
+                             &templ, 0);
+
+  if (!format)
+    throw rdr::Exception("XRenderFindFormat");
+
   picture = XRenderCreatePicture(fl_display, pixmap, format, 0, NULL);
 
   visFormat = XRenderFindVisualFormat(fl_display, fl_visual->visual);
-- 
2.31.1

Places

File U_0003-Fix-rendering-on-big-endian-system.patch of Package tigervnc

Places