Fix crashes and non-portable functionality in QDBusDemarshaller QByteArray extraction

QDBusArgument QByteArray extraction operator and QDBusDemarshaller that
implements the extraction do not check the type of the extracted value.
When extracting a QByteArray when the value actually is e.g. a struct of
mixed types the byte array extraction will crash as it attempts to extract
the struct data as a fixed array.

The fix adds DBus type checks to QDBusArgument byte array extraction
operator implementations.
The checks invalidate extracting arrays of other types than bytes to a
QByteArray that worked with the unchecked implementation. The rationale
for this restriction is
1) extracting a QByteArray to a variant checks already that the array
   element type is byte
2) Results of extracting arrays of types wider than a byte to a QByteArray
   are architecture-dependent making such code inherently non-portable.

Task-number: QTBUG-22840
Change-Id: Iaa284603c65d7a431a3fd020c18240cd8199ceb9
(From Qt5 commit b9acd85b2f92f887521b952f84ced9a2d1a8a57e)
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>

2 files changed, 12 insertions, 2 deletions

diff --git a/src/dbus/qdbusargument_p.h b/src/dbus/qdbusargument_p.h
index e977c99..c04e458 100644
--- a/src/dbus/qdbusargument_p.h
+++ b/src/dbus/qdbusargument_p.h
@@ -213,6 +213,7 @@ private:
     QDBusObjectPath toObjectPathUnchecked();
     QDBusSignature toSignatureUnchecked();
     QStringList toStringListUnchecked();
+    QByteArray toByteArrayUnchecked();
 };
 
 inline QDBusMarshaller *QDBusArgumentPrivate::marshaller()
diff --git a/src/dbus/qdbusdemarshaller.cpp b/src/dbus/qdbusdemarshaller.cpp
index be19b00..215e6a1 100644
--- a/src/dbus/qdbusdemarshaller.cpp
+++ b/src/dbus/qdbusdemarshaller.cpp
@@ -272,7 +272,7 @@ QVariant QDBusDemarshaller::toVariantInternal()
         switch (q_dbus_message_iter_get_element_type(&iterator)) {
         case DBUS_TYPE_BYTE:
             // QByteArray
-            return toByteArray();
+            return toByteArrayUnchecked();
         case DBUS_TYPE_STRING:
             return toStringListUnchecked();
         case DBUS_TYPE_DICT_ENTRY:
@@ -339,7 +339,7 @@ QStringList QDBusDemarshaller::toStringList()
         return QStringList();
 }
 
-QByteArray QDBusDemarshaller::toByteArray()
+QByteArray QDBusDemarshaller::toByteArrayUnchecked()
 {
     DBusMessageIter sub;
     q_dbus_message_iter_recurse(&iterator, &sub);
@@ -350,6 +350,15 @@ QByteArray QDBusDemarshaller::toByteArray()
     return QByteArray(data,len);
 }
 
+QByteArray QDBusDemarshaller::toByteArray()
+{
+    if (q_dbus_message_iter_get_arg_type(&iterator) == DBUS_TYPE_ARRAY
+            && q_dbus_message_iter_get_element_type(&iterator) == DBUS_TYPE_BYTE) {
+        return toByteArrayUnchecked();
+    }
+    return QByteArray();
+}
+
 bool QDBusDemarshaller::atEnd()
 {
     // dbus_message_iter_has_next is broken if the list has one single element