Functional Programming in C++

AsynQt framework: Making QFuture useful

Published by Ivan Čukić in the KDE development section, on 17 January 2016

The idea of having a container for a value that is not yet present, but will be available sometime in the future is a powerful one. Unlike call-callbacks, it decouples the asynchronous process invocation from the result handler. This allows creating nicer asynchronous APIs than the usual approaches.

The Qt library provides a container like this – the QFuture<T>. And, at the same time, it provides incompatible special-cases containers like QDBusReply<T> and other Q*Reply classes. Initially, QFuture was meant to be used only in the QtConcurrent library, and was designed to suit that need – to wrap operations executed concurrently on multiple threads. It was moved to QtCore in 5.x, but apart from the rellocation, it did not receive much love, and its design stayed the same.

Issues with waiting

In order to do something with the result, you need to wait for the future to arrive. It you do not want to block your main thread, you need to wait for the future like this:

auto watcher = new QFutureWatcher<SomeType>();
               // remember to free later

connect(watcher, QFutureWatcherBase::finished,
        … slot to connect to …);
connect(watcher, QFutureWatcherBase::canceled,
        … slot to connect to …);

This makes using QFuture a pain. I’ve been creating wrappers for this on multiple occasions, and realised that I do it often enough to warrant creating a new library.

Issues with creation

Another issue with QFuture is that Qt does not really use it. The previously mentioned classes like QDBusReply are a perfect example. Imagine you are writing an interface library for some DBus service.

You can make it blocking

QString serviceVersion() const;

which is very bad since the service might take a long time to respond.

You can make it asynchronous by returning QDBusReply

QDBusReply<QString> serviceVersion() const;

which exposes the implementation detail via the API. Also a very bad approach.

Or, you can split it into a request slot and a response signal

void fetchServiceVersion() const;
Q_SIGNALS: void serviceVersionRetrieved(const QString &version);

which is plain ugly to write and use.

If the QFuture<QString> is used instead, you get exactly what you need – it is asynchronous, it hides all the implementation details (the user does not know whether it uses DBus, executes an external process and returns its output, etc.) and it is not ugly.

The problem is that Qt does not provide a way to allow you to convert these different future-like constructs to QFuture. This is the second reason behind the creation of the AsynQt framework.

What is AsynQt?

The aims of the framework are:

To provide wrappers for common Qt future-like classes
To add methods for easier manipulation of QFutures

Wrappers

So far, I’ve implemented wrappers for the DBus and QProcess.

You can convert the QDBusReply<T> to a QFuture<T> simply by using the makeFuture template function, or you can invoke a DBus method and get the QFuture as the result immediately:

QFuture<QStirng> future =
    AsynQt::DBus::asyncCall<QString>(interface, method, arg1, …)

QProcess wrapper is a bit trickier because it has one special feature. Apart from the process, it receives a function that allows you to extract the information you want to return from the process, instead of returning the QFuture<QProcess*>.

Examples are worth more than words:

QFuture<int> exitCodeFuture =
    makeFuture(process, [] (QProcess *p) {
        return p->exitCode();
    });

QFuture<QByteArray> outputFuture =
    makeFuture(process, [] (QProcess *p) {
        return p->readAllStandardOutput();
    });

There are also a few convenience methods for creating simpler futures – makeReadyFuture that creates a future which already contains a value, makeCanceledFuture that creates a canceled future and makeDelayedFuture which creates a future that will contain the specified value after the specified duration of time.

auto ready    = makeReadyFuture(10);
auto canceled = makeCanceledFuture<QString>();
auto delayed  = makeDelayedFuture(42, 30s); // with c++17 std::chrono

And more will come.

Transformations

Now, all of the above would not be that useful if we still had to use the usual QFuture API. For this reason, a few new methods are provided – transform, flatten, cast, continueWith. I’ll leave the explanation of these for later (when I add a couple more different ones that I consider necessary). At this time, just a few examples:

QFuture<int> answer = meaningOfLife()
    // answer will eventually contain 42

QFuture<QString> text = transform(answer, toText)
    // text will eventually contain the result of toText(42)

QFuture<QByteArray> future =
    AsynQt::Process::getOutput("echo", { "Hello KDE" });
    // will eventually contain QByteArray("Hello KDE\n")

QFuture<QString> castFuture =
    AsynQt::qfuture_cast<QString>(future);
    // will eventually contain QString("Hello KDE\n")

Pipe or range syntax

The library also supports the pipe syntax. Instead of calling a transformation on a future directly, you can also send the future through a pipe to the transformation.

QFuture<QString> future = meaningOfLife() | transform(toText)
    // text will eventually contain the result of toText(42)

QFuture<QByteArray> future =
    AsynQt::Process::getOutput("echo", { "Hello KDE" }) | cast<QString>();
    // will eventually contain QString("Hello KDE\n")

Status

It is under heavy development. The aforementioned things work, but the API might not be stable still. I’m still deciding whether some of the names are the best possible (cast<T> vs cast_to<T> vs convert_to<T> vs as<T>), etc.

It is implemented as mostly a header-only library, I’m still wondering whether I ought to remove the ‘mostly’ part.

It should be compilable by any compiler that supports decltype, static_assert, lambdas and similar. Nothing fancy, but I am not able to test whether it can be compiled on windows at this time.

The code is in my scratch space at git@git.kde.org:scratch/ivan/libasynqt

Christian Mollekopf
Good stuff!
Are you aware of the work Daniel Vratil and I did on KAsync? There seems to be quite a bit of overlap in terms of a more composable async API, although the design is quite different. We did a rather poor job in communicating how it works, but the akonadi next codebase [2] relies on it rather heavily.
I think we'll eventually want to go over the API of kasync anyways, and better QFuture support (instead of our own custom future that we have right now), would be a good idea anyways.

[1] http://api.kde.org/playground-api/libs-apidocs/kasync/html/namespaceKAsync.html,
kde:kasync
[2] kde:akonadi-next
Stefan
As you contemplate the API you wish to develop, you may find of interest the solution that the JavaScript community seems to have settled on. You can read the details at https://promisesaplus.com/. I've made use of it, and find it to be quite intuitive.
azadkuh
great. it's such a required framework for Qt.

in your sample:

QFuture<int> exitCodeFuture =
makeFuture(process, [] (QProcess *p) {
return p->exitCode();
});

1) what will happen if process (p) instance get deleted before the lambda starts to run?

2) probably the functor/lambda/... will run in different threads, is it safe to use QObject based instances (due to QObject thread affinity)?
Ivan Čukić
@Christian Mollekopf

I remember checking it out quite some time ago when Akonadi Next was barely born. And I completely forgot about it. I just remember not being thrilled by the design (though I did like that you have created a sane Future class - but the unfortunate thing was that I already used QFuture in KActivities API and changing that would be a serious violation :) ). I was also considering the Folly Future, but still, too late.

@Stefan

I've checked PA+. The problem is that Qt already has its API (and no .then, .onRejected, .onFulfilled). Maybe adding .then to QFuture might not be a bad idea...

There are more than a few libraries for manipulation of promises/futures, though mostly functional ones, that predate PA+. My current stance is to create something that uses the usual FP terminology, in the case where the standard C++ one does not exist (when I say standard C++ one, I'm including the c++17 plans, ranges and boost).

@azadkuh

1) You need to pay attention to object lifetime yourself. If you pass a QProcess that you have created to the library, you need to dispose of it (I do not like implicit transfer of ownership). If the library creates a process, and does not return it to you in a future, only in that case it will dispose of the object.

In this case, the connection will be broken, and I guess (haven't tested) you will get a future that will never finish. I truly don't consider this a valid use-case. :)

2) the function is run in the owner thread. There is no point in spinning another thread just to process a result of a process. Like with the ownership, I don't really like the idea of implicitly creating threads.
azadkuh
@ivan
may be i misunderstood your makeFuture(), but if its job is to create a future for an async call (like std::async() or QtConcurrent::run()), then passing QObject based class is problematic.

hope the coroutine/await proposal for c++17 come asap.
Ivan Čukić

@azadkuh

The process, the dbus call and such are already asynchonous operations. There is no need to put them in a separate thread to make them even more asynchronous :)

makeFuture is meant only to wrap the common asynchonous operations into a QFuture. QtConcurrent::run and std::async are different beasts - they get a piece of normal code, and run them on (potentially) a separate thread, thus creating something that is executed asynchronously.

Now, since QtConcurrent::run returns a QFuture, there is nothing to be done for it.

But, for std::async, if it turns out that it would really be useful to have std::future or boost::future wrapped in a QFuture, this library might provide a makeFuture that does that. Ideally, it should not do anything apart from wrapping - no need to create another thread nor anything - it just needs to be able to run a function on the result, when the result becomes ready. (the sad truth with std::future in C++11/14 is that the only way to wait without blocking would be to spin another thread, but that is an unfortunate situation, and not something that should be the common practice - when std::future gets the .then, this will be trivial).

Yes, the co_await seems like it will be a nice (albeit wrongly named) addition to the language.
mhogo mchungu

Speaking of futures,check out tasks[1],a Qt/C++ library that completes the "missing" parts in std::future and QFuture.

[1] https://github.com/mhogomchungu/tasks
Ivan Čukić

@mnogo mchungu

I've seen the library (on reddit). It is interesting, especially the future class design, but I have a few issues with it. I'll reply on github. :)

For this library, I needed to use QFutures (would not be my first choice) and not roll out my own, because I already have a QFuture-based library in which I want to use this code.
litb
Ah what a pity that QFutureInterface is not documented and considered "private". I made my own future library internally to have event-loop asynchronous futures, similar to your makeFuture. Had I known of QFutureInterface before, perhaps I would have taken the risk and abused it aswell :) At least now I can write a toQFuture() member function ^_^
Ivan Čukić
@litb Yes. At least, it has to stay binary compatible...