One of the problems that the Nepomuk team had in the past were
crashes in the strigi library (usually because of a corrupted file or
similar). For that reason, the indexer was moved into a separate
process, and it was executed for each file individually.
This is generally a good practice - ensuring that components can not
crash the main server. For those who don’t know this, Nepomuk server
invokes a few out-of-process services so that those can’t crash each
other. And, as already stated, the file indexer service delegates the
work to an external process. This is a rather nice design - hierarchical
separation of risky components into external processes.
While this brings stability, it is not cheap. Launching processes
takes valuable CPU time (loading, linking and other things your OS does
whenever you start a program). This is not a big issue for Nepomuk’s
services since there are only a few of them, and they are started only
once, when the system boots.
But it was the issue with the indexer since the executable was being
run for each file that needed indexing. This is not something that I was
comfortable with, so I decided to make it a bit more sophisticated,
without decreasing stability.
Now, the external indexer process is started only once, and the
server feeds it with a list of files that need indexing - one by one. If
the indexer crashes, the server just restarts it and continues without a
glitch.
Nepomuk is a very nice shared data repository. It is an easy way to
make the data from your application available to others.
But, it is important to know that Nepomuk is not a general purpose
database - everything is peachy until you start using it as such. And
especially if you start treating it as a relational database.
There are a few things to keep in mind when developing Nepomuk-based
programs.
Working on graphs
RDF
So, any query you make is not a restriction on a relation (a table)
but rather a multi-join of a single subject-verb-object table (this is a
bit simplistic view).
As you probably know, doing joins is not really a cheap operation
however optimized it is (and Virtuoso is one of the fastest
graph databases available).
D-Bus connections
The second thing is that while Nepomuk-internal connections are done
via local sockets, your connection to Nepomuk goes through D-Bus which
is not the fastest kid on the block. The more requests you make, the
more time it will take.
Some hints
There are some things you can make to make these issues less relevant
to your applications.
Wide-table queries
One of the common ways people write queries is the following:
select ?r where {
?r a something .
?r something else .
...
}
And then process the results one by one by doing stuff like:
Which means your program creates a lot of queries - one main, and a
couple more for each result.
Instead of making a lot of queries, it is advisable (although
initially not that intuitive) to create one big query like this:
select ?r, ?prop1, ?prop2 where {
?r a ?prop1 .
?r something ?prop2 .
...
}
It does transfer a lot of data at once, but at least it does so in a
single request-response connection, and it doesn’t repeat the same query
(parsing, optimizing, evaluating) multiple times for different
parameters.
Consider storing some data
locally
If you have data that don’t necessarily need to be shared, consider
storing it in config files, embedded database like sqlite3 or
similar.
This way, apart from skipping D-Bus, you can have faster queries in
the cases like these (which are not at all rare):
select ?r where {
?r a someType .
?r property1 "value1" .
?r property2 "value2" .
...
}
This query does a number of joins, whereas its equivalent in SQL does
only filtering:
select * from someType where
property1 = "value1" and
property2 = "value2" and
...
Summary
So, whatever you do (be it Nepomuk or something else), first ask,
experiment, test and learn the system before using it.
Just a short note for everyone using anything from the kext ontology
that is related to activities - since today, kext:Activity and friends
don’t exist. Everything is moved from kext to kao (KActivities
Ontology)
There are a lot of news regarding the activity manager, but nothing
picture-worthy so I’ll skip writing about it.
There is something that a few people here and there have been
requesting - having some automatic (UI) way to create encrypted folders
to keep their sensitive data in.
The thing I’m going to talk about today is exactly that - starting
with KDE SC 4.9 you’ll be able to decide to encrypt specific activities.
When you do that, you’ll get a ~/Activities/Something folder that is
password protected and encrypted using fuse/encfs.
The encryption/decription process will be done automatically on
activity switching.
For example, lets say you have two activities - Leisure and MI5 -
with the latter being an encrypted activity. When you switch to the MI5
activity, you’ll be asked for its password and you’ll be able to access
the data. When you switch back to the Leisure activity, the system for
the previous one will be automatically unmounted.
Plasma Active Three
One of the reasons behind this new feature is PA3. You’ll have a
portable device that can be stolen, that could be used by your children
(while being single-user) for fun etc. and you don’t want some data to
be visible to them.
In the case of PA, since there is no file manager and we don’t want
to expose the file-system to the user, every document that you link to
the activity will be automatically moved to the encrypted folder.
Drawbacks
There are a couple features that will stop working with encrypted
activities - you will not be able to search encrypted documents by
contents since the contents will not be indexed by nepomuk, and
documents will not be able to belong to multiple activities if one of
them is encrypted.
