| Age | Commit message (Collapse) | Author |
|---|
|
|
|
This change adds a new cleanup mode that avoids cleanup having
re-traverse the directories the index pass just looked at.
Additionally, we efficiently query the Xapian database by walking the
term list instead of doing multiple point-wise path lookups.
I'd noticed that most of my time in mu's cleanup pass consisted of
B-tree lookups in Xapian (one 8KB pread64 at a time). The point
lookups forced Xapian to traverse from the root of the B-tree to the
leaf for every single message. Additionally, in order to join on the
message path, we had to do *another* B-tree traversal after locating
each message term. Now we just walk the terms in order, which is much
more efficient, as we touch each B-tree node only once.
On my system, with 1371861 total messages, the total time of mu
index (no lazy check):
--nocleanup: 3.6s
incremental cleanup: 4.2s (0.6s in cleanup)
legacy cleanup: 5.2s (1.6s in cleanup)
With the new mode, we save 1.0s of the 1.6s cleanup, so we're
~63% faster.
But the incremental cleanup works even better with lazy checking.
If I enable --lazy-check, dirty only my INBOX (360778 messages), and
run index, I get:
--nocleanup: 0.9s
incremental cleanup: 1.1s (0.2s in cleanup)
legacy cleanup: 2.5s (1.6s in cleanup)
We save 1.4s out of 1.6s for ~88% speedup.
This change also fixes a timestamp bug: we should be storing
the *start* time of the index pass in metadata, not the end time, so
that on the next index pass, we notice messages that arrived between
the two times.
All tests pass. You can set the environment variable
MU_NO_INCREMENTAL_CLEANUP to use the legacy cleanup path instead.
|
|
They don't add much value, let's remove.
Rework clear_labels in terms of update_labels.
|
|
Split labels-cache and store-labels.
|
|
E.g. with unexpected termination for mu it is possible that the
labels-cache (i.e., the one that is used for auto-completion) gets disconnected
from reality.
Add a --restore option to `mu label list` to restore the actual labels from the
labels seen in the store.
|
|
For triggering saving the contacts / labels caches.
|
|
In particular, clear_labels
And some cosmetics
|
|
For handling mu4e label update requests.
|
|
Move the import/export code to 'lib'.
|
|
Add methods update_labels, clear_labels which update or clear the labels for a
message in the store, and update the cache with the overall counts of labels.
Add a LabelsCache to keep track of the counts and labels_map() to retrieve that
map.
|
|
The content-flags won't change, and the unread-flag can be re-calculated.
Add a unit test, and some small doc improvements.
Fixes #2831.
|
|
We're not using it.
|
|
store-worker is a thread + async queue to throttle requests to a single thread.
|
|
Add store::consume_message, which is like add message but std::move from
the caller such that the messages longer has copies (with
Xapian::Document) on the caller side; this is to avoid threading issues.
|
|
Only include xapian.h in one place, so we can have consistent options.
With that in place, we can enable C++ move semantics.
We don't do anything with that yet, but we check in the meson.build file
to see if we have the required xapian version.
|
|
|
|
Instead of handling transactions in the store, handle it in xapian-db.
Make the code a bit more natural / cleaner-out
Handle transaction automatically (with a batch-size) and add some RAII
Transaction object, which makes all database interaction transactable
for the duration. So, no more need for explicit parameters to
add_message while indexing.
|
|
sort results (for testing)
with the ChangeName flag, also apply to dup messages.
|
|
Add sub-command to move messages; add tests and docs.
Fixes #157
|
|
|
|
simplify things a bit
|
|
A little fmt pixie dust
|
|
Xapian supports an "ngrams" option to help with languages/scripts
without explicit wordbreaks, such as Chinese / Japanese / Korean.
Add some plumbing for supporting this in mu as well. Experimental for
now.
|
|
This gets the current list of maildirs by asking the scanner to do a
file-system search.
|
|
Update test cases as well.
|
|
*Usually* we need Xapian's replace_document() API, but when we know a
document (message) is completely new, we can use the faster
add_document(). That is the case with the initial (re)indexing, when
start with an empty database.
Also a few smaller cleanups.
|
|
Use xapian_try_result
|
|
No need for two near-identical impls
Remove some dead declarations.
|
|
Simplifies the implementation.
|
|
Remove readonly/inmemory from properties. Add read_only() accessor.
|
|
Update the move_message API so to allow for updating duplicate messages
too (not implemented yet), and return all updated messages.
|
|
Support reinitializing, based on some current store. This is useful for
upgrading. Note that this is only the backend implementation + tests.
|
|
|
|
Add a bunch of tests
|
|
This cause havoc / race conditions.
|
|
Keep track of the latest-change/latest-index.
|
|
Make it a Result type, and add auto-upgrade (not enabled yet)
Update dependents.
|
|
One should be enough!
|
|
And turn all "add" into "replace" so old messages get removed.
Update tests too.
|
|
Keep it in the store; much faster than generating on the fly.
|
|
Remove much of the message processing from the store
|
|
|
|
We need the extensions, and/or let's use _one_ optional implementation everywhere.
|
|
|
|
Adapt to the new names / directory. Big commit, but mostly just very boring renaming.
|
|
We were trying to convert a field (string) to a xapian prefix back to a
field (enum). That's unnecessarily complicated and worse, step 2 won't work.
|
|
|
|
Plus dependents.
|
|
The QueryResults must not outlive the lock (when in a MT context), so expose for
clients (mu-server) to handle it.
|
|
Allow for storing metadata in the database, and consider the cache.
|
