#+TITLE: Paced #+AUTHOR: Ian Dunn #+EMAIL: dunni@gnu.org #+DATE: {{{modification-time}}} #+STARTUP: overview #+STARTUP: indent #+TODO: FIXME | FIXED #+OPTIONS: toc:2 num:nil timestamp:nil \n:nil |:t ':t email:t #+OPTIONS: *:t <:t d:nil todo:nil pri:nil tags:not-in-toc #+TEXINFO_DIR_CATEGORY: Emacs #+TEXINFO_DIR_TITLE: Paced: (paced) #+TEXINFO_DIR_DESC: Predictive Abbreviation Completion and Expansion using Dictionaries * Copying Copyright (C) 2017-2018 Free Software Foundation, Inc. #+BEGIN_QUOTE This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . #+END_QUOTE * Introduction :PROPERTIES: :DESCRIPTION: Brief Introduction to paced :END: Paced (Predictive Abbreviation Completion and Expansion using Dictionaries) scans a group of files (determined by "population commands") to construct a usage table (dictionary). Words (or symbols) are sorted by their usage, and may be later presented to the user for completion. A dictionary can then be saved to a file, to be loaded later. Population commands determine how a dictionary should be filled with words or symbols. A dictionary may have multiple population commands, and population may be performed asynchronously. Once population is finished, the contents are sorted, with more commonly used words at the front. Dictionaries may be edited through EIEIO's customize-object interface. Completion is done through ~completion-at-point~. The dictionary to use for completion can be customized. ** Similar Packages :PROPERTIES: :DESCRIPTION: Packages with similar goals :END: There are a few Emacs packages that have similar goals to paced, and provided some of the inspiration and motivation behind it. *** pabbrev The [[https://github.com/phillord/pabbrev][pabbrev]] package by Phillip Lord automatically scans text of the current buffer while Emacs is idle and presents the user with the most common completions. One of the major downsides to pabbrev is that the data it collects doesn't persist between Emacs sessions. For a few files that are always open, such as org agenda files, pabbrev works great. If you want to train it from a few files that aren't always open, you'll have to open each file and retrain pabbrev from that file. And you'll have to do this every time you restart Emacs. It keeps up-to-date usage and prefix hashes of all buffers of the same mode, and scanning, or "scavenging", blends seamlessly into the background. Completion is just a hash table lookup, so it can handle completion in microseconds. There's also no setup required; it will start working right away. The downside to this is that dictionaries aren't flexible; each dictionary corresponds to a major mode, and there's no way to change that. *** predictive The [[https://www.dr-qubit.org/predictive.html][predictive]] package by Toby Cubitt scans text of the current buffer on user command. The usage data is stored in a dictionary, which can then be saved to a disk. Extensions are provided to ~completion-at-point~, or predictive's built-in frontend can be used. It has a safety precaution where it only adds existing words to a dictionary, unless the user allows this. This is to avoid adding typos to a dictionary. Completion was also done intelligently, grouping commonly used words together and optionally suggesting shorter words before longer words. While the frontend and backend are separate, the frontend is required to populate a dictionary. There is no way to exclude part of the buffer's text from dictionary population. The safety precaution predictive has where it only adds a word to a dictionary if it already exists was tedious, since I didn't need it to do that. ** Installation :PROPERTIES: :DESCRIPTION: How to install paced :END: *Requirements* | Emacs | 25.1 | | async | 1.9.1 | Paced may be installed from source, or from GNU ELPA. From ELPA: #+begin_example M-x package-install RET paced RET #+end_example From Source: #+begin_src shell bzr branch https://bzr.savannah.gnu.org/r/paced-el paced #+end_src After installing from source, add the following to your init file (typically .emacs): #+BEGIN_SRC emacs-lisp (add-to-list 'load-path "/full/path/to/paced/") (require 'paced) #+END_SRC However you install paced, you must also make sure dictionaries are loaded on startup: #+begin_src emacs-lisp (paced-load-all-dictionaries) #+end_src ** Basic Setup :PROPERTIES: :DESCRIPTION: The simplest setup :END: Paced needn't have a lot of setup to run. In fact, the simplest setup is as follows: 1. Create a new dictionary, "Default" (See [[#dictionary_creation][Creating a Dictionary]]) 2. Set ~paced-global-dictionary-enable-alist~ to ~((t . "Default"))~ (See [[#selective_dictionaries][Selective Dictionaries]]) 3. Run ~M-x global-paced-mode~ 4. To add a file to the dictionary, use ~M-x paced-add-buffer-file-to-dictionary~ This will create a default dictionary and populate it from buffers you specify. * Dictionaries :PROPERTIES: :DESCRIPTION: Paced's bread and butter :END: ** Creating a Dictionary :PROPERTIES: :DESCRIPTION: First steps :CUSTOM_ID: dictionary_creation :END: Now that you've got paced installed, it's time to create a new dictionary. #+begin_example M-x paced-create-new-dictionary RET DICTIONARY_NAME RET DICTIONARY_FILE RET #+end_example Let's explain those two arguments: First, you've got DICTIONARY_NAME. This is a string that will be used to reference the new dictionary. We recommend something short, like 'new-dict', 'my-dict', 'writing', etc. Next is the file where the dictionary will be stored. This is typically stored in ~paced-dictionary-directory~, from which all dictionaries will be loaded with ~paced-load-all-dictionaries~ (more on that later). For now, it's important to know that ~paced-load-all-dictionaries~ is the easiest way to load dictionaries when paced is loaded. After you've run the above command, you will be taken to the customization buffer. This is where you can set population commands. ** Editing a Dictionary :PROPERTIES: :DESCRIPTION: How to edit your new dictionary :END: In order to edit a dictionary, paced provides ~paced-edit-named-dictionary~ and ~paced-edit-current-dictionary~. The edit buffer provides the options to change the population commands, case handling, dictionary storage name, and sort method. Each of these is documented in the edit buffer. ** Selective Dictionaries :PROPERTIES: :DESCRIPTION: Enabling certain dictionaries under certain conditions :CUSTOM_ID: selective_dictionaries :END: Paced provides a mechanism called the "enable list", that allows a user to enable certain dictionaries for completion given certain conditions. There are two enable lists: a global (~paced-global-dictionary-enable-alist~) and local (~paced-local-dictionary-enable-alist~) one. They both work the same, with the local one taking precedence. Each entry in the list has a condition and a key. The conditions are one of the following: - A mode name, such as ~org-mode~ or ~text-mode~, indicating that the named dictionary should be active in any mode derived from that mode. - A symbol, in which case the named dictionary is active whenever the value of that symbol is non-nil. This includes the symbol ~t~. - A function symbol, in which case the function is called with no arguments to determine if the given dictionary should be enabled. If the function returns non-nil the dictionary is enabled. - A lambda function, in which case it is called with no arguments, and if it returns non-nil, the dictionary is enabled. - The form (or CONDITION1 CONDITION2 ...), which enables the given dictionary if any of the conditions are met. - The form (and CONDITION1 CONDITION2 ...), which enables the given dictionary if all of the conditions are met. Remember that paced-mode must be active for completion to occur. Neither list will activate it, just determine which dictionary is active. The key is the dictionary name you set during dictionary creation. ** Dictionary Files :PROPERTIES: :DESCRIPTION: Loading and Saving the Dictionaries :END: Paced provides ~paced-load-all-dictionaries~ to load all dictionaries in ~paced-dictionary-directory~. Paced determines which dictionaries to load based on two variables: ~paced-dictionary-directory-whitelist-regexp~ and ~paced-dictionary-directory-blacklist-regexp~. Paced can also be told to search recursively by setting ~paced-load-all-dictionaries-recursively~ to t. All four of these variables may be set using Emacs's customization interface. An individual dictionary file may also be loaded: #+begin_example M-x paced-load-dictionary-from-file RET /path/to/file RET #+end_example Once a file has been modified, it may then be saved: #+begin_example M-x paced-save-named-dictionary RET dictionary name RET #+end_example Or, all dictionaries may be saved: #+begin_example M-x paced-save-all-dictionaries RET #+end_example Dictionaries may also be automatically saved whenever changed by setting ~paced-repopulate-saves-dictionary~ to t. Population is covered in the next section. ** Printing a Dictionary :PROPERTIES: :DESCRIPTION: Seeing the contents of a dictionary :END: Paced allows a user to print the contents of a dictionary to a buffer. Uses for this might be to tweak population commands or exclude functions, or to simply make sure a dictionary is populating correctly. To use this feature, run: #+begin_example M-x paced-print-named-dictionary RET NAME-OF-DICTIONARY RET #+end_example Or for the current dictionary: #+begin_example M-x paced-print-current-dictionary RET #+end_example * Population Commands :PROPERTIES: :DESCRIPTION: The good stuff :END: Part of the beauty of paced is the ease of reconstructing a dictionary. When you've got a bunch of files from which you want to populate your dictionary, it'd be a pain to go to each of them and say "populate from this one, next, populate from this one, next". Instead, paced provides population commands. Each dictionary has one or more population commands it uses to recreate its contents, run in order during population. In order to trigger population, run the following: #+begin_example M-x paced-repopulate-named-dictionary RET DICTIONARY-NAME RET #+end_example ** Built-in Commands :PROPERTIES: :DESCRIPTION: Basics :END: There are five built-in population commands: - file :: Populates a dictionary from all words in a given file - buffer :: Populates a dictionary from all words in a given buffer, which must exist during population - file-function :: Like the file command, but allows a custom setup function. This function is called with no arguments in a temporary buffer containing the file's contents, and must return non-nil if population may continue. - directory-regexp :: Populates from all files in a directory that match the given regexp. Also optionally allows recursion. - file-list :: Populates from all files returned by a generator function. ** Properties :PROPERTIES: :DESCRIPTION: Tweaking the defaults :END: When setting the population commands of a dictionary, one may also set certain properties. Each property is a variable binding, bound while the population command runs. A few variables are of note here: - paced-exclude-function :: Function of no arguments that returns non-nil if the thing at point should be excluded from population. - paced-thing-at-point-constituent :: Symbol defining thing on which population works. Typically set to either 'symbol or 'word. - paced-character-limit :: Maximum length of a thing to include it in a dictionary. If set to 0 (default), no limit is imposed. For convenience, properties that are intended for all population commands of a given dictionary may be set in the dictionary itself. In the event of a conflict, population command properties take precedence over dictionary properties. ** Custom Commands :PROPERTIES: :DESCRIPTION: Defining new population commands :END: Since the population commands all derive from paced-population-command, it's possible to add additional commands. As an example, let's make a population command that populates a dictionary from a file like so: #+begin_example alpha 5 beta 7 gamma 21 delta 54 epsilon 2 #+end_example We want to make a population command that takes a file like this, with word in one column and weight in the other, and add it to a dictionary. There are two ways to approach this, but we're going to start with the basic one. We need to define two functions: paced-population-command-source-list and paced-population-command-setup-buffer. The first returns a list of sources from which to populate, and the second sets up a temporary buffer based on those sources. For our command, we want to return the specified file, and replicate each word by the amount given. Inheriting from ~paced-file-population-command~ gives us the source list and file slot for free. #+begin_src emacs-lisp (defclass paced-weight-file-population-command (paced-file-population-command)) #+end_src Now, we need to set up the buffer to replicate the words. #+begin_src emacs-lisp (cl-defmethod paced-population-command-setup-buffer ((cmd paced-weight-file-population-command) source) ;; Use the built-in `paced--insert-file-contents' to insert contents. (paced--insert-file-contents source) ;; Jump to the start of the buffer (goto-char (point-min)) ;; Search for lines with the form WORD WEIGHT (while (re-search-forward (rx line-start ;; Start of line (submatch (one-or-more (not (syntax whitespace)))) ;; Our word (syntax whitespace) ;; Space between word and weight (submatch (one-or-more (any digit))) ;; Weight line-end) ;; End of line nil t) (let* ((word (match-string 1)) (weight (string-to-number (match-string 2))) ;; Repeat WORD WEIGHT times (new-text (string-join (make-list weight word) " "))) ;; Replace the matched text with our repeated word (replace-match new-text)))) #+end_src That's all there is to it. When you go to edit a dictionary, the "weight-file" population command will automatically be added as an option for a population command. The even easier way to do this would've been to use ~paced-file-function-population-command~, but it doesn't make for a good example in this case. ** Asynchronous Population :PROPERTIES: :DESCRIPTION: Populating without blocking :END: A common problem is that population can take a long time. Some of us populate dictionaries from org agenda files, which can get pretty big. To solve this, paced uses the [[https://github.com/jwiegley/emacs-async][async]] package. Setup is seamless; just stick whatever code you need in ~~/.emacs.d/paced-async.el~, and use one of the two population commands: A named dictionary: #+begin_example M-x paced-repopulate-named-dictionary-async RET NAME RET #+end_example Or the current dictionary: #+begin_example M-x paced-repopulate-current-dictionary-async RET #+end_example A few things to note about this: 1. Dictionaries will be automatically saved by this method after population 2. Asynchronous population doesn't change anything until after population is finished, so a user may continue to use their dictionary while population is happening. This also means that multiple populations may run in parallel without interfering with one another. 3. Because async runs population in a separate Emacs process, any custom code required for population must be in paced-async.el. This includes additional population command types, but doesn't include the following variables: - load-path - paced-thing-at-point-constituent - paced-async-load-file * Example Setups :PROPERTIES: :DESCRIPTION: Some examples :END: ** Org Agenda Files As some of us record everything about our lives in our agenda files, it might be helpful to have a dictionary tuned to ourselves. We use a file-list command that returns the agenda files, and an exclude command to block out all of Org's extra features such as source code and drawers. The generator for file-list is easy: #+begin_src emacs-lisp org-agenda-files #+end_src Done. Now, the exclude command, which sits inside the properties option. This can be added to ~paced-async.el~: #+begin_src emacs-lisp (require 'org) (defun org-at-tag-p () (let* ((p (point))) ;; Ignore errors from `org-get-tags-string'. (ignore-errors ;; Checks the match string for a tag heading, setting match-string 1 to the ;; tags. Also sets match-beginning and match-end. (org-get-tags-string) (when (match-string 1) (<= (match-beginning 1) p (match-end 1)))))) (defun org-at-keyword-p () "Return non-nil if point is at a keyword such as #+TITLE." (save-excursion (beginning-of-line) (looking-at-p "^#\\+"))) (defun org-at-heading-prefix-p () "Return non-nil if looking at the leading stars of a heading." (looking-at outline-regexp)) (defun org-at-hline-p () (save-excursion (beginning-of-line) (looking-at-p "^-----"))) (defun org-paced-exclude () (or ;; Drawers (org-between-regexps-p org-drawer-regexp ":END:") ;; Doesn't catch END (org-in-regexp ":END:") ;; but this does (org-at-tag-p) ;; tags (org-at-keyword-p) ;; Keywords, such as #+TITLE (org-at-heading-prefix-p) ;; Leading stars of a heading (org-at-item-bullet-p) ;; Item Bullets (org-at-timestamp-p) ;; Timestamps (looking-at-p org-todo-regexp) ;; TODO keywords (org-at-hline-p) ;; H-lines (org-at-comment-p) ;; comments (org-in-regexp org-any-link-re) ;; links (org-in-block-p '("src" "quote" "verse")) ;; blocks (org-at-planning-p) ;; deadline, etc. (org-at-table-p) ;; tables )) #+end_src As explained earlier, this can be put inside properties in the customize buffer as such: #+begin_example Properties : [INS] [DEL] Variable: paced-exclude-function Lisp expression: 'org-paced-exclude #+end_example And you're done. See how easy that was? ** Project Files Now we get to the interesting one. There are tons of ways to collect project files in Emacs, so we're going to stick with one for now, being Emacs's built-in VC package. #+begin_src emacs-lisp (defun vc-paced-find-project-files (path-to-project-root) "Use VC to collect all version-controlled files." (let ((file-list)) (vc-file-tree-walk path-to-project-root (lambda (f) (push f file-list))) file-list)) #+end_src We'd then need to use the following for our file-list generator: #+begin_example Generator : (lambda nil (vc-paced-find-project-files "/home/me/programming/paced")) #+end_example Now, we (probably) don't want commented code to get in our way, so we'll use a small function for excluding those: #+begin_src emacs-lisp (defun paced-at-comment-p () (nth 8 (syntax-ppss))) #+end_src Use that for paced-exclude-function, and you're done. We can't necessarily recommend this for any programming language, as there are dedicated solutions for almost everything, but it makes an excellent fallback. ** Markdown Files Another common request is markdown files. In order for this to work, you'll need to install ~markdown-mode~: #+begin_example M-x package-install RET markdown-mode RET #+end_example After that, add the following to your ~paced-async.el~ file: #+begin_src emacs-lisp (require 'markdown-mode) (defun paced-markdown-exclude-p () "Taken from `markdown-flyspell-check-word-p'." ;; Exclude anything markdown mode thinks flyspell should skip. (or ;; Ignore code blocks (markdown-code-block-at-point-p) (markdown-inline-code-at-point-p) ;; Ignore comments (markdown-in-comment-p) ;; Ignore special text (let ((faces (get-text-property (point) 'face))) (if (listp faces) (or (memq 'markdown-reference-face faces) (memq 'markdown-markup-face faces) (memq 'markdown-plain-url-face faces) (memq 'markdown-inline-code-face faces) (memq 'markdown-url-face faces)) (memq faces '(markdown-reference-face markdown-markup-face markdown-plain-url-face markdown-inline-code-face markdown-url-face)))))) #+end_src That excludes anything that the developers of markdown-mode felt should be excluded from flyspell. Set this as your exclude function in your dictionary's settings, then add each markdown file by hand. ** Repopulating Dictionary After Saving This is a common request, although with the power of async, it's an easy one to fulfill. This will repopulate the current buffer's dictionary every time you save a file with a dictionary. This may seem daunting, but the dictionary will remain usable during population, and multiple populations won't interfere with one another. #+begin_src emacs-lisp ;; Repopulate the current dictionary after saving (add-hook 'after-save-hook 'paced-repopulate-current-dictionary-async) #+end_src Add that to your .emacs file, and paced will take it from there. If you decide that's too much, do the following: #+begin_example M-: (remove-hook 'after-save-hook 'paced-repopulate-current-dictionary-async) RET #+end_example ** Repopulating Dictionary After Spellchecking the Buffer Another request, although much trickier to do. This one involves using Emacs's advice mechanism: #+begin_src emacs-lisp (define-advice ispell-pdict-save (:after (&optional _no-query _force-save) paced-populate) ;; Repopulate the current dictionary after running spell check (paced-repopulate-current-dictionary-async)) #+end_src If you decide this isn't for you, do the following to revert the changes: #+begin_example M-: (advice-remove #'ispell-pdict-save #'ispell-pdict-save@paced-populate) RET #+end_example * Contributing :PROPERTIES: :DESCRIPTION: I wanna help! :CUSTOM_ID: contributing :END: We are all happy for any help you may provide. First, check out the source code on Savannah: https://savannah.nongnu.org/projects/paced-el #+BEGIN_SRC shell bzr branch https://bzr.savannah.gnu.org/r/paced-el/ paced #+END_SRC Build the Makefile with EDE: 1. Open any file from paced (See [[#ede][Working with EDE]] if you encounter "Corrupt object on disk" error) 2. Run ~C-c . C~ or ~M-x ede-compile-project~ ** Bugs :PROPERTIES: :CUSTOM_ID: bugs :DESCRIPTION: Submitting bug reports :END: There are two ways to submit bug reports: 1. Using the bug tracker at Savannah 2. Sending an email using ~paced-submit-bug-report~ When submitting a bug report, be sure to include a description of the dictionary or population command that caused the problem, with as much detail as possible. ** Development :PROPERTIES: :CUSTOM_ID: development :DESCRIPTION: Helping with development :END: If you're new to bazaar, we recommend using Emacs's built-in VC package. It eases the overhead of dealing with a brand new VCS with a few standard commands. For more information, see the info page on it (In Emacs, this is C-h r m Introduction to VC RET). To contribute with bazaar, you can do the following: #+begin_src shell # Hack away and make your changes $ bzr commit -m "Changes I've made" $ bzr send -o file-name.txt #+end_src Then, use ~paced-submit-bug-report~ and attach "file-name.txt". We can then merge that into the main development branch. There are a few rules to follow: - New population commands should be named paced-POPULATION-COMMAND-TYPE-population-command - Run 'make check' to verify that your mods don't break anything - Avoid additional or altered dependencies if at all possible - Dictionary commands come in threes ("the operation triad"): 1. paced-dictionary-OPERATION, a cl-defmethod which performs OPERATION on a dictionary 2. paced-OPERATION-on-named-dictionary, an interactive only function that prompts for a dictionary name and performs OPERATION on that dictionary: #+begin_src emacs-lisp (interactive (list (paced-read-dictionary))) (paced-ensure-registered name) (paced-dictionary-OPERATION (paced-named-dictionary name)) #+end_src 3. paced-OPERATION-on-current-dictionary, an interactive function that performs OPERATION on the current dictionary #+begin_src emacs-lisp (interactive) (paced-dictionary-OPERATION (paced-current-dictionary-or-die)) #+end_src ** Documentation :PROPERTIES: :CUSTOM_ID: docs :DESCRIPTION: Improving the documentation :END: Documentation is always helpful to us. Please be sure to do the following after making any changes: 1. Update the info page in the repository with ~C-c C-e i i~ 2. If you're updating the HTML documentation, switch to a theme that can easily be read on a white background; we recommend the "adwaita" theme ** Working with EDE :PROPERTIES: :CUSTOM_ID: ede :DESCRIPTION: And all its quirks :END: EDE can be a little finicky at times, but we feel the benefits, namely package dependency handling and Makefile generation, outweigh the costs. One of the issues that many will likely encounter is the error "Corrupt file on disk". This is most often due to EDE not loading all its subprojects as needed. If you find yourself dealing with this error often, place the following in your .emacs file: #+begin_src emacs-lisp ;; Target types needed for working with paced (require 'ede/proj-elisp) (require 'ede/proj-aux) (require 'ede/proj-misc) #+end_src These are the three target types that paced uses: elisp for compilation and autoloads; aux for auxiliary files such as documentation; and misc for tests. When creating a new file, EDE will ask if you want to add it to a target. Consult with one of the paced devs for guidance, but usually selecting "none" and letting one of us handle it is a good way to go. * Changelog :PROPERTIES: :DESCRIPTION: List of changes by version :END: ** 1.1.3 - Fixed bug with printing an empty dictionary - Fixed bug with paced crashing on non-existent thing at point ** 1.1.2 - Fixed bug with printing dictionaries ** 1.1.1 - Fixed bug with asynchronous population throwing an error on no dictionary - Set ~paced-throw-error-on-no-current~ to nil by default ** 1.1 - Cleaned up the code to reflect the "operation triad" - -OP, OP-on-named, OP-on-current - Retained backwards compatibility by obsoleting a bunch of functions, but didn't remove any of them - Also removed the use of dict- in global variables and functions - Added the ability to print the contents of a dictionary in a separate buffer - Added the option to limit the words added during population by size - Various documentation improvements ** 1.0.1 Bug fix release - Save dictionaries right after they're created - Added "force" parameter to save functions ** 1.0 Initial release.