文件列表:

.dir-locals.el (155, 2023-06-05)
ACM-Reference-Format.bst (90333, 2023-06-05)
acmart.cls (117087, 2023-06-05)
bb.edn (878, 2023-06-05)
bibliography.bib (14062, 2023-06-05)
deps.edn (698, 2023-06-05)
dev/ (0, 2023-06-05)
dev/latex.clj (17694, 2023-06-05)
dev/user.clj (148, 2023-06-05)
template-sigconf.tex (33399, 2023-06-05)

# Clerk: Moldable Live Programming for Clojure ```clojure (ns nextjournal.clerk.px23 {:nextjournal.clerk/toc true :nextjournal.clerk/open-graph {:url "https://px23.clerk.vision" :title "Clerk: Moldable Live Programming for Clojure" :description "Clerk is an open source Clojure programmer’s assistant that builds upon the traditions of interactive and literate programming to provide a holistic moldable development environment. Clerk layers static analysis, incremental computation, and rich browser-based graphical presentations on top of a Clojure programmer's familiar toolkit to enhance their workflow."} :nextjournal.clerk/visibility {:code :hide}} (:require [nextjournal.clerk :as clerk] [applied-science.edn-datasets :as datasets])) ``` ```clojure ^{::clerk/visibility {:result :hide}} (defn figure [{:as opts ::clerk/keys [width]}] (clerk/with-viewer {:transform-fn (clerk/update-val (fn [{:keys [id src caption video?]}] (clerk/html [:div.not-prose.overflow-hidden {:id id :class (when-not (= :full width) "rounded-lg")} (if video? [:video {:loop true :controls true} [:source {:src src}]] [:img {:src src}]) (when caption [:div.bg-slate-100.dark:bg-slate-800.dark:text-white.text-xs.font-sans.py-4 [:div.mx-auto.max-w-prose.px-8 [:strong.mr-1 "Figure:"] caption]])])))} opts opts)) ``` ## Abstract ```clojure (clerk/html [:div.flex.flex-col.not-prose {:class "min-[860px]:flex-row"} [:div [:p.italic.leading.leading-snug "Clerk is an open source Clojure programmer’s assistant that builds upon the traditions of interactive and literate programming to provide a holistic moldable development environment. Clerk layers static analysis, incremental computation, and rich browser-based graphical presentations on top of a Clojure programmer's familiar toolkit to enhance their workflow."]] [:div.font-sans.flex-shrink-0.mt-6.text-sm {:class "min-[860px]:w-[165px] min-[860px]:ml-[40px] min-[860px]:text-xs min-[860px]:mt-1"} [:a.hover:opacity-70 {:href "https://nextjournal.com"} [:img.block.dark:hidden {:src "https://nextjournal.com/images/nextjournal-logo.svg" :width 100 :class "min-[860px]:-ml-[8px]"}] [:img.hidden.dark:block {:src "https://nextjournal.com/images/nextjournal-logo-white.svg" :width 100 :class "min-[860px]:-ml-[8px]"}]] [:a.block.mt-2.hover:opacity-70 {:href "https://twitter.com/mkvlr"} "Martin Kavalar"] [:a.block.mt-1.hover:opacity-70 {:href "https://twitter.com/unkai"} "Philippa Markovics"] [:a.block.mt-1.hover:opacity-70 {:href "https://twitter.com/jackrusher"} "Jack Rusher"] [:a.block.mt-2.hover:opacity-70 {:href "https://2023.programming-conference.org/home/px-2023/"}"Presented at PX/23"]]]) ``` ## Introduction: Literate Programming, Notebooks and Interactive Development Knuth's _Literate Programming_[^literateprogramming][^knuth84] emphasized the importance of focusing on human beings as consumers of computer programs. His original implementation involved authoring files that combine source code and documentation, which were then divided into two derived artifacts: source code for the computer and a typeset document in natural language to explain the program. [^knuth84]: [Literate Programming](https://doi.org/10.1093/comjnl/27.2.97) [^literateprogramming]: An extensive archive of related material is maintained [here](http://www.literateprogramming.com). At the same time, other software was developed to target scientific use cases rather than program documentation. These systems, which prefigured modern computational notebooks, ranged from REPL-driven approaches like Macsyma and Mathematica to integrated WYSIWYG editors like Ron Avitzur's _Milo_, PARC's _Tioga_ and _Camino Real_, and commercial software like _MathCAD_.[^mathematical-software] [^mathematical-software]: See [A Survey of User Interfaces for Computer Algebra Systems](https://people.eecs.berkeley.edu/~fateman/temp/kajler-soiffer.pdf) for a history of these systems up until 1998 ([DOI](https://doi.org/10.1006/jsco.1997.0170)). In contemporary data science and software engineering practice, we often see interfaces that combine these two approaches, like [Jupyter](https://jupyter.org), [Observable](https://observablehq.com), [Pluto][pluto], and [Livebook][livebook]. In these notebooks, a user can mix prose, code, and visualizations in a single document that provides the advantages of Knuth's Literate Programming with those of a scientific computing environment. Unfortunately, most such systems require the programmer to use a browser-based editing environment (which alienates programmers with a strong investment in their own tooling) and custom file formats (which cause problems for integration with broader software engineering practices).[^notebook-pain-points] [^notebook-pain-points]: See [What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities](https://doi.org/10.1145/3313831.3376729) by Souti Chattopadhyay, Ishita Prasad, Austin Z. Henley, Anita Sarma and Titus Barik. Although notebooks of this kind present an improvement on the programming experience of many languages, they often feel like a step backward to experienced Lisp programmers. In Lisp environments, it is common to be able to place the cursor after a single Lisp form and evaluate it in the context of a running program, providing finer granularity of control compared to the per-cell model of most notebooks. This workflow leads to a development style that these programmers are in no hurry to lose. > That LISP users tend to prefer structured growth rather than stepwise refinement is not an effect of the programming system, since both methods are supported. I believe, however, that it is a natural consequence of the interactive development method, since programs in early stages of growth can be executed and programs in early stages of refinement cannot.[^sandewall] > > – Erik Sandewall [^sandewall]: See [Programming in an Interactive Environment: the "Lisp" Experience](https://doi.org/10.1145/356715.356719) by Erik Sandewall At the same time, though a number of Lisp environments have included graphical presentations of program objects[^mcclim], most modern tooling relies on text-based representations of evaluation output and doesn't include the ability to embed widgets for direct manipulation of program state. Additionally, problems often arise when printing structurally large results, which can cause editor performance to degrade or lead to the truncation of output, and there's limited room for customization or support for requesting more data. [^mcclim]: See, for example, the [Common Lisp Interface Manager](https://en.wikipedia.org/w/index.php?title=Common_Lisp_Interface_Manager&oldid=1121151005). In comparison, interactive programming in Smalltalk-based systems has included GUI elements since the beginning, and work to further improve programmer experience along these lines has continued in Smalltalk-based systems like [Self](https://selflanguage.org)[^Ungar87], [Pharo](https://pharo.org), [Glamorous Toolkit](https://gtoolkit.com)[^moldable-tools] and [Newspeak](https://newspeaklanguage.org) or Ampleforth[^ample-forth], which offer completely open and customizable integrated programming environments. Glamorous Toolkit, in particular, champions the idea of using easily constructed custom tools to improve productivity and reduce time spent on code archeology, which is also a big inspiration for what we'll present here. This paper contributes a description of the Clerk system, along with its background and the motivation for its construction. We include a number of examples of things built by users of the tool, and some discussion of the feedback that it has thus far received. [^moldable-tools]: [Towards Moldable Development Tools](https://doi.org/10.1145/2846680.2846684) by Andrei Chi, Oscar Nierstrasz and Tudor Grba [^ample-forth]: [Ampleforth: A Live Literate Editor](https://blog.bracha.org/Ampleforth-Live22/out/primordialsoup.html?snapshot=Live22Submission.vfuel) by Gilad Bracha [^Ungar87]: [Self: The power of simplicity](https://doi.org/10.1145/38807.38828) ## Programming with Clerk > In such a future working relationship between human problem-solver and computer ‘clerk’, the capability of the computer for executing mathematical processes would be used whenever it was needed.[^latex-cit:Engelbart_1962] > > – Douglas Engelbart [^latex-cit:Engelbart_1962]: See [Augmenting Human Intellect: A Conceptual Framework](https://www.dougengelbart.org/pubs/augment-3906.html) by Douglas Engelbart. We have built Clerk on top of Clojure[^clojure], a functional-by-default Lisp dialect primarily hosted on the [Java Virtual Machine](https://en.wikipedia.org/w/index.php?title=Java_virtual_machine&oldid=1144897244). Several aspects of the language make it an appealing target for this project: [^clojure]: For a description of the language and its motivations, see [A history of Clojure](https://doi.org/10.1145/3386321). * being a Lisp, there is limited syntax with which to contend, and the language comes with good libraries for meta-linguistic programming * an emphasis on pure functions and immutable data structures makes static analysis easier * when mutable state is needed, there are idiomatic thread-safe boxes that are read and updated in a functional style While there are some rough edges around a few particularly tricky language features, these aspects have mostly worked out in our favor. ### Basic Interaction: Bring-Your-Own-Editor Clerk combines Lisp-style interactive programming with the benefits of computational notebooks, literate programming, and moldable development, all without asking programmers to abandon their favorite tools or give up their existing software engineering practices. Its design stems partially from the difficult lessons we learned after years of unsuccessfully trying to get our _own team_ to use an [online browser-based notebook platform][nextjournal] that we also developed. When working with Clerk, a split-view is typically used with a code editor next to a browser showing Clerk’s representation of the same notebook, as seen in [_Clerk side-by-side with Emacs_](#clerk-side-by-side-with-emacs). ```clojure (figure {:src "https://cdn.nextjournal.com/data/QmVYLx5SByNZi9hFnK2zx1K6Bz8FZqQ7wYtAwzYCxEhvfh?content-type=video/mp4" :poster-frame-src "https://cdn.nextjournal.com/data/QmSVeZpH3a5ag52AvodLDgBMA3AwnRrfawg5h2SQHk1rY3?filename=side-by-side.png&content-type=image/png" :id "clerk-side-by-side-with-emacs" :caption "Clerk side-by-side with Emacs" ::clerk/width :full :video? true}) ``` As shown here, our _notebooks_ are just source files containing regular Clojure code. Block comments are treated as markdown text with added support for LaTeX, data visualization, and so on, while top-level forms are treated as code cells that show the result of their evaluation.[^maria] This format allows us to use Clerk in the context of production code that resides in revision control. Because files decorated with these comment blocks are legal code without Clerk loaded, they can be used in many contexts where traditional notebook-specific code cannot. This has led, among other things, to Clerk being used extensively to publish documentation for libraries that are then able to ship artifacts that have no dependency on Clerk itself. [^maria]: We have borrowed this approach from [maria.cloud][maria], a web-hosted interactive Clojure learning tool created by [Matt Huebert](https://matt.is), [Dave Liepmann](https://www.daveliepmann.com/), and [Jack Rusher](https://jackrusher.com/). Maria grew out of [work presented at PX16](https://px16.matt.is) by Matt Huebert. Clerk’s audience is experienced Clojure developers who are familiar with interactive development. They are able to continue programming in their accustomed style, evaluating individual forms and inspecting intermediate results, but with the added ability to `show!` a namespace/file in Clerk. A visual representation of the file is then re-computed either: * every time the file is saved, using an an optional file watcher; or alternatively, * via an editor hot-key that can be bound to show the current document. (The authors generally prefer the hot-key over the file watcher, as it feels more direct and gives more control over when to show something in Clerk.) Control and configuration of Clerk primarily occurs through evaluation of Clojure forms from within the programmer's environment, rather than using outside control panels and settings. This integration with the programmer's existing tooling eases adoption and allows advanced customization of the system through code. ### Fast Feedback: Caching & Incremental Computation To keep feedback loops short, Clerk uses dependency analysis to limit recomputation to forms that haven't previously been evaluated in Clerk. In practice this means most changes to a Clerk document are reflected instantly (within 100ms) after saving a file or hitting the keybinding to update the open document. The caching works on the level of top-level forms. A hash is computed for each top-level form. A change to the form or one of its transitive dependencies will lead to a new hash value. When Clerk is asked to show a notebook, it will only evaluate forms that aren't cached in one of Clerk's two caches: * an in-memory cache stores a map of the hash of a given form to its current result. This cache is limited to the current forms of the active document. * An on-disk-cache stores the same information but to allow the user to continue work after a restart without recomputing potentially expensive operations.[^data-ingestion] Because Clojure supports lazy evaluation of potentially infinite sequences, safeguards are in place to skip caching unreasonable values. [^data-ingestion]: In tasks with intensive data preparation steps, this savings can be considerable. This caching behavior can be fine-tuned (or disabled) down to the level of individual forms. The on-disk caches use a content-addressed store where each result is stored using a filename derived from the SHA-2 hash of its contents. We use the self-describing [multihash format](https://multiformats.io/multihash/) which combines an identifier of the hash function with its digest length and value to support future changes of the hash algorithm. Additionally, a file named after the hash of a form contains a pointer to its results filename. This combination of immutability and indirection makes distributing the cache trivial using last-write wins for the tiny (90 bytes) pointer files. The content-addressed result cache files are never changed and can thus be synchronized without conflict. > While I did believe, and it has been true in practice, that the vast majority of an application could be functional, I also recognized that almost all programs would need some state. Even though the host interop would provide access to (plenty of) mutable state constructs, I didn’t want state management to be the province of interop; after all, a point of Clojure was to encourage people to stop doing mutable, stateful OO. In particular I wanted a state solution that was much simpler than the inherently complex locks and mutexes approaches of the hosts for concurrency-safe state. And I wanted something that took advantage of the fact that Clojure programmers would be programming primarily with efficiently persistent immutable data.[^history-of-clojure] > > – Rich Hickey [^history-of-clojure]: [A History of Clojure](https://doi.org/10.1145/3386321), Rich Hickey It is idiomatic in Clojure to use boxed containers to manage mutable state.[^clojure-state] While there are several of these constructs in the language, in practice [atoms](https://clojure.org/reference/atoms) are the most popular by far. An atom allows reading the current value inside it with [`deref/@`](https://clojuredocs.org/clojure.core/deref) and updating it's value with [`swap!`](https://clojuredocs.org/clojure.core/swap!). [^clojure-state]: [Values and Change: Clojure’s approach to Identity and State](https://clojure.org/about/state) When Clerk encounters an expression in which an atom's mutable value is being read using `deref`, it will try to compute a hash based on the value _inside_ the atom at runtime, and extend the expression's static hash with it. This extension makes Clerk's caching work naturally with idiomatic use of mutable state, and frees programmers from the need to manually opt out of caching for those expressions. ### Semantic Differences from Regular Clojure Clojure uses a single-pass, whole-file compilation strategy in which each evaluated form is added to the state of the running system. One positive aspect of this approach is that manually evaluating a series of forms produces the same result as loading a file containing the same forms in the same order, which is a useful property when interactively building up a program. A practical concern with this sort of "bottom-up" programming is that the state of the system can diverge from the state of the source file, as forms that have been deleted from the source file may still be present in the running system. This can lead to a situation where newly written code depends on values that will not exist the next time the program runs, causing surprising errors. To help avoid this, Clerk defaults to signaling an error unless it can resolve all referenced definitions in the runtime to the source code. It is our goal to match the semantics of Clojure as closely as possible but as a very dynamic language, there are limits to what Clerk's analysis can handle. Here are some of the things we currently do not support: * Multiple definitions of the same var in a file * Setting dynamic variables using [`set!`](https://clojuredocs.org/clojure.core/set!) * Dynamically altering vars using [`alter-var-root`](https://clojuredocs.org/clojure.core/alter-var-root) * Temporarily redefining vars using [`with-redefs`](https://clojuredocs.org/clojure.core/with-redefs) We have included a mechanism to override Clerk's error checking in cases where the user knows that one or more of these techniques are in use. ### Presentation Clerk uses a client/server architecture. The server runs in the JVM process that hosts the user's development environment. The client executes in a web browser running an embedded Clojure interpreter.[^sci] [^sci]: [Small Clojure Interpreter](https://github.com/babashka/sci) by Michiel Borkent The process of conveying a value to the client is a _presentation_, a term taken from Common Lisp systems that support similar features.[^presentations] The process of presentation makes use of _viewers_, each of which is a hash map from well-known keys to quoted forms containing source code for Clojure functions that specify how the client should render data structures of a given type. When a viewer form is received on the client side, it is compiled into a function that will be then called on data later sent by the server. [^presentations]: This feature originated on the Lisp Machine, and lives on in a reduced form as a feature of the emacs package [Slime](https://slime.common-lisp.dev/doc/html/Presentations.html). When the `present` function is called on the server side, it defaults to performing a depth-first traversal of the data structure it receives, attaching appropriate viewers at each node of the tree. The resulting structure containing bo ... ...

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：