Introduction

Having played with the great Obsidian/Dataview APIs enough recently, but wanting to not always have to go through the sometime clunky electron app, I started wanting to find ways to get data in and out of Obsidan through my usual tool (tmux, nvim etc).

Alternatively, here are few gripes I had with Dataview:
Dataview tables cannot be annotated (are always refreshing, consuming way more resources than needed to print my 3 rows table) and cannot be copy pasted among other things

Another gripe I had was with tasks:

While the Tasks-plugin itself allows for a special kind of query that returns interactive tasks with a dedicated link to that task (page and line), this has a lot of limitations and cannot be done in Dataview/js.

I personally mix, indiscriminately, between page-level results (the stuff in the frontmatter) and line-level results (the tasks)

Before we go any further, it's good to know that Dataview will treat any list element with a [x] a the start will be considered a task of status x where x can be any single character.

Let's see how we can improve on that.

Replacing dv.table()

Dataview's interface to markdown objects is great and most of it can be kept.

The main thing to reimplement is dv.table() or any of the "frontend" function that create elements.

As far as I understand, Dataview block directly returns HTML to be renderer and mounted in the DOM, they don't produce markdown as a middle step.

Instead we can just run the same query in pure JS/templater, collect the output of the query, slap some headers and coat the whole thing in simple markdown syntax, right?

In fact we don't really have to do the last part because dataview prodives an abstraction for thi as well: dv.markdownTable()

Solving the "direct reference" issue

Displaying the result of a query we sometimes want to annotate locally without having to modify the remote data. That is not how dataview works though.

By having a "mutable version" of the result of our query we can locally modify as text like without having to modify the remote data.

More than just for annotations, with markdown working in a lot of places it can for instance work in most chat apps. In fact you could write your "trigger" script to directly put the markdown result into clipboard instead of using a file as an intermediary.

But we can push this idea of serializing a hard copy of a query's result even further by writing the result of the query in json instead of mardkdown. This allows any other program to be able to parse and use the data.

Even better, with a remote execution of this function through Advanced URI and a templater module, we are getting data back, essentially creating a basic interprocess communication with Obsidian.

Specific example

let's consider the myQuery.js in my templater script folder to get all the item tagged with myTag at both the page-level and the line-level into the same table

async function query(tp) {
  let results = [];
  let headers = ["type", "status", "file", "summary", "line"];
  let dv = tp.app.plugins.plugins.dataview.api;

  dv.pages('Whatever pages you want there')
    .where((p) => p.type !== "journal" && p.tags?.includes("myTag"))
    .forEach((p) =>
      results.push([
        "Note",
        p.type || "none",
        p.file.path,
        p.title || p.file.name,
        0,
      ]),
    );

  dv.pages('Whatever files you also want here')
    .file.tasks.where((t) => t.tag === "myTag")
    .forEach((t) => results.push(["Tasks", t.status, t.path, t.text, 0]));

  let tableString = dv.markdownTable(results, headers);
  return tableString;
}
module.exports = query;

The important part here is that we have collected both tasks and files and will be displayed side by side.

Then, we can take the output of tp.user.myQuery(tp) and insert it at cursor with Templater insert's

Or, we can overwrite another file with:

let name = "theNameOfMyFile"
app.vault.adapter.write("_tmp/" + name + ".md", tableString);

And lasty, we could capture content of a file, filter some block marked with properties or the ^ syntax and replace it with the tableString.

Triggering a refresh

You can send individual signals based on needs or you can document all the queries and have a script that run them all. Or I can then simply re-materialize all my queries with a:

let tp = app.plugins.plugins["templater-obsidian"].templater.current_functions_object
tp.user.serialAllQuery(tp)

Which I have bound to key of my HydraShard or can call external from the OS with Advanced URI

theJsBit=$(echo 'let tp=app.plugins.plugins["templater-obsidian"].templater.current_functions_object;tp.user.serialAllQuery(tp);' | jq -sRr @uri)
obsidian "obsidian://advanced-uri?eval=$theJsBit)"

Templated rendering

Going further than the simple dataview built-in solution we can create personal archetypes of types of queries and program how to render them:

(cleaning internal variables, adding glyphs, emojis, ordering certain field etc)

We can then keep track of our queries and associated rendering templates like this

let queries = [
  ["query1", tp.user.myquery1, 2],
  ["query2", tp.user.myquery2, 1],
  ["query3", tp.user.myquery3, 1],
];

  const processQuery = async (query) => {
    const [name, func, mode] = query;
    try {
      const data = await func(tp);
      const tasks = [];
      if (mode === 1) {
        tasks.push(tp.user.marshalQueryToMarkdown(tp, name, data));
      }
      if (mode === 2) {
        tasks.push(tp.user.marshalQueryToMarkdownList(tp, name, data));
      }
      if (mode === 3) {
        tasks.push(tp.user.marshalQueryToThumbnailList(tp, name, data));
      }
      tasks.push(tp.user.marshalQueryToJson(tp, name, data));
      await Promise.all(tasks);
      return { name, success: true };
    } catch (error) {
      console.error(`Error processing ${name}:`, error);
      return { name, success: false, error };
    }
  };

Here is how my serialAll script works with each query renders according to its model and writes its results as json as well.

Coming up

Next let's see how we can leverage this setup to have line-level links (for tasks) which is not strictly possible in obsidian.