Dataview table query materialization and line-level links to tasks in Obsidian

Published on:2025/01/13
Banner for Dataview table query materialization and line-level links to tasks in Obsidian

Introduction

Diving deeper into Dataview queries in Obsidian recently brought a few unexpected challenges.

And having played with the Obsidian/Dataview API enough recently, I knew that the technology supporting it was robust enough and that my issues had more to do with the orchestration than with the implementation.

Dataview tables cannot be annotated, are always refreshing, consuming way more resources than needed to print my 3 rows table and cannot be copy pasted among other things

Another gripe I had was with tasks:

While the Tasks-plugin itself allows for a special kind of query that returns interactive tasks with a dedicated link to that task (page and line) this has a lot of limitations and cannot be done in Dataview/js.

I mix, indiscriminately, between page-level results (the stuff in the frontmatter) and line-level results

Any list element with a [x] a the start will be considered a task of status x where x can be any single character.

Additionally, Dataview will take the in-line properties on that line and add them to the line only and not the file, essentially working around the file-level only approach of Obsidian.

I guess I'm still chasing that Orgmode high.

The dv.table() Re-Implementation

Dataview's interface to markdown objects is great and most of it (well dv.pages() and its file and file.tasks objects only but that's plenty) can be kept.

The main thing to reimplement is dv.table() or similar functions that actually created the element.

As far as I understand, Dataview block directly returns HTML to be renderer and mounted in the DOM, they don't produce markdown.

That comes with a few limitations if we're going another route like the ability to group rows.

Instead we can just run the same query, collect the output of the query into an object, slap some headers onto that object and coat the whole thing in simple markdown syntax, right?

Yes, there's even a Dataview function to help with that last part: dv.markdownTable(). And it even has a dv.markdownList() but this one you probably have to reimplement for yourself if you want nested lists

The "Shallow Copy" problem

There's maybe a better way to call it but any change made into a note will have to be reflected into the results.

Great, but maybe not always or not immediately at least.

Wether we don't want the change in the note to immediately reflect in the query or that we want to make some changes in the query result without changing the notes or the query: the ability to write into, copy, move and in general deal with a full copy of the markdown table is a great improvement:

It can be:

  • copied into another software like a chat that usually is compatible with markdown
  • annotated by adding or remove rows and columns
    • you can then let it be overwritten on next run
    • or copy this new version somewhere else new
  • it can be pasted into excalidraw

But we can push this idea of serializing a hard copy of a query's result even further:

  • write to JSON so it can be parsed by another program
    • maybe a web interface?
    • a nvim's Telescope function
    • a Python or Go TUI?
  • Process the markdown resulting of the query even further
    • why not run pandoc onto the results to create an always up-to-date morning standup material? Daily presentation, here we go

Specific example

let's consider the myQuery.js in my templater script folder to get all the item tagged with myTag at both the page-level and the line-level into the same table

async function query(tp) {
  let results = [];
  let headers = ["type", "status", "file", "summary", "line"];
  let dv = tp.app.plugins.plugins.dataview.api;

  dv.pages('Whatever pages you want htere')
    .where((p) => p.type !== "journal" && p.tags?.includes("myTag"))
    .forEach((p) =>
      results.push([
        "Note",
        p.type || "none",
        p.file.path,
        p.title || p.file.name,
        0,
      ]),
    );

  dv.pages('Whatever files you also want here')
    .file.tasks.where((t) => t.tag === "myTag")
    .forEach((t) => results.push(["Tasks", t.status, t.path, t.text, 0]));

  let tableString = dv.markdownTable(results, headers);
  return tableString;
}
module.exports = query;

Simple as that. Then, we can take the output of tp.user.myQuery(tp) and insert it at cursor with Templater insert's

Or, we can overwrite another file with:

let name = "theNameOfMyFile"
app.vault.adapter.write("_tmp/" + name + ".md", tableString);

And lasty, we could capture content of a file, filter some block marked with properties or the ^ syntax and replace it with the tableString.

Triggering a refresh

But I personally avoid having dataview codeblocks in my notes.

Instead, I make heavy use of properties both in frontmatter and in tasks while I have a handful of queries live as single-use document that I can embed somewhere else.

With this new approach I can extend the previous snippet to overwrite a new markdown file every time instead

I then have a templater module to gather all the other queries and document them into a single object

let queries = [
  ["query1", tp.user.myquery1, 2],
  ["query2", tp.user.myquery2, 1],
  ["query3", tp.user.myquery3, 1],
];

Where the first element is the name of the file to output in my _tmp folder, the second element the function to gather the data and the headers and the third the "mode" or what kind of query I'm expecting to have.

The last step allows me to not have to rewrite the usual helper (including the link makers, more on that later) functions when most of my queries have the same output. Namely a simple markdown table and a list for the Minimal theme cards.

I can then simply re-materialize all my queries with a:

let tp = app.plugins.plugins["templater-obsidian"].templater.current_functions_object
tp.user.serialAllQuery(tp)

Which I have bound to key of my HydraShard or can call external from the OS with Advanced URI

theJsBit=$(echo 'let tp=app.plugins.plugins["templater-obsidian"].templater.current_functions_object;tp.user.serialAllQuery(tp);' | jq -sRr @uri)
obsidian "obsidian://advanced-uri?eval=$theJsBit)"

Alternatively, I have a cronjob to re-run all queries in the morning.

Performance

Because the queries don't run as often, and more importantly, not when interacting with their output, this results in a buttery smooth, flicker free experiencce even on complex dashboards.

Even if the query was to take an awful long time, the final i/o step is always essentially instateneous.

The Elusive Task Links

And then there was the task linking problem. Returning tasks didn’t allow me to link directly back to the line where the task was.

Enters Dataview, once more.

Dataview exposes "tasks" (anything with a checkbox basically) as we saw in the query earlier, and they are are accesible in dv.page('myfileName).file.tasks.

One of the many properties of it is a task.position.start.line that simply returns the line number that the tasks starts at.

Great, we just need to build a link that brings us there.

Afterall, Electron is a whole browser dumped on a node.js backend, there must be a way to build a link handling behavior.

Surely, yes but let's try to find a plugin that already did the work.

AdancedURI comes to the rescue again. Not only does it have a line parameter to open a file at a given line but also an openmode one to control how does the file open. In this case making sure in silently opens in a new tab even with a simple click.

Now, as always it comes as a cost, since these are techncially external link, there are no more previews on hover.

We can start building links with something like:

const createAdvancedLink = (filePath, displayText, options = {}) => {
  const encodedPath = encodeURIComponent(filePath);
  let uri = `obsidian://advanced-uri?filepath=${encodedPath}`;

  if (options.line) {
    uri += `&line=${options.line}`;
  }
  if (options.openmode) {
    uri += `&openmode=${options.openmode}`;
  }

  return `[${tp.user.truncate(displayText)}](${uri})`;
};

then later

if (row.type === "task") {
        newRow.summary = createAdvancedLink(newRow.file, truncatedText, {
          line: row.line + 1,
          openmode: "tab",
        });
} 
if(row.type === "file") {
        newRow.summary = createAdvancedLink(newRow.file, truncatedText, {
          openmode: "tab",
        });
}

In order to get target links like:

obsidian://advanced-uri?filepath=myFileHere.md&openmode=tab&line=12

and markdown table cells like [Some Title Here for a task](obsidian://advanced-uri?filepath=myFileHere.md&openmode=tab&line=12)

This is why the query in the example was collecting the line number and the type of result.

Conclusion

We've lost some functionality: we can't see a preview of a file on hover in our queries and we can't group rows (well we could actually).

But we've gained so much:

  • Great performance. (and even the ability to let a single machine run the queries while the other sync the file, essentially creating a dataview server)
  • "Materialized" and static results opens the door to a lot more productivity.
  • Despite sounding pretty involved, documenting a new query takes very little effort, a simple js file in for the core and a new entry in the serialAllQuery.js module
  • Seemless line-level links for tasks