3 Years of VueJS (2021)

The code for all this is available here.

Have you ever wondered what a complex vue js application really looks like?

First let’s set the stage. This is a project that started nearly 3 years ago. There are issues we are aware of — and many we probably aren’t. If you happen to spot something please open an issue here.

Here are some examples of what the screens can look like.

Annotation Focused Modules

At a really high level the architecture is divided into 3 parts as shown here:

UI Components

These components are user interaction focused. The idea here is that these components remain relatively modular. As a developer, I can add or remove components from sending or receiving data. They can of course be positioned on the page as desired visually.

Annotation Core

This component is the “orchestrator” it coordinates between the group of UI components desired and the literal HTML canvas operations we want to do. Any data that’s shared lives here. It “mediates” between the user interaction, the app state, and the canvas.

Vue Canvas

This is responsible for literally rendering the visual. For example there are separate components for drawing the current Instance, drawing multiple instances, etc.

Example — Drawing a Polygon

Let’s zoom into a specific example. We want to draw a polygon. Perhaps a machine learning model predicted a polygon, or another user already drew it, or I drew it earlier.

In Diffgram a polygon is an instance of an annotation — or just an Instance for short.

We expect there to be many instances so we render them using a component called Instance

Overview of the Instance List Component


The component `instance_list.vue` receives an `instance_list` prop, a list of instances.

The instance_list component doesn’t care how the instance_list was loaded. instance_list could be a ground truth list from human annotation, a prediction list from a model, etc. All it cares about is how to render the given list.

This matters in part because it allows us to reuse the component in the video context too. So an animation frame concept can pass an instance list for each frame and this will render that just as well as a static list from an image.

Each instance has a `type`. For example, `type==’polygon’`. The main benefit for this approach is that we can arbitrarily add more instance types (box, cuboid, ellipse, etc), and none of the core structure of the code needs to change.

The main draw loop code is:

for (var i in this.instance_list) {   this.draw_single_instance(ctx, i)}

Within draw_single_instance there is a hook `draw_single_instance_limits` that checks for reasons we would not want to draw that instance. For example, it’s soft deleted, the parent label file is hidden etc.

Conditioning on type

else if ([“polygon”, “line”].includes(instance.type)) { ctx.beginPath() this.draw_polygon(instance, ctx, i)}

Execution Flow

draw_polygon() then has, for example this flow:

  1. draw_polygon_control_points()
  2. draw_many_polygon_circles()
  3. draw_circle_from_instance()
  4. draw_circle()
  5. is_mouse_in_path()


So part of the “render” function is to report back if the instance is the one being hovered.

At time of writing, this is used 23 times inside the instance_list. Why so many? Because some more complex annotations have multiple paths — for example the Ellipse type.

User control

Where the user can set a `v-model=”label_settings.vertex_size”`.

This gets passed down to `instance_list` as :vertex_size=”label_settings.vertex_size”.

Deeper in the polygon draw function, it eventually calls draw_circle() which uses that prop:

draw_circle: function (x, y, ctx) { ctx.arc(x, y, this.$props.vertex_size, 0, 2 * Math.PI); ctx.moveTo(x, y) // reset},

For medical use cases the ability to “hide” points completely is very useful!

Why such deep control? Why not use a higher level drawing library?

  1. Our needs often don’t appear to line up with those of the library. For example when it comes to rendering video at the breadth and depth we want to, it’s simply not something these higher level libraries tend to support.
  2. Support for Vue was largely missing when we started. While support is getting better, generally it ends up being thing wrappers.
  3. We have also found at least so far that often performance issues are rarely to do with these types of drawing functions themselves.

The built in “standard library” of HTML canvas is actually quite good already. While `.arc()` and `.moveTo()` may feel kind of manual, it’s actually still relatively abstracted from actual graphics rendering. We plan to support multiple screen annotation and multi-modal annotation in the future and suspect this type of control will come in very handy.

And finally, while the above polygon example focused on the drawing aspect, in general once a new type is implemented we don’t have ongoing problems with it. The bigger problems are usually more around user interaction — for example, changing the label (our application concept) of an existing instance, adding more detail to it, etc.

While in general we felt it was something we had to do, we can still analyze some other pros and cons

Other Pros

  1. It allows us to maintain a consistent definition. If we call something `x_min` — we can use that up and to the point we have to interact with `canvas`. Given we are in a scientific realm, it feels a lot more consistent to be very precise about this.
  2. Easy to modify. For example, we were able to “drop in” the user modifiable vertex size (and similar features) relatively easily.

Other Cons

  1. Sometimes it’s a bit of a distraction — reasonable user expectations like moving an instance, resizing, etc. can sometimes become a surprising amount of work to get right.


Why is this so hard? What’s the big deal?

Crucially — these concepts all interact with each other very closely. For example above we showed how a small user setting propagated through to the polygon rendering portion. In general this has led to a continued need to be more and more rigorous about what interacts with what, where state is stored, etc.

Annotation core got large

Code in JS — Presentation Layer in Vue

A challenge with Vue is that by default the methods aren’t readily reusable because the entire component gets imported in order to use methods and passing data between components is awkward for these more complex cases.

While the new Vue3 composition API may aim to solve some of this I think it’s better to let Vue focus on what it was meant for — the presentation layer.

Current Goal: All major logic that’s not directly related to changing the user visible screen is to be in normal JS classes. See userscripts.ts and userscripts.vue as a rough example of this new direction.

Each Instance Type has its Own Class

Goals: Isolation. Can edit class without affecting others, eg an update to Polygon doesn’t effect Box

Code direction Instance and KeypointInstance

User Interaction Paradigm

Tracking state becomes confusing. For example, if a user is in the middle of auto bordering between two polygons, “mouse down” has a different meaning than say when drawing a new instance.

The goal overtime is to essentially create an “interaction” layer that tracks and maps low level things into our system level concepts about what a user is doing.

More on this

Drawable Canvas — Divide Canvas and rest of Annotation UI

Example code

Make Error Propagation Easy — Regular Error

What if you could get something like this (expandable to as many errors as are present) virtually for “Free”?

Full JS code. Full component.

The net effect is that you can essentially add this one line:

.catch(error => { this.error = this.$route_api_errors(error) })

And then this one component

<v_error_multiple :error=”error”> </v_error_multiple>

And that’s it! We have an generation of this format with a backend component here. But … really it’s just any dict with an `error` key

Why does this matter

At the time of writing we use it in 73 places in the codebase. Of course for cases where you want other types of messages, more control, etc. you can do so — but it gives a relatively standardized feel across the app.

I’m not saying any of this is exactly best practice — but it has worked very well for us so far and feels like a reasonable direction to expand.

Wrap Library Components

We now have a small collection of “regular” methods, many of which are thin wrappers around library components. Code

One specific example is tooltip_button. It turns what is a fairly painful syntax with the library into one line:

To be clear — I’m not saying the library syntax is wrong — simply that for our use case it makes more sense to wrap it into 1 component.

At the time of writing <tooltip_button> is used in 112 places in the code base.

Overall, doing this across a number of components has allowed us to move faster. Implementing a new tooltip menu with a button can be a few minutes of coding — leaving more time for thinking about the actual product usage, testing, etc.

Button with Menu example

Why regular components


  • Gives us protection that if the library changes again(or we wish to move to another library, implementation etc) we can do so in one place.
  • Consistency. Encourages new developers to put icons and a descriptive message.
  • Is simple to read and less error prone.
  • Allows us to modify the library, eg adding the <a> tag so that default actions like right click work normally.
  • Allows us to set defaults, enforce typescript validation if needed, and generally control what library features are “enabled”.
  • Gives us a clear expansion point in the future — eg for multiple language support.


  • It is less flexible. There are some cases that it simply doesn’t cover.
  • There are occasionally awkward syntax changes that if “forces”. For example `data-cy=”data_cy”` for testing (`-` is changed to `_`). Most of these likely have a more elegant solution but the trade off of time to determine it is present.
  • Certain things become more difficult to pass around, for example `style`.


  • In theory it introduces a performance overhead. In reality most of this is so many calls deep that it’s not really relevant.
  • It creates more “risk” in that edits to it potentially have more far reaching effects. However — this is somewhat offset in that it forces us into a more rigorous testing mindset and makes bugs more apparent. It’s almost like a micro library dependency in a sense.

While we have a ways to go, so far it’s felt like a very strong move, and in general I hope to never have to call library components directly if we can avoid it.

Vuex — a comment

I have generally started avoiding it for any of the intense UI stuff. (There’s a few legacy things to move away from still)

This is mainly because it’s simply not fast enough for a lot of this intense UI work.

What I have found success using Vuex:

Like `$store.state.project.current.project_string_id`. The project is a very high level scope, and it’s significant in general if a user changes it. We have seen some issues with multiple tabs with this however so we are reviewing it.


eg `@change=”$store.commit(‘set_user_setting’, [‘studio_right_nav_width’, right_nav_width])”`. It feels pretty nice to just be able to set that key value pair from anywhere, and then load it as needed from` this.right_nav_width = this.$store.state.user.settings.studio_right_nav_width`. (Our user settings module has a ton of work to go but that part seems ok so far.)


I have found a balance of the two is often best.

Specifically for example, that the more complex the system becomes, the more we really want certain things to be in “once place”. It’s hard enough to reason about a complex chain, and having multiple “heads” (that aren’t part of the required logic) complicates things quickly and introduces errors.

On the other hand, overly attempting to build everything into a single “one true component” has its own set of issues. It creates tight coupling, and often seems to encourage “linear” programming — where everything must happen at a certain time. Essentially lots of assumptions.

WET makes less assumptions about the future, is often faster to write, and sometimes provides great insight to shift to a DRY approach later. There’s a cycle like nature to it. Start with WET, move to DRY as we learn the best structures, create WET concepts from those DRY structures, rinse and repeat!

Closing thoughts

Please consider starring and sharing the codebase. Especially with your machine learning friends!

Also we started a small server on Discord if you happen to be on it! Or even consider joining our team!






Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store