3 Years of VueJS (2021)

The code for all this is available here.

Have you ever wondered what a complex vue js application really looks like?

Today I’m happy to take a deep dive into the architecture of Diffgram’s Open Annotation UI. This is intended for people who are curious about VueJS at scale.

First let’s set the stage. This is a project that started nearly 3 years ago. There are issues we are aware of — and many we probably aren’t. If you happen to spot something please open an issue here.

Here are some examples of what the screens can look like.

Annotation Focused Modules

While there are hundreds of components in the system, one of the most interesting is the Annotation focused ones.

At a really high level the architecture is divided into 3 parts as shown here:

UI Components

These components are user interaction focused. The idea here is that these components remain relatively modular. As a developer, I can add or remove components from sending or receiving data. They can of course be positioned on the page as desired visually.

Annotation Core

This component is the “orchestrator” it coordinates between the group of UI components desired and the literal HTML canvas operations we want to do. Any data that’s shared lives here. It “mediates” between the user interaction, the app state, and the canvas.

Vue Canvas

This is responsible for literally rendering the visual. For example there are separate components for drawing the current Instance, drawing multiple instances, etc.

Example — Drawing a Polygon

Let’s zoom into a specific example. We want to draw a polygon. Perhaps a machine learning model predicted a polygon, or another user already drew it, or I drew it earlier.

In Diffgram a polygon is an instance of an annotation — or just an Instance for short.

We expect there to be many instances so we render them using a component called Instance

Overview of the Instance List Component

Now I unpack what the instance list component is doing. Here the black are functions shared between all instance types. The purple is polygon specific.

Code

The component `instance_list.vue` receives an `instance_list` prop, a list of instances.

The instance_list component doesn’t care how the instance_list was loaded. instance_list could be a ground truth list from human annotation, a prediction list from a model, etc. All it cares about is how to render the given list.

This matters in part because it allows us to reuse the component in the video context too. So an animation frame concept can pass an instance list for each frame and this will render that just as well as a static list from an image.

Each instance has a `type`. For example, `type==’polygon’`. The main benefit for this approach is that we can arbitrarily add more instance types (box, cuboid, ellipse, etc), and none of the core structure of the code needs to change.

The main draw loop code is:

for (var i in this.instance_list) {
   this.draw_single_instance(ctx, i)
}

Within draw_single_instance there is a hook `draw_single_instance_limits` that checks for reasons we would not want to draw that instance. For example, it’s soft deleted, the parent label file is hidden etc.

Conditioning on type

Code

else if ([“polygon”, “line”].includes(instance.type)) {
 ctx.beginPath()
 this.draw_polygon(instance, ctx, i)
}

Execution Flow

draw_polygon() then has, for example this flow:

1. draw_polygon_control_points()
2. draw_many_polygon_circles()
3. draw_circle_from_instance()
4. draw_circle()
5. is_mouse_in_path()

is_mouse_in_path?

One of the common questions other components ask is “Which instance is the user on?” Often, it’s easiest to determine this while drawing the path.

So part of the “render” function is to report back if the instance is the one being hovered.

At time of writing, this is used 23 times inside the instance_list. Why so many? Because some more complex annotations have multiple paths — for example the Ellipse type.

User control

How does the canvas know how large to make the control points? What if the user doesn’t want any control points? Zooming out back up to Annotation Core, we see there is a little<v-slider> component.

Where the user can set a `v-model=”label_settings.vertex_size”`.

This gets passed down to `instance_list` as :vertex_size=”label_settings.vertex_size”.

Deeper in the polygon draw function, it eventually calls draw_circle() which uses that prop:

draw_circle: function (x, y, ctx) {
 ctx.arc(x, y, this.$props.vertex_size, 0, 2 * Math.PI);
 ctx.moveTo(x, y) // reset
},

For medical use cases the ability to “hide” points completely is very useful!

Why such deep control? Why not use a higher level drawing library?

Why:

1. Our needs often don’t appear to line up with those of the library. For example when it comes to rendering video at the breadth and depth we want to, it’s simply not something these higher level libraries tend to support.
2. Support for Vue was largely missing when we started. While support is getting better, generally it ends up being thing wrappers.
3. We have also found at least so far that often performance issues are rarely to do with these types of drawing functions themselves.

The built in “standard library” of HTML canvas is actually quite good already. While `.arc()` and `.moveTo()` may feel kind of manual, it’s actually still relatively abstracted from actual graphics rendering. We plan to support multiple screen annotation and multi-modal annotation in the future and suspect this type of control will come in very handy.

And finally, while the above polygon example focused on the drawing aspect, in general once a new type is implemented we don’t have ongoing problems with it. The bigger problems are usually more around user interaction — for example, changing the label (our application concept) of an existing instance, adding more detail to it, etc.

While in general we felt it was something we had to do, we can still analyze some other pros and cons

Other Pros

1. It allows us to maintain a consistent definition. If we call something `x_min` — we can use that up and to the point we have to interact with `canvas`. Given we are in a scientific realm, it feels a lot more consistent to be very precise about this.
2. Easy to modify. For example, we were able to “drop in” the user modifiable vertex size (and similar features) relatively easily.

Other Cons

1. Sometimes it’s a bit of a distraction — reasonable user expectations like moving an instance, resizing, etc. can sometimes become a surprising amount of work to get right.

Problems

Why is this so hard? What’s the big deal?

To try and give an idea of scope, if I imagine “flattening” all of the annotation imports (non library code), there are probably over 45,000 unique lines.

Crucially — these concepts all interact with each other very closely. For example above we showed how a small user setting propagated through to the polygon rendering portion. In general this has led to a continued need to be more and more rigorous about what interacts with what, where state is stored, etc.

Annotation core got large

Annotation Core become a bottleneck. Despite importing 35 components, the file has grown to an insane 7569 lines of code. While that’s “only” 15% of the overall sub system, it’s still waaaaaay too large to reasonably reason about. There’s many reasons for this. I’ll unpack a few and speak to our new direction for it.

Code in JS — Presentation Layer in Vue

First — to be clear we are entirely “on team Vue”. We are fans. It’s easier to use then react and just as scalable.

A challenge with Vue is that by default the methods aren’t readily reusable because the entire component gets imported in order to use methods and passing data between components is awkward for these more complex cases.

While the new Vue3 composition API may aim to solve some of this I think it’s better to let Vue focus on what it was meant for — the presentation layer.

Current Goal: All major logic that’s not directly related to changing the user visible screen is to be in normal JS classes. See userscripts.ts and userscripts.vue as a rough example of this new direction.

Each Instance Type has its Own Class

As we add more instance types, and as these interactions become more complex, we have realized that it is needed to have dedicated classes for each type.

Goals: Isolation. Can edit class without affecting others, eg an update to Polygon doesn’t effect Box

Code direction Instance and KeypointInstance

User Interaction Paradigm

As interactions become more complex, the built-in primitives of “mouse down” “mouse up” etc start to lose their meaning.

Tracking state becomes confusing. For example, if a user is in the middle of auto bordering between two polygons, “mouse down” has a different meaning than say when drawing a new instance.

The goal overtime is to essentially create an “interaction” layer that tracks and maps low level things into our system level concepts about what a user is doing.

More on this

Drawable Canvas — Divide Canvas and rest of Annotation UI

Another divide and conquer opportunity — separating the concern of canvas drawing, from annotation UI aspects. We made great progress on this and have an example here (being used in new Templating setup).

Example code

Make Error Propagation Easy — Regular Error

Let’s say that you wanted to make a new endpoint. How do you propagate errors back to the user?

What if you could get something like this (expandable to as many errors as are present) virtually for “Free”?

Full JS code. Full component.

The net effect is that you can essentially add this one line:

.catch(error => { this.error = this.$route_api_errors(error) })

And then this one component

<v_error_multiple :error=”error”> </v_error_multiple>

And that’s it! We have an generation of this format with a backend component here. But … really it’s just any dict with an `error` key

Why does this matter

At the time of writing we use it in 73 places in the codebase. Of course for cases where you want other types of messages, more control, etc. you can do so — but it gives a relatively standardized feel across the app.

I’m not saying any of this is exactly best practice — but it has worked very well for us so far and feels like a reasonable direction to expand.

Wrap Library Components

A little while back, a UI component library made a big modification. It caused a bunch of breaking changes that were a huge pain to fix.

We now have a small collection of “regular” methods, many of which are thin wrappers around library components. Code

One specific example is tooltip_button. It turns what is a fairly painful syntax with the library into one line:

To be clear — I’m not saying the library syntax is wrong — simply that for our use case it makes more sense to wrap it into 1 component.

At the time of writing <tooltip_button> is used in 112 places in the code base.

Overall, doing this across a number of components has allowed us to move faster. Implementing a new tooltip menu with a button can be a few minutes of coding — leaving more time for thinking about the actual product usage, testing, etc.

Button with Menu example

Code

Why regular components

Both the Regular Error and Button with Menu are examples of reusable, and fairly abstract, components. While abstract and reusable components is standard practice in many paradigms, in the context of 3rd party UI libraries it can be surprisingly hard to do. I think the effort is worth it because the pros outweigh the cons. Overall it allows us to move faster and ship better quality code.

Pros

• Gives us protection that if the library changes again(or we wish to move to another library, implementation etc) we can do so in one place.
• Consistency. Encourages new developers to put icons and a descriptive message.
• Is simple to read and less error prone.
• Allows us to modify the library, eg adding the <a> tag so that default actions like right click work normally.
• Allows us to set defaults, enforce typescript validation if needed, and generally control what library features are “enabled”.
• Gives us a clear expansion point in the future — eg for multiple language support.

Cons

• It is less flexible. There are some cases that it simply doesn’t cover.
• There are occasionally awkward syntax changes that if “forces”. For example `data-cy=”data_cy”` for testing (`-` is changed to `_`). Most of these likely have a more elegant solution but the trade off of time to determine it is present.
• Certain things become more difficult to pass around, for example `style`.

Neutral

• In theory it introduces a performance overhead. In reality most of this is so many calls deep that it’s not really relevant.
• It creates more “risk” in that edits to it potentially have more far reaching effects. However — this is somewhat offset in that it forces us into a more rigorous testing mindset and makes bugs more apparent. It’s almost like a micro library dependency in a sense.

While we have a ways to go, so far it’s felt like a very strong move, and in general I hope to never have to call library components directly if we can avoid it.

Vuex — a comment

We do use Vuex for some things.

I have generally started avoiding it for any of the intense UI stuff. (There’s a few legacy things to move away from still)

This is mainly because it’s simply not fast enough for a lot of this intense UI work.

What I have found success using Vuex:

Things that change infrequently, but are needed everywhere.

Like `$store.state.project.current.project_string_id`. The project is a very high level scope, and it’s significant in general if a user changes it. We have seen some issues with multiple tabs with this however so we are reviewing it.

Settings

eg `@change=”$store.commit(‘set_user_setting’, [‘studio_right_nav_width’, right_nav_width])”`. It feels pretty nice to just be able to set that key value pair from anywhere, and then load it as needed from` this.right_nav_width = this.$store.state.user.settings.studio_right_nav_width`. (Our user settings module has a ton of work to go but that part seems ok so far.)

WET & DRY

Zooming out for a moment. Two major paradigms in programming are WET (Write everything twice) and DRY (Don’t repeat yourself).

I have found a balance of the two is often best.

Specifically for example, that the more complex the system becomes, the more we really want certain things to be in “once place”. It’s hard enough to reason about a complex chain, and having multiple “heads” (that aren’t part of the required logic) complicates things quickly and introduces errors.

On the other hand, overly attempting to build everything into a single “one true component” has its own set of issues. It creates tight coupling, and often seems to encourage “linear” programming — where everything must happen at a certain time. Essentially lots of assumptions.

WET makes less assumptions about the future, is often faster to write, and sometimes provides great insight to shift to a DRY approach later. There’s a cycle like nature to it. Start with WET, move to DRY as we learn the best structures, create WET concepts from those DRY structures, rinse and repeat!

Closing thoughts

Thank you very much for reading!

Please consider starring and sharing the codebase. Especially with your machine learning friends!

Also we started a small server on Discord if you happen to be on it! Or even consider joining our team!

Best,

Anthony