HTML5 Canvas on Rails? Part 4

Conclusion

Using the approach laid out in these blog posts, we’re able to keep with the spirit of using “sprinkles of Javascript” to make our server-rendered HTML “sparkle”. Delegating state management to the DOM and canvas manipulation/interactivity to our Stimulus controllers provides clear guidelines for how to separate concerns for different functionality. The approach lets each language/technology perform the role for which they are berst suited, and has an added benefit of minimizing the amount of Javascript, and arguably complexity, necessary to achieve the desired result.

While the Rails controller provides a RESTful approach to working with our resources, RawImage and Label, in a particular context, the StimulusJS controller is focused on handling click events to read/write data stored in the DOM and presenting those data visually.

Exercises for the reader

You didn’t think that this was as far as the journey goes, did you? There are a ton of additional other features that can, and probably should, be added in order to bring this idea to working production state. These include:

• persisting a scaled version of the annotation, regardless of the canvas size
• relatedly, enable dynamic scaling of canvas and annotations when resizing the window
• additional features that Defne’s version has implemented, including:
  • adding dots to indicate vertices of the annotation
  • moving a single vertex
  • moving the entire annotation
  • binding the annotation to the borders of the canvas
  • closing the polygon
  • undo functionality
• can you think of more?

Despite that, I hope this tutorial was helpful in demonstrating how we might extend the StimulusJS philsophy of injecting our HTML with “sprinkles of Javascript”. Happy coding!

Final State of the Rails Form and Stimulus Controller

Here are the 3 key files we built over the course of this tutorial in their entirety. Considering the complex interactivity of this feature, the two controllers are still relatively small (<100 lines of code).

# app/controllers/annotation_editor/raw_images_controller.rb
class AnnotationEditor::RawImagesController < ApplicationController
  before_action :set_raw_image, only: %i[show update]
  before_action :set_label, only: %i[show update]

  # GET annotation_editor/labels/1/raw_images/1
  def show
    @images = RawImage.all.order(:id)
    @labels = Label.all.order(:id)
    @previous_image = previous_record(@images, @raw_image)
    @next_image = next_record(@images, @raw_image)
    @previous_label = previous_record(@labels, @label)
    @next_label = next_record(@labels, @label)
    @annotation = Annotation.find_by(
      raw_image: @raw_image,
      label: @label
    ) || Annotation.new
  end

  # PATCH/PUT annotation_editor/labels/1/raw_images/1
  def update
    @annotation = Annotation.find_by(
      raw_image: @raw_image,
      label: @label
    ) || Annotation.new(raw_image: @raw_image, label: @label)
    @annotation.landmarks = raw_image_params[:landmarks]
    if @annotation.save
      flash[:notice] = "Annotation saved!"
    else
      flash[:alert] = "There was an error saving your annotation"
    end
    redirect_to annotation_editor_label_raw_image_path(@label, @raw_image)
  end

  private

  # Use callbacks to share common setup or constraints between actions.
  def set_raw_image
    @raw_image = RawImage.find(params[:id])
  end

  # Use callbacks to share common setup or constraints between actions.
  def set_label
    @label = Label.find(params[:label_id])
  end

  # Only allow a list of trusted parameters through.
  def raw_image_params
    params.require(:raw_image).permit(:landmarks)
  end

  def next_record(collection, record)
    record_idx = collection.find_index(record)
    record_idx + 1 >= collection.length ? collection[0] : collection[record_idx + 1]
  end

  def previous_record(collection, record)
    record_idx = collection.find_index(record)
    record_idx - 1 < 0 ? collection[-1] : collection[record_idx - 1]
  end
end

// app/javascript/controllers/annotation_canvas_controller.js
import { Controller } from '@hotwired/stimulus';
import Konva from 'konva';

// Connects to data-controller="annotation-canvas"
export default class extends Controller {
  static targets = ['canvas', 'landmarks'];

  static values = { imageUrl: String };

  connect() {
    this.stage = new Konva.Stage({
      container: 'annotation-container',
      width: this.canvasTarget.clientWidth,
      height: this.canvasTarget.clientHeight,
    });

    this.layer = new Konva.Layer();
    this.stage.add(this.layer);
    this.group = new Konva.Group({ name: 'annotation' });
    this.layer.add(this.group);

    const currentImage = new Image();
    currentImage.src = this.imageUrlValue;
    currentImage.onload = () => {
      this.image = new Konva.Image({
        x: 0,
        y: 0,
        image: currentImage,
        width: this.stage.width(),
        height: this.stage.height(),
      });

      // add the shape to the layer
      this.layer.add(this.image);
      this.drawAnnotation();
    };
    this.polygon = new Konva.Line({
      stroke: '#00F1FF',
      strokeWidth: 3,
      closed: false,
      fill: 'rgb(140,30,255,0.25)',
      id: 'annotation',
      name: 'polygon',
    });

    this.group.add(this.polygon);
  }

  handleClick() {
    const points = this.getCurrentPoints();
    const coordX = this.group.getRelativePointerPosition().x;
    const coordy = this.group.getRelativePointerPosition().y;
    points[points.length] = [coordX, coordy];
    this.landmarksTarget.value = JSON.stringify(points);
    this.drawAnnotation();
  }

  getCurrentPoints({ flattened } = { flattened: false }) {
    const points = JSON.parse(this.landmarksTarget.value)
    if (flattened) return points.reduce((a, b) => a.concat(b), []);
    return points;
  }

  drawAnnotation() {
    this.polygon.remove();
    this.polygon.points(this.getCurrentPoints({ flattened: true }));
    this.group.destroyChildren();
    this.group.add(this.polygon);
    this.polygon.draw();
    this.group.moveToTop();
  }

  reset() {
    this.landmarksTarget.value = JSON.stringify([]);
    this.drawAnnotation();
  }

  disconnect() {
    // clean up the canvas when navigating away
    this.stage.destroy();
  }
}

# app/views/annotation_editor/raw_images/_form.html.erb

<%= form_with(
  model: [:annotation_editor, label, raw_image], 
  class: 'd-flex flex-column page-header',
) do |form| %>
  <% if raw_image.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(raw_image.errors.count, "error") %> prohibited this raw_image from being saved:</h2>

      <ul>
      <% raw_image.errors.full_messages.each do |message| %>
        <li><%= message %></li>
      <% end %>
      </ul>
    </div>
  <% end %>
  <div class="d-flex flex-column " data-controller="annotation-canvas" 
    data-annotation-canvas-image-url-value=<%= url_for(raw_image.image) %> 
    id="annotation-controller">
      <div class='d-flex flex-row justify-content-between'>
        <div id="image-carousel">
          <%= link_to annotation_editor_label_raw_image_path(label, previous_image) do  %>
            <i class="bi-chevron-left fas my-5" id="previous-image"></i>
          <% end %>
          <span  class="my-5 px-2"><strong>Image:</strong> <%= raw_image.id %>
          <%= link_to annotation_editor_label_raw_image_path(label, next_image) do  %>
            <i class="bi-chevron-right fas my-5" id="next-image"></i>
          <% end %>
        </div>
        <div id="label-carousel">
          <%= link_to annotation_editor_label_raw_image_path(next_label, raw_image) do  %>
            <i class="bi-chevron-left fas my-5" id="previous-image"></i>
          <% end %>
          <span class="my-5 px-2"><strong>Label:</strong> <%= label.name %>
          <%= link_to annotation_editor_label_raw_image_path(previous_label, raw_image) do  %>
            <i class="bi-chevron-right fas my-5" id="next-image"></i>
          <% end %>
        </div>
        <div class="btn btn-primary btn-lg my-4" 
             style="--bs-btn-padding-x: 3rem; --bs-btn-padding-y: 0.25rem;"
             data-action="pointerdown->annotation-canvas#reset"
            >
          <i class="bi-trash fas my-4" id="save"></i> 
        </div>

        <%= form.button class: "btn btn-primary btn-lg my-4", style:"--bs-btn-padding-x: 3rem; --bs-btn-padding-y: 0.25rem;" do %>
          <i class="bi-save fas" id="save"></i> 
        <% end %>
      </div>
       <div 
          style="height: 70vh;"
          id="annotation-container"
          data-annotation-canvas-target="canvas"
          data-action="pointerdown->annotation-canvas#handleClick"
        />
    </div>
    <%= form.hidden_field :landmarks, data: { annotation_canvas_target: 'landmarks'}, value: "#{@annotation.landmarks || []}" %>
  </div>
<% end %>