Bye-bye Latency! Building a Private Skin-Lesion Screening AI Using WebLLM and WebGPU

Privacy in healthcare tech is no longer just a "nice-to-have"—it's a legal and ethical requirement. When dealing with sensitive photos, like skin lesion screenings, the traditional approach of sending pixels to a remote server feels... wrong.

In this tutorial, we are diving deep into Edge AI, WebGPU acceleration, and In-browser Vision Models. We’ll build a high-performance screening tool that performs local inference directly on the user's device. By leveraging WebLLM and the TVM Runtime, we ensure that high-sensitivity medical images never leave the browser, providing zero-latency feedback while maintaining 100% data privacy. 🛡️

The Vision: Why Edge AI for Healthcare?

Most AI developers default to API calls (OpenAI, Anthropic, etc.). However, for vision-based medical screening, the "Cloud-First" approach has two major flaws:

1. Privacy: Users are rightfully hesitant to upload photos of their skin to a third-party server.
2. Latency: Uploading high-res images, processing them, and waiting for a response kills the user experience.

By using WebGPU, we tap into the raw power of the local graphics card directly through the browser. Combined with WebLLM (powered by Apache TVM), we can run compressed vision-language models or classification backends at near-native speeds.

🏗️ The System Architecture

Here is how the data flows from the user's camera to the local GPU without ever touching a backend server:

graph TD
    A[User Camera/Upload] -->|Image Data| B(React Frontend)
    B --> C{WebGPU Supported?}
    C -->|Yes| D[WebLLM Engine]
    C -->|No| E[Wasm Fallback / Error]
    D --> F[TVM Runtime / Wasm Module]
    F -->|Local GPU Inference| G[Model Weights - Sharded]
    G --> H[Classification Results]
    H -->|Local State| B
    B --> I[Visual Feedback / Privacy Shield]

🛠️ Tech Stack

• WebLLM: The high-performance in-browser LLM/Vision engine.
• WebGPU: The next-gen API for GPU acceleration in the browser.
• TVM Runtime: Compiled machine learning kernels for the web.
• React: For building a responsive, state-driven UI.

🚀 Step-by-Step Implementation

1. Initializing the WebLLM Engine

First, we need to set up the engine to utilize the local GPU. Unlike standard API wrappers, WebLLM requires us to manage the model's lifecycle and the WebGPU device.

import * as webllm from "@mlc-ai/web-llm";

const useVisionModel = () => {
  const [engine, setEngine] = useState<webllm.MLCEngine | null>(null);
  const [loadingProgress, setLoadingProgress] = useState(0);

  const initEngine = async () => {
    // We use a Vision-capable model sharded for the web
    const modelId = "Llama-3-Vision-8B-WebGPU-q4f16_1"; 

    const engineInstance = await webllm.createMLCEngine(modelId, {
      initProgressCallback: (report) => {
        setLoadingProgress(report.progress * 100);
        console.log("Loading Progress:", report.text);
      },
    });

    setEngine(engineInstance);
  };

  return { engine, initEngine, loadingProgress };
};

2. Processing the Image for Inference

Since we are doing zero-latency screening, we need to convert the user's image into a format the model understands (typically a base64 string or a tensor) and prompt the model for a screening analysis.

const handleInference = async (imageFile: File) => {
  if (!engine) return;

  // Convert File to Base64 for the Vision Model
  const reader = new FileReader();
  reader.readAsDataURL(imageFile);
  reader.onload = async () => {
    const base64Image = reader.result as string;

    const messages = [
      {
        role: "user",
        content: [
          { type: "text", text: "Analyze this skin lesion. Focus on symmetry, border, and color. Is this high risk?" },
          { type: "image_url", image_url: { url: base64Image } }
        ],
      },
    ];

    const reply = await engine.chat.completions.create({
      messages,
      temperature: 0.2, // Keep it deterministic for medical screening
    });

    console.log("Screening Result:", reply.choices[0].message.content);
  };
};

3. Creating the React UI

The UI needs to be intuitive. We’ll use a simple file input and a progress bar to show the model being cached into the browser's IndexedDB.

export const SkinScreeningTool = () => {
  const { engine, initEngine, loadingProgress } = useVisionModel();

  return (
    <div className="p-8 max-w-2xl mx-auto bg-white rounded-xl shadow-md">
      <h2 className="text-2xl font-bold mb-4">🛡️ Private Skin Screening</h2>

      {!engine ? (
        <button 
          onClick={initEngine}
          className="bg-blue-600 text-white px-4 py-2 rounded"
        >
          Initialize Local AI ({loadingProgress.toFixed(0)}%)
        </button>
      ) : (
        <div className="space-y-4">
          <input 
            type="file" 
            accept="image/*" 
            onChange={(e) => e.target.files && handleInference(e.target.files[0])}
            className="block w-full text-sm text-slate-500"
          />
          <div className="p-4 bg-gray-50 border-l-4 border-yellow-400">
            <p className="text-sm italic">
              Note: Processing happens 100% on your device via WebGPU. 
              No data is sent to our servers.
            </p>
          </div>
        </div>
      )}
    </div>
  );
};

💡 The "Official" Way to Optimize Edge AI

While this tutorial gets you started with basic vision inference, building production-ready medical tools requires advanced techniques like model quantization (4-bit vs 8-bit), memory management for mobile devices, and multi-threading with Web Workers.

For more advanced patterns and production-ready examples of local AI deployment, I highly recommend exploring the deep-dive articles over at the WellAlly Tech Blog. They cover the intricate details of TVM optimization and cross-platform WebGPU performance that are essential for high-stakes applications. 🥑

Conclusion 🏁

We’ve just bypassed the cloud entirely. By using WebLLM and WebGPU, we've built a tool that is:

1. Blazing Fast: Inference happens at the speed of the local hardware.
2. Private by Design: Zero server-side storage or processing.
3. Cost Effective: No more $0.01 per token API bills!

The future of the web is heavy. Not in terms of bloat, but in terms of local intelligence. Edge AI is democratizing access to powerful models while finally solving the privacy paradox.

What are you planning to build with WebGPU? Let me know in the comments below! 👇