Training Neural Nets in the Browser

Two months ago, Google announced its new TensorflowJS, for training and inference in the browser. This opens the door to a lot of amazing web applications.

Try out Webcam Pacman and Mobile Phone Emoji Hunt for a taste of what’s possible.

With TensorflowJS, we can take a pretrained model and personalize it with the user’s own data (keeping that data local and private!). We can also make fast predictions, without needing to wait to send data to a model up in the cloud.

I’m interested in how to teach deep learning well, and so I decided to take a week and explore TensorflowJS thoroughly. As Tensorflow Playground demonstrated, it’s so helpful for a deep learning newcomer to have the chance to tinker easily.

Continuing the idea of experimenting with hyperparameter and architectural choices, I created a demo page where you can try out training different kinds of models on the MNIST set and see in real time how your parameter choices affect the final accuracy. This is roughly based off of one of the TFJS tutorials, but I added in several different model and parameter options, and I fixed the bias in their train/test split. Hop over to my demo page to try it out.

Like everything in deep learning, TFJS is moving fast. Keep an eye on ml5.js. They’re building a wrapper around TensorflowJS that aims to be a “friendly machine learning library” for artists, hobbyists, and students. They have some beautiful looking demos. Personally I wouldn’t call it “friendly” just yet — they’re still missing a lot of documentation, and at this point nothing I tried out worked easily. I suspect in another month or two it’ll be in great shape.

Keep an eye on ml5js. It’s still developing (this is from the demo on their homepage and I’m 99.07% sure that’s *not* a robin!), but looks like it’ll be very exciting soon, especially for artists!

Getting to know TensorflowJS

I came to TensorflowJS with a fair amount of Tensorflow experience, but only a little web design experience. I suspect the path would actually be easier coming from the opposite direction. TensorflowJS itself is quite easy and a very natural extension of regular Tensorflow. I had no problems creating fun local extensions of the tutorials, though I then had a trickier time sorting out deploying to production (long story short, tfjs-angular-firebase seems a good way to go).

I began the week by working through the TensorflowJS Tutorials. Much like Tensorflow itself, you can work with TFJS at different levels of abstraction. Each tutorial focuses on one of these levels:

    1. Math: Polynomial Regressions introduces the lowest level. Here you can do math operations directly (adding, matrix multiplications, etc.). It’s not the level where you’d normally implement a model, but the neat thing is here we don’t need to be doing neural nets at all. In fact, check out the Realtime tSNE Visualization. It uses TFJS to create real-time interactive visualizations of high-dimensional data.


    3. Layers: MNIST Training moves one level up. This will feel very familiar to anyone who has used Keras. We start with model=tf.sequential() and then add layers (convolutional, fully connected, pooling, relu, etc.) to this model. Then we can compile the model and train it using
    4. Pretrained Model: The Pacman Tutorial is by far the most fun. This introduces importing a pretrained model (here we use mobilenet) and then we finetune that on a set of webcam images. We start with four categories (up/down/left/right), although it’s trivial to change this in the code. After fine tuning, we switch to prediction mode and feed these streaming results directly to the Pacman game, so we can now control the game with our webcam.


    Personally, I like to work through tutorials by looking over the code, then switching to a blank page and trying to recreate it myself. It’s more painful than just reading through the code samples, but I highly recommend this for code that you want to understand well — it’s amazing how many little details you notice by taking this extra step. In my case, the pacman tutorial took me a morning to recreate from scratch (initially it involved a fair amount of glancing back to look at the original code, but I soon felt increasingly independent).

    TensorflowJS Summary

    Based on my few days of playing with TensorflowJS – it works best when you have a straightforward pretrained model that needs fine tuning. TFJS isn’t geared for customized loss functions, lambda layers, or other personalizations. Things are changing fast though, and this may soon be easier to do. My main annoyance with TFJS is the lack of guidance – the tutorials are nice, but I couldn’t find good documentation beyond that. I found often I’d try to use a Tensorflow function, and later find that it doesn’t exist in TFJS.

    Between TFJS and TFLite (Tensorflow geared for mobile devices), new deep learning web and mobile apps will be very exciting to watch. In particular I think it’ll be a fantastic tool for artists, musicians, and educators.

    Demo Page

    As part of my tinkering this week, I created a TensorflowJS demo page that lets you try out several different models for training on MNIST. You can try convolutional nets with different filter sizes, or you can go the fully connected route and see how that compares. You can try removing the pooling layers or the reLU layers. Most variations eventually reach good accuracy, although some train more slowly than others. You can also play around with the learning rate and the number of batches to train.


    I’m excited for next week as I begin my dive into Reinforcement Learning. I’m planning to study the Deep RL Bootcamp and UC Berkeley 294 over the next three weeks and will continue to tell the tales here.

Learning About Deep Learning

I’m thrilled to be starting in the Scholars Program at OpenAI this June. I was a physics major and I’ve always loved math, but a year ago I didn’t have any deep learning or AI knowledge. I’d also stepped away from science for a few years while my kids were very young. This is how I got back up to speed. I’d love to hear from everyone else what they’ve found useful – please add comments about ideas, courses, competitions, scholarships, etc. that I’ve missed.

It’s both thrilling and completely overwhelming the amount you can learn online at this point.  Here’s what I found to be the main challenges of learning independently:

  1. Choosing what to work on
  2. Staying on schedule
  3. Learning actively (not passively!)
  4. Proving how much you’ve learned

Choosing what to work on

These are the courses I’ve found to be very high-yield and worth the time. I do browse at other courses, particularly if there’s a specific topic I need to learn, but these are the ones I recommend start-to-finish.

  • Jeremy Howard & Rachel Thomas’ FastAI sequence. The 2017-2018 course uses PyTorch. There’s also an older version of the course which has mostly similar material in Keras and Tensorflow.
  • Andrew Ng’s Deep Learning Specialization on Coursera. This dives more into the math behind deep learning and is a fantastic overall introduction. Sometimes the techniques taught are less cutting-edge than the FastAI ones.
  • Jose Portilla’s Python courses on Udemy, particularly Python for Data Science. I came into this not knowing Python at all, so appreciated having this great introduction to python, numpy, scipy, and pandas.
  • way to test your skills and learn the current cutting edge.
  • HackerRank – great place to prepare for interviews and programming tests (it’s not deep learning specific)

Staying on Schedule

Here I think the most important is to know your own personality and to play to that. I’m very project-oriented and goal-oriented. Once I’m working on a specific task, I have no trouble staying with that for hours. So I’ve tended to binge-watch courses (particularly Andrew Ng’s series). On the other hand, I know I’m not very good at jumping between several different projects, and when I don’t have a specific goal. I try to keep this in mind when planning my schedule for the week.

I also like Jeremy Howard’s advice for working on Kaggle competitions. He suggests without fail working a half-hour *every* day on the project. Each day you make some slight incremental progress.

With that in mind, I try to learn one new topic every day (even if it’s a relatively small detail). Either by reading a paper, watching a course video, or reading a blog post. Recently I met some of the Google Brain team, and when the topic turned to Variational Auto-Encoders, by chance I knew all about them since they’d happened to be my chosen topic one day. I keep a small journal with a half a page of notes on whatever I learn each day.

Learning actively (not passively!)

The big danger of online courses is that it’s far too easy to watch a bunch of videos (on 1.5x speed) and then a week later not remember any of it.

Both the FastAI and Deep Learning Specialization have very active forums. It’s definitely worth participating in those – both asking questions and trying to answer others. After taking the Coursera sequence, I was invited to become a mentor for the CNN and RNN courses, and I’m sure I’ve learned far more from trying to teach others than I did taking the course on my own.

This is also where the Kaggle competitions are extremely valuable. It’s one thing to be able to follow through a Jupyter Notebook. It’s something totally different to start with just the data and create all the structure from scratch.

Proving how much you’ve learned

After all the online courses, it’s helpful to create some tangible proof of what you know. My suggestions are a github project, a Kaggle competition, and some blog posts.

Jeremy Howard gave me the advice to focus on one great project, rather than several decent, half-baked ones. He says to really polish it up, even to make a nice web interface.

Along the same lines, it’s great practice to try out several different Kaggle competitions, but it’s important to focus on one and score highly on that one.

I’ve written a lot as a mentor for Andrew Ng’s courses. I’ve always been impressed how I have to understand things so much more deeply in order to explain them well in writing. This is my first foray into blog post writing – I’m naturally a fairly quiet and reserved person, so I’m having to consciously push myself to do this, but it’s also an exciting way to connect with the data science world.