The Spider and the Librarian: A Self-Teaching Framework

I had a huge response from friends and new readers who were curious to learn more about the productivity tools I referenced in my last post. I thought it would be helpful to take a step back and describe the mental framework I use when charting a self-teaching path. To really make the most of the vast resources available to you out on the web, it's helpful to think of yourself as filling two distinct roles or personas: I call them "the spider" and "the librarian."

The Spider: Being a Human Web-Crawler

Have you ever wondered how a search engine like Google works? It's actually not extremely complicated, at least conceptually (in fact, if you take Udacity's Intro to Computer Science course, you'll end up creating a simple search engine as one of your very first projects).

It's easy in the era of complex, single-page web applications to forget that the web is simply a collection of documents, joined together by hyperlinks. Anybody who has ever "gotten lost in Wikipedia" for a few minutes (or hours) is familiar with this concept. You're on one page reading up about the French Revolution, and somehow that leads you to click a link over to an article about Haiti... follow enough links in these documents, and you could end up on a page about a subject that isn't even related to where you started - like an article about electric cars or DNA or anthropology. It's like Six Degrees of Kevin Bacon on steroids.

Making sense of this massive tangle of interconnected knowledge is a job best-suited for computers - hence the need for search engines. Can you imagine how difficult it would be to find what you needed on the web if you had to start only from the sites you were familiar with, and discover new content by following links? You might never find what you were looking for.

Fortunately, computers are our tireless servants, and people like Google and Microsoft use programs called "spiders" or "web crawlers" that systematically do this job for you - they go from page to page, following links, generating a massive index of all of the sites on the world wide web.

There are, however, two distinct disadvantages to search engines: 1) You have to know what you're looking for, more or less, and 2) the content that will be served up in response to your query is ranked by an algorithm that you don't really understand. This second point is arguably less of an issue nowadays, as Google has gotten very clever at determining how to rank the sites that show up in your search results based on how useful they think each site will be to you.

The first issue is a bit more of a sticking point. So much of learning... of self-teaching... is a process of discovery. What do you do if you don't know what you're really looking for?

The answer is to become your own web crawler.

Don't worry, this isn't as onerous as it sounds. Thanks to tools like Todoist, it's very easy to capture your travels across the internet and stash content for you to review later.

It all comes down to the idea of "universal capture" - using a task management tool like Todoist that works across many platforms (email client, browser, mobile) allows you access to an inbox for all of the "stuff" you find on the internet, usually with only a few mouse clicks or key strokes.

Let's use a concrete example: last night I attended a meetup in Arlington about "Front-end Development for Full-Stack Engineers" - if you're not into web development, this basically just means it was a talk about building the web apps you see in your browser when you use stuff like gmail, your online banking site, etc. As often happens at these events, the presentation and conversation went in all sorts of interesting directions.

At some point, the presenter mentioned the idea of web software moving more and more towards "microservices." If you don't know what that means, don't worry - I didn't either! Rather than let that important piece of information just wash over me, however, I pulled out my phone, tapped the "Add New Task" Todoist button that sits right on my home screen, and just threw a simple, one-word task into my inbox: "microservices."

If you're a fan of the "Getting Things Done" style of productivity, you'll recognize this as part of the "collection" phase - I realized there was something I needed to do (look up info about microservices), so I chucked it in my inbox and put it out of my mind so I could focus on the rest of the presentation.

Later, when I got home, it was time to process my inbox in Todoist. When I got to my note about "microservices," I decided that it would probably take less than 2 or 3 minutes to find some good resources explaining this term, so I decided to knock that out right away.

I did a quick search for "building microservices in node.js" (since node.js is the software development environment I'm most familiar with), and quickly found a few blog posts that looked interesting. Rather than get bogged down in each post, however, I quickly just scanned through and played spider for a bit: I followed links in the blog posts that led to other articles on related concepts, until I felt like I had a decent starting point for researching all of the related technologies.

Here is the crucial point, and the point related to the idea of "universal capture" - using Todoist's browser extension in Chrome, with just two clicks you can "Add website as task." So, with every new resource I located, I clicked on my trusty Todoist button in Chrome, and added a link to each resource to my todo list. Later, when I was organizing my todo list, I grabbed each of these resources and placed them in a new project called "Learn about microservices."

I can't stress enough how awesome it is to have a common inbox for collecting all of the "stuff" that you encounter during the day. In just this example, I basically started with one seed: a simple note to study up on microservices, jotted down during an in-person meetup. This seed led to a quick survey of the learning resources available to me out on the web - with each of these resources also captured in my inbox and todo lists. This whole process took less than 5 minutes.

The Librarian: Building an Easy-Access Filing Cabinet for your Brain

So what happens next? When I get the time to work on my "Learn about microservices" project, I'll sit down with a cup of tea and do some focused reading. Chances are I'll come across some more links to other articles and resources - that's great! I'll use Todoist to drop the link into my inbox to look at later, so I don't get distracted during my reading time.

While I'm reading, there's a good chance that I'll find some important resources that I should save to be referenced later. This is where Evernote's web-clipper really comes in handy.

There are a few features of the web-clipper that have earned Evernote a permanent place in my self-teaching workflow:

  1. Rather than just saving a link, the web-clipper actually copies over the page into your notebook. That way, if the site you are referencing disappears or they move the link, you'll still have your copy.
  2. Evernote has apps for all sorts of platforms, so you can refer back to your captured articles right on your iPhone or Android phone.
  3. Here's the coolest feature: When you install the Evernote web-clipper extension in your browser, it will insert a cool little box on the right side of the page whenever you do a Google search - with links to your saved notes and clipped articles related to your search! This effectively integrates your normal behavior (searching for stuff on Google) with your personally curated store of reference materials! Check out the screen shot below to see what I mean:

![Justin's Todoist]({{ site.baseurl }}/assets/evernote-search.png)

How cool is that?

There are a few other "reference tools" that you could use instead of Evernote, depending on your needs - and some of them even have a similar ability to plop your reference material right into your Google searches. Kifi and Declara are two examples you might want to check out.

Go Forth and Conquer

Hopefully, this post has been useful for you as you find the best way to achieve your self-teaching goals. As I've said before, the sheer volume of valuable information out there on the web can be overwhelming. But as the old saying goes, the best way to eat an elephant is one bite at a time. Just keep seeding your inbox with interesting concepts and resources to explore - and then make sure you save the useful stuff in a way so that you can effortlessly find it when you need it. Good luck! Drop a comment below if you have questions or other recommendations for getting organized, or shoot me a message on Twitter.