My Anaconda/Jupyter Setup

This is mainly for myself so that I could rebuild my setup from scratch, but it would be nice if anyone else can benefit from it!

This is incomplete. I have class in 30 minutes. But this should have the majority of the information necessary for me to complete this post. Some of this information or these urls may even be repeated here...

https://www.anaconda.com/download/ - it will automatically detect your operating system

note: when installing python packages in anaconda, always first try to install using conda, e.g. `conda install somepackage`. if that doesn't work, try googling to find if it can be installed using conda under a different channel, e.g. "-c ericmjl" in "conda install -c ericmjl environment_kernels" (from below). Finally, if those fail, resort to "pip install somepackage".

conda create python=3.6 -n env_name
# source the environment. after sourcing that environment, all installations will go in there.
source activate env_name
conda install -c conda-forge jupyter_contrib_nbextensions
# this is automatically installed as a dependency of jupyter_contrib_nbextensions, so there's no need to install it if installing the other one
# conda install -c conda-forge jupyter_nbextensions_configurator

# ipdb (IPython debugger): from conda, on github
conda install -c conda-forge ipdb

https://anaconda.org/conda-forge/ipython-sql
conda install -c conda-forge ipython-sql

to see which environment the notebook is currently running in:
conda install -c ericmjl environment_kernels
https://stackoverflow.com/a/39070588/2821804
which links to http://stuartmumford.uk/blog/jupyter-notebook-and-conda.html

https://github.com/Cadair/jupyter_environment_kernels
http://stuartmumford.uk/blog/jupyter-notebook-and-conda.html
https://stackoverflow.com/questions/37085665/in-which-conda-environment-is-jupyter-executing

pip install environment_kernels # can this not be done with "conda install" instead?

Good example of a Jupyter Notebook: https://www.kaggle.com/ash316/novice-to-grandmaster/notebook (which analyzes survey results of people who participate on kaggle.com, where Kaggle is "The Home of Data Science & Machine Learning").

Naming Conda Environments so they can be seen from inside Jupyter Notebooks:
http://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments
ipykernel should be installed. the url explains how you can install it.
python -m ipykernel install --user --name myenv --display-name "Python 3.? (myenv)"
If you are already in a Jupyter Notebook in that environment, you can reload the page, and go to Jupyter Notebook Menubar -> Kernel -> Change Kernel -> and you should be able to see the kernel you just renamed. (even if you are already in that kernel/environment, the name won't show up until you reload that kernel.

How I learn efficiently

Have you ever felt like you should be able to remember something you recently learned, but can't? Do you forget most of the information you learned in a class within months after it ends?

Dr. Piotr Wozniak, creator of the software SuperMemo that was released by 1995, has spent much of his life experimenting to develop an algorithm that helps people learn the most information in the least amount of time while also letting people retain that information for a longer period after learning it (e.g. days/weeks/months/years).

The software works by calculating the "forgetting curve", using past information about your history of correct recall among all flash cards and each specific flash card, and then predicting when you will forget that information, in an attempt to ask you again right before you forget.

By asking right before you forget (the algorithm might have calculated the next forgetting curve within days or hours), it has been shown to help you retain the information for a longer period of time after reviewing it. Asking you right before or right as you are forgetting can create a small struggle for you to remember (while still being able to actually remember), which seems to help strengthen the memory pathways in your brain.

SuperMemo's website has a fresh look now (2018-01-20) since the last time I looked at it a couple years ago, which could indicate that their software has also improved, but the SuperMemo software is known for being complicated and Windows-only. There is a more modern software that I use, which is based on SuperMemo and one of its older algorithms that is publicly posted, called Anki.

I used Anki to get As in my college courses in Calculus, Calculus II, and Linear Algebra. This list would probably be longer if I knew about Anki earlier in my life.

However, flash cards should not be your only source of studying and learning. If you try to memorize information that you would otherwise never use in your life, you may be able to consistently answer those flash cards correctly, but you may not be able to remember that information outside of flash card sessions. It also helps to memorize information that is related to information you already know. Dr. Piotr Wozniak wrote an article with helpful information on rules of learning and flashcard-making called Effective learning: Twenty rules of formulating knowledge.

Gary Wolf wrote an in-depth article on WIRED about Dr. Piotr Wozniak and his development of SuperMemo called Want to Remember Everything You'll Ever Learn? Surrender to This Algorithm.

Anki's documentation has an Introduction section that explains its purpose and differences compared to SuperMemo.

SuperMemo's main website is https://www.supermemo.com/en.

You may also be able to improve your memorization skills by using Harry Lorayne's methods in The Memory Book: The Classic Guide to Improving Your Memory at Work, at School, and at Play (1996). It is easier to remember information that is related to things we already know, for example, if you already know about history, and you are presented with a new piece of information about history that relates to your existing knowledge of history, then it will be easier to remember because it fits in and makes sense. Lorayne's methods basically allow you to create associations between completely unrelated information using a variety of systems, such as turning numbers into words, and memorizing lists of items by creating ridiculous mental images that tie those items together (for an example and explanation of some memory systems/techniques, see https://www.memory-improvement-tips.com/memory-association.html). While I still remember all of the number-sound associations for turning numbers into words from that book, I have experienced difficulty in forming a habit to use his techniques on a daily basis. This stuff should probably be taught in elementary/middle/high school to help form long-lasting habits that will stick with people for life. He also has newer books that I have not reviewed, which could be even better than the one I listed.

In Anki, I use reversible cards, and the type-answer method.

Type the answer and writing the answer on a whiteboard (or with a tablet computer): To get the type-answer method, follow this reddit answer (there's a reddit-anki community). Writing out the answer improves my ability to gauge how well I knew the answer. If I just try to think of the answer in my head and then reveal the back of the card, it's easier to fool myself into thinking I knew the answer when really maybe I was thinking between two different answers, or I didn't fully produce the answer in my head. By typing the answer or writing it on a whiteboard, I can't fool myself into thinking I knew the answer when I didn't, because my response and the true answer are clearly there for comparison (the typed answer doesn't have to be exact if you're not trying to memorize a poem or a legal definition, because then you're memorizing verbatim words instead of ideas). However, I have found it interesting when it comes to certain mathematical definitions, that I might remember the concept but forget a detail, for example wither i=1,2,3,...,k or i=1,2,3,...,infinity. At that point, it's up for you to decide based on context whether to penalize yourself for forgetting that detail (like you'll miss points if you were to get wrong on an upcoming midterm) vs letting it go (you're just trying to get a good overview of the material).

Reversible cards: To get reversible cards, follow part 2 of this tutorial. Not all cards make sense to have a reverse, for example, "how many bits are there in the key of the DES encryption algorithm?" should have the answer "64", but the reverse shouldn't say "what is 64?", because 64 can be many things besides the number of bits in the DES key, like 2^6, a Beatles song, the name of a magazine, etc (wikipedia). However, if you really want a mental cue for that reverse, it could say "what is 64 regarding the DES encryption algorithm?" as long as 64 doesn't appear anywhere else in the DES algorithm.

Data I Track

Data I track (this is not a complete list, just the things I managed to find/remember so far). I hope to eventually have at least one blog post regarding each one of these. If I get enough blog posts for a single topic, I'll turn it into a category/tag/label and link to it. This post is pinned.

San Jose Parking Garage Data (https://violeteldridge.com/2018/05/26/san-jose-parking-garage-data/) (since 2018-05-22)

  • Health:
    • Basis Peak wristwatch (need to dig around to find out when I got it, but it was totally recalled and service officially shut down on 2016-12-31)
    • Fitbit Charge 2 wristwatch (need to find when I started using it). By creating an Open Humans account and connecting your Fitbit account, you can download/view your data for free online in a Jupyter Notebook. It's not necessary to write any code, because they already wrote the code. All you have to do is run the cells by clicking a cell and typing either shift-return or option-return (or meta-return depending on your operating system). https://www.openhumans.org/activity/fitbit-connection/
    • Symple iPhone health tracker
    • Weight using a Garmin Index Smart Scale (since December 2016)
    • Sleep time using Azumio's iPhone "Sleep Time" app, but unfortunately I don't know of any way to get my raw data out of it - they just let you download the times when you hit the start/stop sleep buttons (since August 2013), which would just be a repeat of what I already have in aTimeLogger
    • Apple Health (just for the functionality of downloading the data from all the applications that integrate with it)
  • Time:
    • aTimeLogger 2 (2013-01 to 2013-03, then consistently from 2013-10 to present (2018-01))
    • RescueTime (since July 2015)
    • Transportation
      • MileLogger (automatic drive capturing) (since December 2017)
      • Waze, but they don't let you download your data and only let you see a very shallow view, limited to the last 30-or-so days, and I don't know when I started because they don't let you download any of your data
  • Money: YNAB (since October 2013); the old iPhone version (which used to be the only version but got renamed to "Classic" when they introduced a new app) that I still have stopped syncing properly a few iOS updates ago. I didn't update because according to reviews, there are still problems with it. Their new app is a subscription model, but I haven't used the budgeting features since 2013 anyways (I only use it for tracking transactions) so I don't see enough value in paying a subscription for this when I could get functionality equivalent to what I use from the application by using an Excel spreadsheet. So now I only use the desktop version.
  • Weather:
  • Computer (since January 2016):
    • every second, current application and title of current window (since December 2017) (see blog post Personal RescueTime (Computer activity time tracker))
    • every hour
      • `uptime`
      • firefox processes: `ps aux | grep -i [f]irefox`
      • memory usage of windowserver (windowing system on mac computers)
    • every 2.5 hours: network and location: public IP, gps coordinates, output of `/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I` and `ifconfig`
    • every 8 hours: disk usage: `df -h .`
    • every day: disk size of iPhone backup: `echo $(du -s ~/Library/Application\ Support/MobileSync/) $(du -hs ~/Library/Application\ Support/MobileSync/)
    • battery status `pmset -g batt`, every hour 2017-07-26--2018-03-04, every 10 minutes 2018-03-05-present
  • Other:
    • Nomie 2 https://nomie.io/ (makes it easy to manually track many little things) (since January 2018)
    • Family AT&T Data Usage (since January 2017)
    • Things I track (i.e. this blog post, since 2018-01-15)

I'm interested in figuring out whether it's possible for someone with a Garmin fitness tracker that tracks location data to be able to download the raw location data. I would expect it to be a list of latitude/longitude with their respective timestamps, every few seconds or minute depending on the situation.
Another possibility is also to use a USB data tracker like https://www.amazon.com/Elitech-RC-5-Temperature-Recorder-Accuracy/dp/B00MQSCZF2/.
And a weather station like https://www.amazon.com/Ambient-Weather-WS-2902-Professional-Monitoring/dp/B01N5TEHLI; it says it's $30.01 off, down from $180.00 to $149.99 on 2018-01-21. (see price history at CamelCamelCamel.com)

Keep time while reading a book

I wrote this little script to help me keep on track with my reading. I needed to read 2 chapters of each of 3 different books today, and I wanted to assign one hour to each chapter, but in order to keep on track, I wanted to see what page I should be on at each moment, so that's why I wrote this script.

It's in python. Just tell it the amount of time or the time you want to finish (e.g. '1hour' or '2pm', anything that is supported by the gnu date `-d` argument), the page you're starting on, and the page you're finishing on, and it will continuously tell you where you should be in your reading.

https://github.com/mica5/reading-progress

Why I sleep how/when I do

People are often surprised by my sleep schedule, especially since it's considered to be the "opposite" of people in a similar situation as my own (Software Engineer, University student).

I need about 8 hours of sleep each night to feel my best each day. I don't set an alarm unless I go to bed late and need to get up early and haven't been getting enough sleep for some days in a row.

I'm more likely to go to sleep at regular time on weekend nights (e.g. Friday and Saturday) if I'm doing something I don't particularly enjoy (e.g. reading or writing for a class I don't want to take but have to take), rather than doing something I enjoy such as reading or doing work for a class I do enjoy.

By going to bed 9 hours before I need to be up, even if I'm in bed 20 minutes late, that still gives me an extra 40 minutes for sleep. Allowing myself an extra 20-60 minutes for sleep also allows my body to get extra sleep if it needs it, which I sometimes do.

Since I'm usually out of bed before 4:40am, I can do my morning routine and be out of the house by 5:40am. In my experience, the number of cars on the road sharply increases starting 5 minutes before 6am, so leaving at 5:40am usually means there's still empty lanes on the freeway.

By getting to my destination approximately 3 hours before anyone else does, that's nearly 3 hours of focused, uninterrupted work time.

Maybe someday I'll support all of this with data and visualizations. I have a lot of data for this, I just need some time to analyze it.

Miscellaneous

2018-01-01 Here are some ideas that don't have enough content to be individual blog posts, so I'll write here until they become developed enough to warrant their own posts.

<h1>
Python</h1>
2018-01-01 "One of the curses of Python is that it is such an easy and expressive language that developers often find it easier to re-invent and re-implement functionality that exists in other packages than work to integrate code from other packages." <a href="http://www.aosabook.org/en/matplotlib.html">http://www.aosabook.org/en/matplotlib.html</a> (my attention was brought to this post as a result of course 2/5 "<a href="https://www.coursera.org/learn/python-plotting">Python Plotting</a>" in the Coursera specialization "<a href="https://www.coursera.org/specializations/data-science-python">Applied Data Science with Python</a>")

2018-01-01 Compared to typed languages, Python will let you do anything, which allows inexperienced programmers (and experienced ones) to cause run-time errors in user-facing systems, but also provides an expressive language in which a masterpiece can be written.

<h1>
Internet and search engines</h1>
2018-01-01 One of the great things about the internet is that many people can say the same thing in many different ways. Consider the concept of pointers from the C language - you can pass around references. Just remember where the original is and make pointers back. On the internet, there exist many ideas, but many people use different language to search for or ask about the same ideas. Many forum or stackoverflow answers show "this is a duplicate, see [this other url here with the answer]". But the benefit of that is that there are many pointers laying around the internet that allow many users who search using different language and terminology to find the same answer by many of the pointers that are laying around. I think that could make it much easier for AI-/NLP- based search engines to connect content together with many variations of the same language.

Personal RescueTime (Computer activity time tracker)

I have been a paid user of RescueTime every month since June 2015, but since I love collecting and analyzing data, I decided to replicate the functionality and keep track of the data in my own local database. Using applescript and postgresql, I've been recording computer usage (application name and window title) since "2017-12-08 20:06:00.670021". I was just completing a survey on Coursera, which is asking "How many hours per week do you plan to spend working on this course?", and because I started recording data on December 8th and started taking the class on the 6th, I knew I had data for the majority of my coursera time. This doesn't account for course-related material that wasn't on coursera's website or didn't have "Coursera" in the title, and now I'm also realizing I should try to grab a url of the current application I'm in, if I can (unfortunately, some quick google searches don't return anything obvious for firefox, although chrome does have it built in..).


with base as (
    select
            *
            , datetime-lag(datetime) over (partition by true order by datetime) as time_between
            , extract(year from datetime)::text||'-'||extract(week from datetime)::text as week
        from user_activities
        where (window_name ilike '%coursera%')
            or (application ilike '%firefox%' and (
                window_name ilike '%week+%'
                or window_name ilike '%assignment%'))
        order by datetime
    ) , filtered as (
        select *
        from base
        where time_between
week   | time_spent_on_coursera
---------+------------------------
 2017-49 | 00:48:47.559574
 2017-50 | 09:45:30.929945
 2017-51 | 04:02:11.548172
(3 rows)

Note that for week 2017-50, that's 9 hours and 45 minutes. If anyone is interested, I could come back and explain this a bit. It's not completely accurate, because Coursera refers to some materials that don't have "coursera" in the title, and I could have been busy doing something non-coursera-related for less than 15 minutes, which would make it less accurate.

I recommend RescueTime, as it is an awesome tool for finding out a quantitative truth about where you spend your time on the computer: https://www.rescuetime.com

Useful bash commands

Remove the last 1 line of output:

printf "1\n2\n3\n4\n5\n" | head -n-1
1
2
3
4

without " | head -n-1", the printf command yeilds:

printf "1\n2\n3\n4\n5\n"
1
2
3
4
5

Remove the first 1 line of output:

printf "1\n2\n3\n4\n5\n" | tail -n+2
2
3
4
5

You can change the number in any command you like. tail -n+1 doesn't seem to remove any lines, so add 1 to whatever number of lines you want to remove.

Coffee Notes

Forager (in San Jose) opened late February 2017.

Devout Coffee brand, San Sebastian beans - from Guatemala, natural process.


Coffee Classes at Chocolate Fish Coffee Roasters in Sacramento at 4749 Folsom Boulevard, East Sacramento.

Coffee books/authors:

2017-12-18
In coffees I look for fruity flavors. Usually when coffee people say they taste specific fruit flavors like nectarine, strawberries, etc, I don't usually taste it. The only kind-of fruit flavor I have ever really been able to taste in a coffee was blueberries in a coffee from Ethiopia (unfortunately, I don't remember which one specifically).

The best coffee I've ever had was Guji Highland by Devout, but unfortunately, its season is over. I don't know whether it will have another season in the future. Forager Tasting Room and Eatery in Downtown San Jose uses their beans, but I'm not sure if they sell whole beans. Another really good coffee I'm having lately is San Sebastian, also by Devout. Other good coffee bean producers are Verve (Santa Cruz) and Temple (Sacramento). I haven't enjoyed any coffees from Allegro, which is too bad, because Whole Foods carries a lot of their beans. Starbucks coffees mostly taste like burnt coal, and Philz Coffee "Lighter Blend" "Sooo Good" also tastes burnt.