Adding an RSS feed to this site

Here is the link to the RSS feed of this blog.

Thanks to the instructions on Joel Glovier’s blog post.

Python Library and scripts for downloading ERA-Interim Data

{
    "url"   : "https://api.ecmwf.int/v1",
    "key"   : "(...)",
    "email" : "(...)"
}

#!/usr/bin/env python
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()

param_u, param_v, param_t = "131.128", "132.128", "130.128"

for param_string, param in zip(["_u", "_v", "_t"],
                               [param_u, param_v, param_t]):

    server.retrieve({
        "class": "ei",
        "dataset": "interim",
        "date": "2017-07-01/to/2017-07-31",
        "expver": "1",
        "grid": "1.5/1.5",
        "levelist": "1/2/3/5/7/10/20/30/50/70/100/125/150/175/200/225/250/300/350/400/450/500/550/600/650/700/750/775/800/825/850/875/900/925/950/975/1000",
        "levtype": "pl",
        "param": param,
        "step": "0",
        "stream": "oper",
        "format": "netcdf",
        "time": "00:00:00/06:00:00/12:00:00/18:00:00",
        "type": "an",
        "target": "2017-07-01/to/2017-07-31" + param_string + ".nc",
    })

Resources on deep learning

I have been searching for solutions how to use Recurrent Neural Networks for text classifications. Here are some useful resources I’ve found:

• Open Source Deep Learning Server has a library of pre-trained neural nets.
• An article on Analytics Vidhya discuss about transfer learning & The art of using pre-trained models in deep learning. They also have an article about word embeddings.
• Keras has a list of deep learning models with pre-trained weights.
• Hyperopt is a python library for Distributed Asynchronous Hyperparameter Optimization.
• Hands-On Machine Learning with Scikit-Learn and TensorFlow has discussion on deep learning from Ch.10 onward.
• Kaggle forum has a beginner tutorial of using RNN to classify toxic comments on wikipedia editing page.
  (To be updated.)

Three co-authored papers submitted

The publication page has been updated with 3 submitted manuscripts.

Updates on Feb 9, 2018: The manuscript “Role of Finite-Amplitude Rossby Waves and Nonconservative Processes in Downward Migration of Extratropical Flow Anomalies” has been accepted by Journal of Atmospheric Sciences.

The subroutine wrapper.qgpv_eqlat_lwa_ncforce for computing effective diffusivity, which quantifies the damping on wave transiences by irreversible mixing in the stratosphere during a stratospheric sudden warming event, can be found in my python package.

Setting up algs4 on Linux

I am interested in going through the exercise from Princeton University’s Algorithm course. I found someone wrote a handy bash script to set up the environment on Mac OS/Linux:

https://gist.github.com/JIghtuse/021604bee56bddab6173c919da7dd2ad

My python library updated to v0.2.0!

I have updated my python library hn2016_falwa to v0.2.0 (see release note! Now it includes functions to compute the contribution of non-conservative forces to wave activity.

Moreover, the documentation page generated with Sphinx is now hosted on readthedocs.org! Check it out!

A side note: somehow I made multiple commits to remedy mistake. The git commands to squash the (3, for example) commits are:

git rebase -i origin/master~3 master
git push origin +master

Wrapping Fortrain Codes in Python

To start with, the documentation in Numpy explains how we can wrap fortran code in python using f2py.

You need a fortran compiler to run f2py. I’ve found a pre-compiled version of GCC readily installed on Mac OS X.

(To be continued)

Compiling tensorflow on Mac with SSE, AVX, FMA etc.

(Ideally, I shall run tensorflow somewhere else rather than on my MacBook.)

When I install keras with Anaconda on my Mac OS X, with tensorflow as the backend, the following warning comes up when running the sample script:

I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA

To use those instructions (SSE4.1 SSE4.2 AVX AVX2 FMA), tensorflow has to be compiled from source. The instructions are available here. Using the following command to build the source:

bazel build -c opt --copt=-march=native --copt=-mfpmath=both --config=cuda -k //tensorflow/tools/pip_package:build_pip_package

I got the following error:

Problem with java installation: couldn't find/access rt.jar in /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home

It happens that rt.jar is not present in Java 9. To solve it, install a version of Java 8:

brew cask install caskroom/versions/java8

and then specify the path of Java 8 (Change the version number ‘1.8.0_162’ to that of the version you installed):

export JAVA_HOME=$(/usr/libexec/java_home -v 1.8.0_162)

Afterwards, I try to build tensorflow from source again, and it successfully includes the instructions above.

Setting up MySQL / access with python on Mac OS

Today I wanted to setup an automated kickstarter scraper on Python Anywhere but realized that only MySQL is freely supported there (while I’ve been using PostgreSQL). So, a time to switch?

Here is how I install MySQL on my Mac and have it accessed with SQLAlchemy:

1. Download and install MySQL from Oracle.
2. Go to System Preferences to start the MySQL server.
3. Navigate to the bin directory and login with the temporary password shown at the end of the installation:

   cd /usr/local/mysql/bin
   ./mysql -u root -p
   

4. Create another set of username and password that you use instead of root.

   CREATE USER username@localhost IDENTIFIED BY 'password'
   

5. I have installed pymysql and sqlalchemy in Python to access the MySQL database. To access the database:

from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database

# dbname is the database name
# user_id and user_password are what you put in above

engine = create_engine("mysql+pymysql://%s:%s@localhost:3306/%s"
                       %(user_id,user_password,dbname),echo=False)
if not database_exists(engine.url): 
    create_database(engine.url)			# Create database if it doesn't exist.
    
con = engine.connect() # Connect to the MySQL engine
table_name = 'new_table'
command = "DROP TABLE IF EXISTS new_table;" # Drop if such table exist
con.execute(command)

Executing SQL commands is rather easy by using:

con.execute(command)

Enjoy! :)

Software Engineering Project Note-taking

Learnt a lot from peers today! :D Here are quick notes on packages they have used for their software engineering project:

Docker Hub: You store your docker images there.

DigitalOcean: The docker images are pulled there, together with the images.

Sphinx tutorial: Useful instructions how to set Python documentation up

pydot and graphviz: Draw graphs of objects and arrows

Free Online Deep Learning Resources

These are resources related to deep learning from conversation with friends:

Deep Learning Paper Reading Roadmap

Neural Networks and Deep Learning

Books introduced in newsletter of Data Science Central:

Deep Learning Book

Learning html, CSS, javascripts and jinja2

On my way building a webapp with python and Flask, I need to include input options to make the app interactive. Here are some great sites I’ve learnt things from:

• w3schools.com
• jQuery user interface
• Jinja2
• CSS-Tricks
• ShapeBootstrap

More updates later.

Setting up ubuntu on AWS

Solution for the error libSM.so.6: cannot open shared object file.

Python packages for Sentiment Analysis

On top of utilities in nltk.sentiment, there are also some packages for training and combining classifiers:

• NLTK-trainer by Jacob Perkins, accompanied by the NLTK-cookbook he wrote.
• Pattern.en developed by the Computational Linguistics Research Group
• Data sets for training a sentiment classifier: Movie Review Data

(More to be updated)

Published a paper on Local Wave Activity Budget!

I’ve published a new paper on Geophysical Research Letters!

Climate dynamicists have derived a conservation relation based on small-amplitude wave assumption for wave activity (A) that describes evolution of Rossby wave packets:

However, only the wave activity flux vector on the RHS has been used to diagnose realistic climate data. A is ill-defined when wave amplitude is large (i.e. ‘of finite-amplitude’). In Huang & Nakamura (2016), we introduced a new theory of wave activity applicable to large waves. We thus can obtain a well-defined A even from real data. This is the first piece of work that compare LHS and RHS of the conservative part of equation above for reanalysis data. This advance allows us to estimate the overall non-conservative contribution (natural/human-induced forcings) to the observed flow.

Major results include:

(1) Our estimation of transient wave activity (top panel) is consistent with previous work (bottom panel, assuming small-amplitude waves) and is better behaving.

(2) We can break down the local wave activity budget at seasonal time-scale.

(3) We can also break down the budget in synoptic time-scale with the use of co-spectral analysis.

Switching to Jekyll

I’m switching from a traditional html webpage builder to Jekyll user! Hope to update more often!

I set up Jekyll in my Mac OS X with homebrew, rbenv and RubyGems.

To see how to set jekyll up on Windows, refer to my older post.

Installation of Jekyll on Windows 10

Installing Python Library for downloading ERA-Interim Data

Update: ECMWF API Clients on pip and conda

The ECMWF API Python Client is now available on pypi and anaconda.
The Climate Corporation has distributed the ECMWF API Python Client on pypi. Now it can be installed via:

pip install ecmwf-api-client

If you are using anaconda, OS X/linux users can install that via

conda install -c bioconda ecmwfapi