Python, Pandas and Matplotlib 1.2 : Data frames and HDF5 storage

After running into Unicode indexing errors in Python 2.7, a solution was found by installing the Anaconda Python 3.4 distribution. This will also set up your System Environment Variables.

This will be an example that shows you an easy way to capture the ticker data from btc-e.com for all currency pairs. HDF5, also known as Hierarchical Data Format, and Pandas DataFrame will demonstrate to be powerful for quick analysis on large data-sets.

After you have installed the distribution, open the Anaconda Launcher:

Anaconda_Launcher

Launch an ipython-notebook. A new window should open directed towards localhost:8888/tree or  similar. Under files go to the new button and select “Python 3” under the notebooks section.

To begin, import pandas and the time library by doing the following:

1

Also, set a variable named url  to be the BTC-e api ticker url. In this instance we are concerned with capturing data for each pair available on BTC-e.

Next will be the soul of our data capturing, pd.read_json() .

2

I recommend setting the delay to time.sleep(2). If you read the API information on BTC-e, it states that it refreshes at 0.5Hz.

We simply read the json from the api url into a data-frame named frame and transpose it. Next, we define keylist as the values of the index (pair names for ex. ‘btc_usd’ to be used as the data-frame key); of which is accomplished by setting keylist = frame.index.values.

In the for loop we then separate out each pair with its index set to the series ‘updated’. We then use store.append(‘key’,data) to append each key and data-set to pd.HDFStore(‘filename.h5’).

Once you stop the loop, you can view the data with the follow:

pd.read_hdf(‘filename.h5′,’key’)

Type any pair combination on BTC-e as the key such as ‘ltc_usd’ or ‘ppc_usd’ to get the logged data.

3

Here’s a quick example of plotting the data.

cover

Next we will learn about real time plotting using matplotlib!  

if you have any questions or concerns feel free to ask away!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s