Python, Pandas and Matplotlib 1.1 : Analysis and Storage of Large Time-Series Data from the BTC-e API

First off I would like to thank Wes McKinney for his great work compiling together the pandas library over the past years. His book Python for Data Analysis (~$25 on Amazon) is fantastic and a great addition to your home library.

This post will be an introduction to help you better understand the resources and tools available to you in python and the pandas library. If you have been following the MATLAB data capture tutorials you will transition into these python tutorials fairly well.

Our main objective is to capture the JSON data from the BTC-e API and continually store and analyze it to our liking. To accomplish this we will be utilizing Data Frames (Not to be confused with the Data Frames found in “R”) and Hierarchical Indexing.

Now, I started off hacking away in Python 2.7 with the following Libraries/Packages/IDE:

  • IPython
  • Pandas
  • Matplotlib
  • Spyder
  • Numpy
  • SQLite3

I recommend downloading the following Python(x,y) Package.

It will provide you with the following:

  • IPython 2.4.1-10
  • Pandas 0.16.2-15
  • Matplotlib 1.4.3-7
  • Spyder 2.3.5.2-17
    • If you’re comfortable working in the MATLAB environment the Spyder IDE will feel like your home away from home.
  • Numpy 1.9.2-8

Our main working environments will be the Spyder IDE along with instances of IPython Notebooks. I have found that this is a great combination for testing out new strategies.

Once you have accumulated the necessary resources for this tutorial continue on to Part 1.2. If you have any trouble feel free to leave comments or contact embeddedthought directly.

PART 2.1: Plotting BTC-e API Data in Matlab

This next tutorial will guide you through how to get nice, clean, and readable plots out of Matlab. The data that you have fetched with the API and have stored, will be plotted so you can have a view of what everything looks like while double checking that your code is working correctly.

Please note that there are many ways to do what I’m about to show you, this is one way that i find simple and easy to follow.

From this point, we will assume that you have completed the Capturing Data for each pair that you are monitoring. Your code may look similar to mine if you followed the Part 2: Matlab Data Collection. I have added a couple lines of code at the end of each fetching round for user feed back in the console output. This is not necessary but I like it for Debugging and its nice to see where the code is if matlab is working.

Ticker_Grabber Loop

Note: My current if statement will allow the Ticker_Grabber to run 40 rounds, there are some issues with the workspace data getting updated in between rounds. We are working on forcing the data to update in the workspace for live plots. (Simulink may be the answer or we might integrate matlab with python.)

Anyways, as you can see the Ticker_Grabber is fetching the order books history for the following pairs: BTC-USD, LTC-USD, LTC-BTC, and PPC-USD. You can add the code for all the other pairs on BTC-e to compare all markets. I will keep it simple for this example but remember that this can be scaled up for all order pairs.

We want to make note of the .mat files that we are creating for each of the pairs. In my case in the btc_usd_orderBook.m will output two .mat files. One for Bidding and one for Asking. If you are not getting two .mat files for each order pair, please visit the Part 2 – Data Capture for more info. For each order pair, you should have two .mat files to work with. In my case I will have a total of 8 .mat files to plot.

For my purposes, I will be plotting the price and volume separate for each asking and bidding strings. You can however get asking and bidding plots on one graph for comparison, but may not be super clear what is going on. Also, keeping the data separate will allow us to use some data manipulation functions later.

Ticker_Grabber Plot code 1Using the figure() command we can group the graphs for each pair and combine them with the subplot to make a nice 2×2 graph. This line of code above is fetching all the values for the asking price of BTC_USD and plotting them against the time stamp of the BTC-e server.

Ticker_Grabber Plot code 2

Subplot 2 is very similar to the code of the asking price, but now we are fetching the volume and plotting those values also against the server time. Be sure to make clear labels for your graphs, that way you can really keep track of what is being displayed. Moving right along.

Ticker_Grabber Plot code 3

By now you should be getting the hang of what is going on here. Pay attention to the numbering of the subplot function. There is more info on the how the subplot works if you type ‘doc subplot’ in the workspace. For what we are doing this will work just fine. If you have done everything correctly your plots should come out as shown below. You can tweak the color and line art with a little more code. The next pair for plotting will start with the figure(2) and repeat the steps above.

Ticker_Grabber Plot

I hope you have found this post interesting and helpful! If you need any assistance please feel free to contact us. Join us in the Mumble server for live trading advice or bot building help!

Thanks All!

Mr.Wong

EMBEDDED THOUGHT Mumble Chat Server!

Look! Embedded Thought now has its own dedicated chat room for trading, trending discussions, bot building help, cryptocoin talk, and anything really! Catch some of the devs in the mumble server that are working on the trading bots or someone like you just trading crypto. Below is the tutorial on how to get inside.

First you will want to head over to Mumble.info <– That is the wiki page, download the correct client for the OS you are running, and install.

Mumble download

After you install the mumble client you will want to create a username and connect to the following server. Note: if you have a btc-e username you can use that and it will make the verification process faster. This process is used to filter out the trolls that may enter the server.

Mumble Login

Add a server to connect to and fill in the ip address below. Connect: 107.170.70.76

If you need Real Pro trading advice be sure to get your account verified and join the Traders Corner!

Preview of What’s to Come: Neural Network Trading on BTC-e

dsfg

Sorry I have been taking so long to post with updated information.
With the market exploding like it is I have been using my time
to continually refine and develop the bot. This project will be expanding
into a collaboration between myself and other developers. Perhaps it might be
best to wait and provide you all with a walk-through of the finalized bot
since it is ever evolving.

Thank you for reading!
-saeveritt

PART 2: MATLAB, NEURAL NETWORKS AND THE BTC-E TRADING API. ( Data collection)

Considerations before starting Part 2:

I will not be going in to full detail on every step and I will be assuming you have a basic understanding of MATLAB. Feel free to leave a reply if you have any questions.

Example of recording order-book movement through time:

After Part 2 we will have the functions necessary to start capturing useful data.

datagathering

Capturing Data:

Initially, we want to have our data constructed in a way that will be easy for us to use and manage.

To start, I have created a main function to initiate when we want to record the status of the order-books and the corresponding price fluctuation. retreiveorders

There are 3 sub-functions that pertain to the three pairs I am collecting order-book data on.

We will get started by creating those first.

ltcbtc_orderbookMovement does the following:

orderbookMovement

Every function underlined in blue you will find in the BTC-e Trading API link I referred to in part one. I underlined urlread2 to emphasize that it needs to be in the same folder as all of the BTC-e API files and functions (Which should be the directory you’re working from).

On the last line we are simply converting the incoming information from a structure  to a data-set.

Continuing ltcbtc_orderbookMovement :

orderbookMovement2

During the for loop we are using strcmp to separate the two types of orders, ‘ask’ and ‘bid’.

We are allocating the ask orders and bid order to individual output cell arrays. After doing so, we then use cellfun to remove all empty cells in the array. This is important because we will be concatenating this data with whatever it was previously. If it is your first time running the function and you do not have variables set up, don’t worry. The following rest of this function will take care of that.

Continuing ltcbtc_orderbookMovement :

orderbookMovement3

Now that you have an idea on how we want the incoming information structured, you can go ahead and create two more of these function for BTC/USD and LTC/USD. Rename the variables however you please but make sure to check line 34 and change “orderbook=dataset.ltc_btc;” to which ever pair you’re waiting to gather data from.

This process can also be repeated to create the function priceHistory() referenced to at the beginning of this entry. Just change the necessary information to pull from the function realtime_ticker() referenced to in Part 1.

Now that data gathering is out of the way. We can get to the fun stuff!

Any and all tips are greatly appreciated.
BTC:16v9zKaAGiRiqHrcr1BXzYxiSjXqGuFaTA 
LTC:LfQKBDrbPcEM6ioKRxCx8iXjrAJx4Yvake

Part 1: MATLAB, Neural Networks and the BTC-e trading API.

Let’s get started:

This might contain around 5 parts, so here we go..

exampleANN

I’m in the beginning process of developing a trading-bot using Matlab. Right now the plan is to save data concerning what the order-books look like before and after a dump. This data will be used for training the Neural Network through identification using pattern recognition.

exampleANN2

Multiple indicators will be incorporated as input data also, but we will save that part for later.

The bot needs to have an understanding of order-book trends and the following implications to the price.

A Brief Walk-through on Data Retrieval:

The data is sourced from the BTC-e Trading API

(You will find the need to add urlread2 to your project folder in MATLAB).

urlread2 can be found at the following:

http://www.mathworks.com/matlabcentral/fileexchange/35693-urlread2

Trading API for MATLAB at the following:

http://www.mathworks.com/matlabcentral/fileexchange/44890-btc-e-trade-api

Here is a list of all possible requests:

% response = GetInfo()
% response = TransHistory()
% response = TradeHistory(‘count’,2)
% response = ActiveOrders()
% response = Trade(‘pair’,’btc_usd’,’type’,’buy’,’rate’,200,’amount’,1)
% response = CancelOrder(‘order_id’,651389)
% ticker_output = realtime_ticker(‘btc_usd’);

The default amount is set to only retrieve 150 listings in the orderbooks, whether they are bidding or asking.

We definitely want to get a larger data set than 150 listings. The reason being is that we are going to be creating an Artificial Neural Network and training it with this data. The larger the data set, the better life we all live. The maximum limit for the GET method from BTC-e is 2000 listings. So we’ll go ahead and use that.

To change that we make our own function to change the get method limit to ‘2000’:

example

You don’t have to do it like this, but this is just one way.

I’ve been recently recording the history of order-book movement for the  LTC/BTC pair.

dump

While resting at a somewhat steady exchange rate of 0.0076 for a few days, we recently witnessed a dump to around 0.0073. Having said that, take a look at the order-book movement through the dump. We will be collecting this data to train the bot to recognize a dump before it happens.

Resting:

ltcbtc2

Let the dump begin:

dumpltcbtc dump

Bid Volume starts increasing:

recover

The Recovery stage:

buyback

I will have a break down on gathering and allocating the data to the neural network in part 2.

Thanks for reading!

Any and all tips are greatly appreciated.
BTC:16v9zKaAGiRiqHrcr1BXzYxiSjXqGuFaTA 
LTC:LfQKBDrbPcEM6ioKRxCx8iXjrAJx4Yvake