Getting price feed for development

[Saver0]

I put this together quickly to download the entire candle history from Oanda through their API. The data seems to be very good but there are small gaps. You can also modify this script to update the database continuously using setTimeout() so that it would get the completed bar for each period.

You will need to have Node.js setup properly and have the following modules installed

var _ = require("underscore");
var async = require("async");
var moment = require("moment");
var prettyjson = require("prettyjson");
var fs = require('fs');
var mysql = require('mysql');

You you will need an Oanda Demo account and an API key to get the data.

• This topic was modified 11 years, 7 months ago by Saver0.

Attachments:

You must be logged in to view attached files.

Focus, Patience, Determination & Order in chaos

[Saver0]

Another good free data source is provided by Integral:

Streaming Market Data

Focus, Patience, Determination & Order in chaos

[planB]

This one is really easy to use and leeches Dukascopys feed, which is quite good as well.

edit:
url not shown (bug?) so again in plain text:

Home

• This reply was modified 11 years, 7 months ago by planB.
• This reply was modified 11 years, 7 months ago by planB.

[Saver0]

Yup, I have used this as well. It was very good. Most pairs went back to 2008 I believe.

I changed the title of the thread to be about data feeds in general. If anybody knows of any other high quality historical data feeds that are either free or pretty affordable, please let us know.

Data can be so expensive and I need good historical data. Oanda seems to have gaps. I haven’t check Dukas for gaps but I’m thinking they probably do too. Hence why some of the feeds cost thousands of dollars

Focus, Patience, Determination & Order in chaos

[Zelo]

Hi Saver0,

If you use JForex flatform, you can filter Weekend and Flat data.
You can automate by choosing multipairs.

[sniperguy]

Hi,
I have tested software : tickstory (http://tickstory.com) & tick data downloader (http://www.strategyquant.com/tickdatadownloader/) to download data from Ducascopy.

There is finish data source for download : TrueFx (just know this from saver0 above: http://truefx.com), histdata (http://www.histdata.com/download-free-forex-data/)

I think it is a good idea to try from other source (Oanda), i will test the Js file “get_oanda_candles.js” and see the result.

Other source for preparing tick data : http://eareview.net/tick-data/convert-tick-data

Thanks

• This reply was modified 11 years, 7 months ago by sniperguy.

[Saver0]

cool! yea I’m not sure how good the Dukascopy data is. I need to test it out as well. Because for things like TZs or neural networks, we got to make sure its an error free data source.

Focus, Patience, Determination & Order in chaos

[sniperguy]

Hi mates,
I think we can NOT use historical data from histdata (http://www.histdata.com/download-free-forex-data/) , because it has so many gaps.
I check it bar per bar, and found some gaps. But this way of approach will need time
Any body know is there a good tool we can use to check for this gap ?
I have downloaded data from TrueFx too, and will try to check it later. So, it will save time if we have a tool to check the gap fast, to check data from TrueFx, Ducascopy, etc …
Thanks in advance

[Anonymous]

Hi everyone,

I am a Japanese and I have been using Alpari data for 1-min and daily data.

http://www.alpari.jp/academy/historical/

1-min data is from 2011/06/07 and daily data is from 2011/01/03 (sorry it is japanese website).

To download the zip file, just click １分足(for 1-min) or 日足(for daily).

I am not very sure its accuracy so please be careful.

Thank you,

[Zelo]

Bad news. Dukascopy have gaps too.
I do a quick check on AUDUSD
766 gaps from 2013.1.1 to now on M5
98 gaps from 2013.1.1 to now on 15
42 gaps from 2010.01.01 to now on H4

We have to filter them ourself

[sniperguy]

@Zelo, thanks for that information my friend…

@all friends,

a) We just need to start Mysql server from XAMPP, without need to start Apache, because the JS script will wait on a specified port and do a connection directly to Mysql without using Apache. Is that right ?

b) The fractal_server.Js and fractal_arrow.mq4 are great. The connection to Mysql running well, i can see red/green arrow on my MT4 charts. Still have not seen the prediction/calculation deeply, will do it later. Thank you saver0.

c) I have tested get_oanda_candles.js script (thanks to saver0  for this),  and in my database “candles” i got 24 tables (AUD_JPY, AUD_USD, EUR_GBP, EUR_JPY) . Is there something wrong ? i did not get other tables like EUR_USD, NZD_USD, etc ….  Any one has experience like this ?

Edit:

*) Each table has around  980,000 rows for M1 TF.
*) I saw at JS code, it will get 4 pairs at the same time. Does it mean that i must stop the JS server, then restart it again to get another 4 pairs ?

 

 

 

 

• This reply was modified 11 years, 7 months ago by sniperguy.

[smallcat]

Good  work Sniper. I think may be your internet connection is not fast enough, so it has delay.

Your idea on restart the node.Js  script will work in my opinion.  After getting all candles of all pairs from Oanda, we need to “convert” the candle from database to MT4  …  Any idea is welcome

Thanks for your contribution : planB, Zelo, JimSterk, Sniperguy … and of course thanks to saver0 for his hard work 

Edit :

Or may be we do not need to convert the candles from Mysql to MT4, but just create a demo account and download Oanda MT4, may be it is much easier … 

• This reply was modified 11 years, 7 months ago by smallcat.
• This reply was modified 11 years, 7 months ago by smallcat.

[dewachen]

Saver0 wrote:

Bad news. Dukascopy have gaps too. I do a quick check on AUDUSD 766 gaps from 2013.1.1 to now on M5 98 gaps from 2013.1.1 to now on 15 42 gaps from 2010.01.01 to now on H4 We have to filter them ourself

Good post mate, thanks.

Any idea on how we slice a big candle ? Ex: the average of last 10 candles (lets say we calculate 10 candles to get the average of them, and define as a standard candle height)  is  0.0004 & we have a big candle now with 0.0017 pips height of a bull candle. If we divide it, we get : 17/4 = 4.25

So, we will define this candle as (4) bull candles [1 1 1 1]  because  4.25 < 4.50    ** OR **  we define this candle as (5)  candles [1 1  1  1 1] because 4.25 > 4.00000000001 ?

• This reply was modified 11 years, 7 months ago by dewachen.

[Saver0]

Zelo wrote:

Bad news. Dukascopy have gaps too. I do a quick check on AUDUSD 766 gaps from 2013.1.1 to now on M5 98 gaps from 2013.1.1 to now on 15 42 gaps from 2010.01.01 to now on H4 We have to filter them ourself

I’m assuming that you ignored the Friday to Sunday gap. Also I noticed that Sunday tends to have a bit of gaps on some currencies due to lack of activity most likely. So just ignore Sunday altogether in the gap analysis.

I wouldn’t be surprised for seeing these gaps. Maybe even placed there on purpose so that our analysis would be misleading. I guess we could try to fill these gaps using different data sources. A bit complicated though.

 

dewachen wrote:

Saver0 wrote:

Bad news. Dukascopy have gaps too. I do a quick check on AUDUSD 766 gaps from 2013.1.1 to now on M5 98 gaps from 2013.1.1 to now on 15 42 gaps from 2010.01.01 to now on H4 We have to filter them ourself

Good post mate, thanks. Any idea on how we slice a big candle ? Ex: the average of last 10 candles (lets say we calculate 10 candles to get the average of them, and define as a standard candle height) is 0.0004 & we have a big candle now with 0.0017 pips height of a bull candle. If we divide it, we get : 17/4 = 4.25 So, we will define this candle as (4) bull candles [1 1 1 1] because 4.25 < 4.50 ** OR ** we define this candle as (5) candles [1 1 1 1 1] because 4.25 > 4.00000000001 ?

I’m not sure of an exact answer and don’t know which one would produce the best results. Shouldn’t be too difficult to test the different scenarios right? GG for example takes the ATR(6) I think to split the candles.

So if the ATR(6) is 2pips  and the current candle is 10pips and its a bull candle. I would record that as 5 candles of 2pips each [1 1 1 1 1].

Focus, Patience, Determination & Order in chaos

[Saver0]

sniperguy wrote:

Does it mean that i must stop the JS server, then restart it again to get another 4 pairs ?

Yea it was probably still downloading. What did the Node.js script show? It should show that YYYY Finished for each currency and timeframe. M1 should be a little over 3million bars in size.

Focus, Patience, Determination & Order in chaos

[sniperguy]

Saver0 wrote:

 

Yea it was probably still downloading. What did the Node.js script show? It should show that YYYY Finished for each currency and timeframe. M1 should be a little over 3million bars in size.

Hi saver0,

I think you are right, may be it is still need time to download.  Until now i did not get any year information (  YYYY ) , i just can see “Finished” at the right of the TF.

I  got only near 1 million (around 980,000) rows for M1 ( not 3 millions) and around 17,000  for H4.  Will see later, the “Finished” text next to the tableName is really finish downloading it or not.

• This reply was modified 11 years, 7 months ago by sniperguy.

[Saver0]

Ahh.. I see. If you look at the code, it’s not resuming from where it left off back to the very beginning. Once the data is loaded, it assumes that all of the history is there and only checks to see what’s new. Try emptying out the tables and run it again and give it the time to finish downloading/inserting. It could take a few hours to complete.

Focus, Patience, Determination & Order in chaos

[Feanor]

When you say gaps you mean missing candles or difference between Close of last candle and Open of next candle?

[sniperguy]

@saver0 , thanks my friend. I will try again from beginning, i will delete the whole database first, to get a clean data.

[Saver0]

Feanor wrote:

When you say gaps you mean missing candles or difference between Close of last candle and Open of next candle?

I mean missing candles. Just missing data that should have existed. What I noticed is at Oanda, there were randomly placed gaps that had 3-4 fixed lengths. That’s what I found really strange. It’s as if they just went into their database and removed data just to create these gaps.

Focus, Patience, Determination & Order in chaos

[sniperguy]

Hi Saver0,

Thanks for your tips. I have deleted the “candles” database, and created it again. Then i changed a little the NodeJs script, i changed from 4 pairs to 1 pair only download at the same time. Then i changed the position of the pairs, i use major currency pair at the top: EurUsd, AudUsd, NzdUsd, UsdChf, UsdJpy, GbpUsd.  Now the download process can be observed clearly.  I got 3.7 million EU rows on TF M1 now, great. It is still downloading the 5th pair UsdJpy (still 22 pairs to go) in around 14 hours, because of my slow internet connection. But at least it works … Thank you very much chief …

[Saver0]

Awesome! Glad to hear that its working. Yea I noticed yesterday I was hitting rate limits at Oanda. Maybe it wasn’t working one of the days and now the rate limit is back on. I was only able to make 1 API call per second.

Focus, Patience, Determination & Order in chaos

[Author]

Posts