- This topic has 14 replies, 5 voices, and was last updated 10 years, 3 months ago by
.
- Posts
Hi Guys!
This is a follow-up of this post I made yesterday. I decided to start a thread on this topic because data quality is essential for many indicators to work properly, especially sophisticated ones which rely on calculating artificial pairs or short term currency strength.
To check the quality of timestamps in my MT4 history I wrote the script which I attached. For 28 pairs and all native MT4 timeframes, it starts at bar 1 and compares timestamps of all pairs at that bar index. When it finds a mismatch, this is logged into a csv file, and the next timeframe is processed. This means: it only logs the FIRST mismatch of each timeframe. Runtime on my system is less than 1 second.
On my development machine, connected to Global Prime Demo Server, the results were horrendous.
Examples:
- On M1, I hardly find more than 20 bars (starting at bar 1) that have the same timestamp for all 28 pairs on the same bar index. Depending on data reloaded, the result is different from run to run.
- On M5, it’s only a little better.
- On M15, I managed after some manual reload to get a stable number of about 1200 bars that can be used.
- On H4, there are obviously faulty data (several H4 bars marked 10:00 instead of 12:00)
- … and many more similar cases.
Pic shows results for M1 with bar 1 out of sync for GBPCHF.
I ask everyone around to download the script, run it on your system(s), and report back here, naming your broker and posting the logfiles. This might be interesting for everyone around in order to find brokers offering a high quality data feed.
Feel free to modify the script and post your enhancements.
Every additional advice regarding high quality price feeds for MT4 or methods to enhance data quality is highly appreciated!
s.
EDIT: Saver0 started a thread about data feed over here about one year ago – useful links in there!
A good trader is a realist who wants to grab a chunk from the body of a trend, leaving top- and bottom-fishing to people on an ego trip. (Dr. Alexander Elder)
Neither custom indicator window opened nor any file has been saved.
Any message in in the ‘Experts’ log?
Do your symbols require a prefix or suffix character? If yes, please modify the symbols array in the code, because I forgot them.
A good trader is a realist who wants to grab a chunk from the body of a trend, leaving top- and bottom-fishing to people on an ego trip. (Dr. Alexander Elder)
@simplex: Sry, my mistake. Add it to the wrong folder (indicators instead of script).
Here are my results – Broker GKFX
I could not find the log file. I looked in both log directories but they did not have any output file like shown above.
simplex,
I don’t understand your analysis.
Ticks are not coming in equal rates, but only when price move. Therefor they are not synchronized across all pairs.
Or, I don’t quite understand something….
G.
I could not find the log file. I looked in both log directories but they did not have any output file like shown above.
First: this is a script, not an indicator. So it has to be installed in the scripts folder. Please check!
Then look in the files directory for results. Filename is the same as script name, filetype .csv
A good trader is a realist who wants to grab a chunk from the body of a trend, leaving top- and bottom-fishing to people on an ego trip. (Dr. Alexander Elder)
Ticks are not coming in equal rates, but only when price move. Therefor they are not synchronized across all pairs. Or, I don’t quite understand something…. G.
Problems will occur if we use indicators that use data from multiple pairs. When some candle’s data are missed, the indices between those pairs asynchronize. Therefor it would be great if MT4 history would have prizes for every pair on every candle. Although prizes don’t change, we can give a prize for every single time point.
Another problem is that some histories contain big gaps (also of hours for my broker) that most probably can’t be explained by missing tick activity.
I tried to force similar results on my system by running the analysis including a pair I’ve never used, like SGDJPY. See pic for results: in this case no output for TF smaller than H1, and H1 pointing to timestamp 2015.03.26,02:00 at bar index = 1.
So not completely the same result as yours, but anyway I would interpret your results in a way that you simply do not have any local history for those pairs indicating a timestamp like 1970.01.01.
I don’t understand your analysis. Ticks are not coming in equal rates, but only when price move. Therefor they are not synchronized across all pairs.
Good point! Yet I don’t analyse tick data here, I simply check whether bar index and timestamp of a bar are matching for different pairs in local history.
Why did I start this?
When working with simple indicators like Stoch, an MA or whatever, it does not matter that much when a bunch of bars are missing in history. When coding more sophisticated indicators, which have to analyse many pairs to calculate an output (like working with synthetic symbols, or CS algorithms) we usually are linking the price data of all underlying pairs using the bar index. Most indicators have a simple loop over all bars for that purpose. To get precise results, it’s essential that at any bar index the timestamps of all underlying pairs are identical. And this is what most coders of such algorithms (including me) are assuming, at least implicitly.
This is the condition my script is checking, showing results I don’t like.
Now when using those indicators in an EA, we can easily check in realtime whether all required bars within a certain lookback period are having correct timestamps, thus suppressing trade signals if faulty bars occur. When trying to test our signals statistically using price history, this won’t work. Any missing bar will add a certain time gap (according to timeframe chosen) to all bars left (the past) of the missing bar, and the underlying pairs are out of sync. It’s the statistical tests that don’t work precisely under these conditions.
Any thoughts anybody?
s.
A good trader is a realist who wants to grab a chunk from the body of a trend, leaving top- and bottom-fishing to people on an ego trip. (Dr. Alexander Elder)
This should be beter..
It says on FXCM demo you’re having 285 M1 bars in sync, on Oanda Live the first async fault occurs on bar 1 (see column IDX). The script will list timestamp data of the first bar (starting at IDX = 1) that is out of sync (faulty).
Now you’ll have to decide for yourself whether this is a problem regarding the strategies / indicators you’re using or testing on those accounts.
If you want to run the script only on a part of those symbols, the script code would have to be modified. It is kind of a quick and dirty hack of last night.
s.
A good trader is a realist who wants to grab a chunk from the body of a trend, leaving top- and bottom-fishing to people on an ego trip. (Dr. Alexander Elder)
Quite a good explanation of CrapT4 crappy price feed can be found over here:
http://privatelink.de/?http://www.forexfactory.com%2Fshowthread.php%3Fp%3D5638648%23post5638648
A good trader is a realist who wants to grab a chunk from the body of a trend, leaving top- and bottom-fishing to people on an ego trip. (Dr. Alexander Elder)
- You must be logged in to reply to this topic.