It’s what every investor has dreamed of: a way of predicting stock market moves. And according to researchers in the UK, it’s easier than you’d think.
Changes in the US Dow Jones index are presaged by the ebb and flow of Google searches for certain financial keywords.
Better still, acting on these trends could make you a huge profit – at least, until others find the same keywords, of course.
Predictably, publication of this claim in the journal Nature Scientific Reports attracted a lot of attention.
It was also hailed as evidence of the power of so-called Big Data: the extraction of insight from datasets.
Now it looks set to become a classic example of what Big Data can do – but not in a good way.
An analysis of the claim published this month suggests it is an object lesson in how digging into Big Data can trigger an avalanche of dross.
With everyone from politicians to billion-dollar corporations looking to Big Data for answers to big questions, it holds vital lessons for all of us.
And topping the list is: never let go of common sense.
When it first began to make headlines around a decade ago, the exploitation of Big Data was hailed as nothing short of a revolution in how we can make sense of the world. In the face of so much information, there was simply no reason to bother dreaming up intricate theories and then testing them.
"Out with every theory of human behaviour, from linguistics to sociology. Forget taxonomy, ontology and psychology," wrote Chris Anderson, editor-in-chief of Wired. "With enough data, the numbers speak for themselves."
At last month’s World Government Summit in Dubai, the potential of Big Data was held up as the means to solve a host of problems, from delivering local public services to tackling global food security.
But now many of those who have looked to Big Data for insight have found that when left to “speak for themselves”, numbers often spout gibberish.
Take that claim that Google search data can predict stock market moves. The idea isn’t so crazy: market swings reflect changes in sentiment among investors, and these might well show up as changes in the volume of searches for certain financial terms.
But which ones? In their Nature Science Reports study, the researchers came up with a list of about 100 terms they thought might pop up, like "risk" and "investment". They then set computers digging into Google's databases, looking for links between search volumes for each term and changes in the Dow Jones index.
And it worked. The computer found correlations between the two.
But the researchers went further, showing that the best correlation could be turned into hard cash.
Their results showed that a trading strategy of buying when the word “debt” is trending upwards and vice-versa would have generated a profit of more than 300 per cent compared to a simple buy-and-hold strategy.
But the key phrase there is “would have”. The researchers did not make any money themselves - not because they were above such things but because they would have needed a time machine. That’s because all their results were retrospective, revealing only past correlations.
And that’s where the whole idea unravels.
By running a list of 100 terms against past records of the Dow Jones, the researchers risked falling into a classic Big Data trap: finding correlations that are meaningless flukes.
Such correlations are shockingly common, as former Harvard law student Tyler Vigen has shown through his now-celebrated eponymous website.
It contains countless silly but impressively strong correlations found by automatically scouring the web.
For example, Vigen’s computers found that the annual amount of honey produced in the US is very strongly correlated to the number of murders using blunt objects.
Of course, no one with any common sense would fall for such “links”. But that’s the problem with using computers to find them: common sense does not come as standard.
In theory, genuine links might exist between search terms like “debt” and stock market changes. But the only way to find out is to check their predictive power.
Dr Wai Mun Fong, associate professor of finance at the National University of Singapore, has now done precisely that, in research published in the current Journal of Index Investing.
Dr Fong took the same list of about 100 terms used in the original study but this time tested their predictive power in years not examined by the original researchers.
Search terms that proved most strongly correlated with the Dow Jones in one year were used to predict market behaviour in the following year.
Sure enough, Dr Fong confirmed that some Google search terms were correlated with market moves.
But they proved to be “society”, “cancer”, “home” and various others with pretty tenuous links to investor sentiment. Significantly, one search term that failed to predict the market over this new timescale was “debt”.
Dr Fong’s conclusions are blunt: using Big Data to predict stock prices that aren’t based on solid economic theory or properly tested “are doomed to be misguided and futile”.
That such basic warnings are still necessary says much about the spell Big Data is weaving over smart people.
Even Google, that titan of Big Data, has proved vulnerable. In 2008 it unveiled Google Flu Trends (GFT), with the aim of helping predict future flu epidemics through spikes in certain search terms.
To find these keywords, computers had looked for links between 50 million candidates and just 1,200 flu outbreak data points. So big a mismatch is a sure-fire way of finding spurious correlations.
And thus it proved. Like the stock market predictor, GFT made headlines, then mistakes and, ultimately, proved useless. Google threw in the towel in 2015.
Now others are going down the same path. The latest victims of Big Data are hedge funds, who have ploughed huge sums into finding patterns that give them an edge. One industry insider recently told Bloomberg that the “insights” they’re finding in Big Data have a failure rate of about 90 per cent.
Few can resist surveying the towering masses of global data without believing the prospector’s credo “There’s gold in them thar hills”. There is, but it’s buried deep in dross - and simply grabbing a shovel and digging is not the way to find it.
Robert Matthews is visiting professor of science at Aston University, Birmingham, UK
Know before you go
- Jebel Akhdar is a two-hour drive from Muscat airport or a six-hour drive from Dubai. It’s impossible to visit by car unless you have a 4x4. Phone ahead to the hotel to arrange a transfer.
- If you’re driving, make sure your insurance covers Oman.
- By air: Budget airlines Air Arabia, Flydubai and SalamAir offer direct routes to Muscat from the UAE.
- Tourists from the Emirates (UAE nationals not included) must apply for an Omani visa online before arrival at evisa.rop.gov.om. The process typically takes several days.
- Flash floods are probable due to the terrain and a lack of drainage. Always check the weather before venturing into any canyons or other remote areas and identify a plan of escape that includes high ground, shelter and parking where your car won’t be overtaken by sudden downpours.
THE LIGHT
Director: Tom Tykwer
Starring: Tala Al Deen, Nicolette Krebitz, Lars Eidinger
Rating: 3/5
SPECS
%3Cp%3E%3Cstrong%3EEngine%3A%3C%2Fstrong%3E%20Dual%20electric%20motors%20with%20102kW%20battery%20pack%3C%2Fp%3E%0A%3Cp%3E%3Cstrong%3EPower%3A%20%3C%2Fstrong%3E570hp%3C%2Fp%3E%0A%3Cp%3E%3Cstrong%3ETorque%3A%3C%2Fstrong%3E%20890Nm%3C%2Fp%3E%0A%3Cp%3E%3Cstrong%3ERange%3A%3C%2Fstrong%3E%20Up%20to%20428km%3C%2Fp%3E%0A%3Cp%3E%3Cstrong%3EOn%20sale%3A%3C%2Fstrong%3E%20Now%3C%2Fp%3E%0A%3Cp%3E%3Cstrong%3EPrice%3A%20%3C%2Fstrong%3EFrom%20Dh1%2C700%2C000%3C%2Fp%3E%0A
How to protect yourself when air quality drops
Install an air filter in your home.
Close your windows and turn on the AC.
Shower or bath after being outside.
Wear a face mask.
Stay indoors when conditions are particularly poor.
If driving, turn your engine off when stationary.
The specs: 2018 Nissan Patrol Nismo
Price: base / as tested: Dh382,000
Engine: 5.6-litre V8
Gearbox: Seven-speed automatic
Power: 428hp @ 5,800rpm
Torque: 560Nm @ 3,600rpm
Fuel economy, combined: 12.7L / 100km
Jeff Buckley: From Hallelujah To The Last Goodbye
By Dave Lory with Jim Irvin
Living in...
This article is part of a guide on where to live in the UAE. Our reporters will profile some of the country’s most desirable districts, provide an estimate of rental prices and introduce you to some of the residents who call each area home.
Moon Music
Artist: Coldplay
Label: Parlophone/Atlantic
Number of tracks: 10
Rating: 3/5
How to avoid crypto fraud
- Use unique usernames and passwords while enabling multi-factor authentication.
- Use an offline private key, a physical device that requires manual activation, whenever you access your wallet.
- Avoid suspicious social media ads promoting fraudulent schemes.
- Only invest in crypto projects that you fully understand.
- Critically assess whether a project’s promises or returns seem too good to be true.
- Only use reputable platforms that have a track record of strong regulatory compliance.
- Store funds in hardware wallets as opposed to online exchanges.
WHY%20AAYAN%20IS%20'PERFECT%20EXAMPLE'
%3Cp%3EDavid%20White%20might%20be%20new%20to%20the%20country%2C%20but%20he%20has%20clearly%20already%20built%20up%20an%20affinity%20with%20the%20place.%3Cbr%3E%3Cbr%3EAfter%20the%20UAE%20shocked%20Pakistan%20in%20the%20semi-final%20of%20the%20Under%2019%20Asia%20Cup%20last%20month%2C%20White%20was%20hugged%20on%20the%20field%20by%20Aayan%20Khan%2C%20the%20team%E2%80%99s%20captain.%3Cbr%3E%3Cbr%3EWhite%20suggests%20that%20was%20more%20a%20sign%20of%20Aayan%E2%80%99s%20amiability%20than%20anything%20else.%20But%20he%20believes%20the%20young%20all-rounder%2C%20who%20was%20part%20of%20the%20winning%20Gulf%20Giants%20team%20last%20year%2C%20is%20just%20the%20sort%20of%20player%20the%20country%20should%20be%20seeking%20to%20produce%20via%20the%20ILT20.%3Cbr%3E%3Cbr%3E%E2%80%9CHe%20is%20a%20delightful%20young%20man%2C%E2%80%9D%20White%20said.%20%E2%80%9CHe%20played%20in%20the%20competition%20last%20year%20at%2017%2C%20and%20look%20at%20his%20development%20from%20there%20till%20now%2C%20and%20where%20he%20is%20representing%20the%20UAE.%3Cbr%3E%3Cbr%3E%E2%80%9CHe%20was%20influential%20in%20the%20U19%20team%20which%20beat%20Pakistan.%20He%20is%20the%20perfect%20example%20of%20what%20we%20are%20all%20trying%20to%20achieve%20here.%3Cbr%3E%3Cbr%3E%E2%80%9CIt%20is%20about%20the%20development%20of%20players%20who%20are%20going%20to%20represent%20the%20UAE%20and%20go%20on%20to%20help%20make%20UAE%20a%20force%20in%20world%20cricket.%E2%80%9D%C2%A0%3C%2Fp%3E%0A
COMPANY PROFILE
Name: Almnssa
Started: August 2020
Founder: Areej Selmi
Based: Gaza
Sectors: Internet, e-commerce
Investments: Grants/private funding
Book%20Details
%3Cp%3E%3Cem%3EThree%20Centuries%20of%20Travel%20Writing%20by%20Muslim%20Women%3C%2Fem%3E%3Cbr%3E%3Cstrong%3EEditors%3A%20%3C%2Fstrong%3ESiobhan%20Lambert-Hurley%2C%20Daniel%20Majchrowicz%2C%20Sunil%20Sharma%3Cbr%3E%3Cstrong%3EPublisher%3A%20%3C%2Fstrong%3EIndiana%20University%20Press%3B%20532%20pages%3Cbr%3E%3C%2Fp%3E%0A
Quick pearls of wisdom
Focus on gratitude: And do so deeply, he says. “Think of one to three things a day that you’re grateful for. It needs to be specific, too, don’t just say ‘air.’ Really think about it. If you’re grateful for, say, what your parents have done for you, that will motivate you to do more for the world.”
Know how to fight: Shetty married his wife, Radhi, three years ago (he met her in a meditation class before he went off and became a monk). He says they’ve had to learn to respect each other’s “fighting styles” – he’s a talk it-out-immediately person, while she needs space to think. “When you’re having an argument, remember, it’s not you against each other. It’s both of you against the problem. When you win, they lose. If you’re on a team you have to win together.”
COMPANY PROFILE
Name: Qyubic
Started: October 2023
Founder: Namrata Raina
Based: Dubai
Sector: E-commerce
Current number of staff: 10
Investment stage: Pre-seed
Initial investment: Undisclosed
COMPANY%20PROFILE
%3Cp%3EFounder%3A%20Hani%20Abu%20Ghazaleh%3Cbr%3EBased%3A%20Abu%20Dhabi%2C%20with%20an%20office%20in%20Montreal%3Cbr%3EFounded%3A%202018%3Cbr%3ESector%3A%20Virtual%20Reality%3Cbr%3EInvestment%20raised%3A%20%241.2%20million%2C%20and%20nearing%20close%20of%20%245%20million%20new%20funding%20round%3Cbr%3ENumber%20of%20employees%3A%2012%3C%2Fp%3E%0A
MOTHER%20OF%20STRANGERS
%3Cp%3EAuthor%3A%20Suad%20Amiry%3Cbr%3EPublisher%3A%20Pantheon%3C%2Fp%3E%0A%3Cp%3EPages%3A%20304%3Cbr%3EAvailable%3A%20Now%3C%2Fp%3E%0A
MATCH INFO
Uefa Champions League semi-final, first leg
Barcelona v Liverpool, Wednesday, 11pm (UAE).
Second leg
Liverpool v Barcelona, Tuesday, May 7, 11pm
Games on BeIN Sports
Fixtures
%3Cp%3E%3Cstrong%3EWednesday%2C%20April%203%3C%2Fstrong%3E%3C%2Fp%3E%0A%3Cp%3EArsenal%20v%20Luton%20Town%2C%2010.30pm%20(UAE)%3C%2Fp%3E%0A%3Cp%3EManchester%20City%20v%20Aston%20Villa%2C%2011.15pm%20(UAE)%3C%2Fp%3E%0A%3Cp%3E%3Cstrong%3EThursday%2C%20April%204%3C%2Fstrong%3E%3C%2Fp%3E%0A%3Cp%3ELiverpool%20v%20Sheffield%20United%2C%2010.30pm%20(UAE)%3C%2Fp%3E%0A
More from Neighbourhood Watch: