Don’t hash what you need to understand
I turned my system’s data into hashes to look professional, and lost the one thing that let me understand what it had learnt.
In 2018 I built a home-made system that recognised patterns in the cryptocurrency market and taught itself which ones came before a rise or a fall. Along the way I transformed its data twice, and both times I lost information: discretisation gave up what didn’t matter so that patterns could be compared; the MD5 hash gave up exactly what did matter, and gave nothing back. Not all information loss is equal, and a hash identifies, but it doesn’t describe.
Topics: hash, MD5, discretisation, pattern recognition, machine learning, technical analysis, cryptocurrencies, traceability, black box
Back in 2018, I was glued to the cryptocurrency market and deep into technical analysis. I bought John J. Murphy’s Technical Analysis of the Financial Markets and made it my personal Bible: I underlined it, reread it and drove my friends up the wall explaining what a “head and shoulders” was.
One of the ideas the book keeps coming back to is that charts behave like fractals: certain patterns show up over long periods and short ones alike, and they tend to signal the same move, up or down. With that idea in my head, I wrote a few scripts that analysed the market in real time over fixed time windows (10 minutes, 30 minutes, an hour...) and drew the result in a clumsily built chart in the terminal.
I wanted a pattern recognition system that would tell me which figure from “the Bible” was forming at any given moment, so I could get ahead of the market. But I ran into a problem: prices have no fixed scale. Whatever I learnt while a coin traded in one price range was useless once it moved to another.
The first loss: the one worth it
I mentioned it to my friend Juanjo, an electronic engineer and systems analyst, and he gave me the key: use a sliding window (a FIFO queue holding the latest readings) and discretise it. So I took the last 10 price and volume readings and slotted each one into a scale from 0 to 9, where 0 was the lowest value in the window and 9 the highest. Whether BTC was at $6,000 or $20,000 made no difference: I always got the same universal “drawing” of the chart. Something like this:
| 9 | ● | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 8 | ||||||||||
| 7 | ||||||||||
| 6 | ● | |||||||||
| 5 | ● | |||||||||
| 4 | ● | |||||||||
| 3 | ● | ● | ||||||||
| 2 | ● | ● | ||||||||
| 1 | ● | |||||||||
| 0 | ● | |||||||||
| t | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| # | 0 | 1 | 3 | 2 | 3 | 4 | 2 | 6 | 5 | 9 |
| 0 | ● | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ● | ● | ||||||||
| 2 | ● | ● | ● | |||||||
| 3 | ● | |||||||||
| 4 | ● | |||||||||
| 5 | ||||||||||
| 6 | ● | |||||||||
| 7 | ||||||||||
| 8 | ||||||||||
| 9 | ● | |||||||||
| t | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| # | 0 | 1 | 2 | 2 | 2 | 1 | 3 | 4 | 6 | 9 |
The drawing translated into a code by reading off the value of each column. The one in the example would be 0132342659;0122213469 (price;volume).
Look at what I had just done: thrown information away. The code no longer said what BTC was trading at, how many dollars it had moved, how much real volume there had been or what percentage rise or fall it meant. All of that was gone. But in exchange I gained something huge: two moments in the market with completely different prices could have the same drawing, and therefore the same code. At last I could compare them. And the code was still readable: one look at it and I could redraw the chart. I had lost detail, but not the way back.
From there, all the program had to do was wait for the next reading and check whether the price had gone up, gone down or stayed the same compared with the previous one. It stored the code with three counters, BULLISH, BEARISH and NEUTRAL, and added one to the right counter every time the pattern came up again. Each stored pattern was simple: PRICE;VOLUME;TIMES-BULLISH;TIMES-BEARISH;TIMES-NEUTRAL.
Without realising it, I was doing machine learning: I was no longer the one labelling the patterns; the program learnt on its own, comparing prices and saving the result. I left it running for weeks. Every so often it recognised a pattern it had seen before and warned me about the next move, and to my surprise it was right fairly often.
But it was painfully slow. In theory there were 1020 possible patterns (a hundred quintillion). Even though in practice the vast majority would never occur, with 10-minute windows the learning was maddening. So it occurred to me to download BTC’s price and volume history and feed it millions of past windows to fill the database. It paid off: predictions, which until then had come in dribs and drabs, became almost constant, or at least very frequent. Using each pattern’s counters, the program gave a probability for each move, and whether it got it right or wrong, it kept correcting itself.
The second loss: the pointless one
As I fine-tuned the system, I started to think that keeping all of that in a CSV file “wasn’t professional”. And it wasn’t: I had millions of CSV lines (yes, that was my “database”), and they were anything but efficient. So I set up PostgreSQL and, while I was at it, turned the codes into MD5 hashes. It seemed like what someone who knew what they were doing would do.
That’s where I blew it. MD5 is a one-way function: the code produces the hash, but there is no getting from the hash back to the code. Where I used to have 0132342659;0122213469, which I could read and draw, I now had a string of 32 hexadecimal characters that meant nothing to me.
The system worked just as well, and was even more efficient. It could tell whether the current drawing matched one it had already seen, and whether that one had been bullish or bearish, but I could no longer ask it what it had seen. I couldn’t pull out the ten most reliable patterns and look at them. I couldn’t check whether the head and shoulders from my Bible actually worked. I knew which hashes worked, but not what shape they had.
The worst part is that I gained nothing in return. The 20-digit code was already a unique, short key, perfect for indexing. The hash didn’t make it faster, smaller or more secure. It only made it unreadable. On top of that, it introduced a chance of collision (vanishingly small, but one that hadn’t existed before) into a system that had no need for it.
Two ways to lose information
Both transformations lost information. The difference wasn’t how much they lost, but what they lost and what they gave in return. Discretisation gave up what didn’t matter (the exact price) to gain what did (being able to compare patterns), and it still let me go from the code back to the drawing. The hash gave up precisely what mattered (being able to understand what the system had learnt), gave nothing back, and cut off the way back altogether.
A hash is for identifying something, not for describing it. I mistook the identifier for the data, and kept the fingerprint while throwing away the finger.
How I would do it today
The irony is that the information wasn’t entirely lost. I still had the history: all I had to do was go through it again, compute the code and the MD5 of every window, and rebuild the lookup table. But being able to fix something doesn’t mean it was done right. These days, before I transform any data, I ask myself two questions:
- What do I lose, and what do I gain?
- Can I go back if I need to?
If I lose something and gain nothing, I don’t do it. If I can’t go back, I keep the original alongside. A few extra bytes per row is a ridiculously small price for being able to look inside your own system and ask it why. Years later, reading about artificial intelligence, I came across the term “black box”: models that get things right but that nobody can explain. The industry has made, on a gigantic scale, a trade much like mine, giving up “understanding” in exchange for performance. The difference is that they, at least, got something for what they lost.
I just wanted to look professional.