The Unfathomably Large Size Of Tiny Things

When we talk about physical objects we have a tendency to describe them in three dimensions. That helps us understand the physical size of a thing. The length, width, and depth. Often described as X, Y, and Z in a three-dimensional coordinate system.

This is very helpful.

Once upon a time, we didn't know this, so somewhere back in the past maybe everything was flat because we hadn't discovered the third dimension yet. Perhaps we only had linear movement because we'd not even discovered the second!

I digress. It's what I do. It's an accidental hobby.

Here's the thing... most people simply can't understand the unfathomably large or unfathomably tiny things in life, even when given dimensions. So we often find the fourth dimension, time, a helpful additional measurement to help describe extreme sizes.

Ahh... four dimensions, now we're talking! Space and time. Fancy.

Let me describe some scenarios.

Take for example a tennis ball.

That's easy to understand because most of us have seen or interacted with one. We can describe it to somebody that hasn't though, by using its dimensions. We can simply say that it's a ball that is approximately 7cm in all three dimensions. People understand because it's easy to conceptualize. There is no value in adding our super helpful fourth dimension to this.

Let's take something a bit bigger: 3 miles (or 5 kilometres, they're almost the same).

That's much harder to conceptualize. So we can add our really useful fourth dimension to help articulate how big that is. It's a one-hour walk. Approximately. So people who walk around at an average human pace will have a rough idea of what 3 miles is, by helping the physical dimension out by using time as an additional measurement.

I mentioned that 3 miles and 5 kilometres are almost the same. Yet, if you were to ask an ant, it will assure you that the difference between 3 miles and 5 kilometres, which is about 0.1 miles, is an awful lot. Ask a snail, and I suspect it will have an opinion even stronger than the ant. It may even roll its protruding eyes at you. Context is really important; relative size and all that. Even when you add time. The difference of 0.1 miles is not a lot for a typical human, yet it's a vast distance for ants, snails, and a bewildering amount of small critters. On a molecular level, it's an almost incomprehensible distance.

This isn't about molecules though. Nope! It's about bits, bytes, and pixels.

Why? Because in about 1986, in my pre-teens, when I was well and truly hooked on computers and coding assembly language on an 8-bit Acorn BBC Micro Model B (the very bestest of 8-bit computers, I might add), I had this idea... possibly, dare I say, an epiphany.

Before I move on to what I'm about to describe, let's just remember that helpful fourth dimension, time. I'm deleting it. Just for a moment. Sorry about that.

Without time, everything that has ever been and everything that ever will be happens simultaneously in an instant. There is no past. There is no future. Technically, there isn't even the present moment. However, the present is the closest concept to having no concept of time at all, so feel free to imagine that without time, everything is happening in the present moment; now. When I say "everything", I truly mean "everything". In italics, no less.

For the excitable among you, please disregard alternate realities - we're not going there. Not in this article, anyway.

Right, back on track... remember; time has been temporarily deleted.

Pixels

On the trusty BBC Micro, I started designing games and trying to code them, because game programming was going to be my destiny. No longer was I going to be a forensic scientist. Computers... that is where it was at.

Back then, every budding games programmer was also a pixel artist out of necessity. We would design little characters (sprites), sometimes in colour, sometimes in monochrome, depending on what our graphics capability was.

In the early days, that meant drawing lots of little 8x8 grids on paper and sitting there with a pencil colouring in the blocks, and putting lots of 8x8 grids together to design larger areas or larger characters. At some point, I upgraded and started buying graph paper where the grid was already printed on the paper - ooh, the efficiency gains were crazy! Back in 1986 time hadn't been temporarily deleted, so saving time was useful. I also would not have talked about efficiency gains in 1986, I'd have said something more like "Wow, this is dead fast now".

It didn't take long to realise it was a good idea to code up a sprite maker. So I did. Now I didn't have to draw everything on paper and figure out the binary representation of what I'd drawn, convert it to hexadecimal, and then pop that into my buffer to recall in code. Instead, now I could colour in my sprites, stitch them together on screen, save them, and they were automatically encoded, and I could see the real size version of them. Fantastic! Of course, I do at times reflect on the simple beauty and nostalgia of doing it all by hand, and I certainly appreciate the fundamentals that it taught me.

At some point during this, I had an idea. The possible epiphany.

Why would I go through the creative exercise of designing all these sprites, backgrounds, alternate fonts, or whatever pixelated thing was being conjured up, when I could simply generate them?

Think about it; all I was doing was coming up with appealing and relevant combinations of pixels in an 8x8 grid. Some would be utter nonsense, and others would be classics that would stand the test of the not-yet-temporarily-deleted time. Take for instance the classic space invader, or pac man. They're essentially built around 8x8 pixels, sometimes larger of course and stitched together. You get the idea though.

So I didn't need to go through the creative pains of creating them; having to think about something, try it, alter it, throw it away, and start over.

Just generate them.

All of them.

I will write that again, decorated with italics because it's one of the important parts of this thing you're reading.

All of them.

Every combination of 8x8 pixels could simply be generated and then I just needed to look at them all and decide which I wanted for whatever I was doing.

No matter what I wanted, whether it was a fancy-looking capital letter A written upside down, a cool eye, something representative of a poison bottle, or something that would pass as a frog... it would have been generated. I just had to look at them and pick out the things I wanted.

The idea didn't stop there.

It occurred to me that an 8x8 pixel grid was just a tiny part of the full screen, and if I could generate every possible combination of an 8x8 pixel grid, I could scale that out and generate every combination of the full screen.

The BBC Micro's highest resolution was the nifty "Mode 0 (zero)", with a monochrome resolution of 640x256. Lower resolutions meant more colours, the ever-popular "Mode 2" had all the colours (all 8 of them, ignoring the flashing colours) at the cost of a much lower resolution of 160x256.

So that was the idea and the reason.

The thought process didn't stop there though.

The thought itself progressed to the realisation that everything that could be drawn within the confines of the resolutions and colour availability would be drawn.

Every 8-bit visualisation of everything that has ever been, or ever will be, would be generated.

For every great artwork out there, I would generate the best and closest 8-bit representation of it. For every face on the planet, I would generate an 8-bit representation; with every possible hairstyle and every age of the same person's face. Every piece of text that has ever been written would be written, albeit in fragments - in every language, even hieroglyphs. So every book, every fact, every future news headline, every poem, every lyric, every piece of sheet music to accompany the lyrics... would all be generated. In fact, the very paragraph that you are reading now would also have been generated.

Because it had to be.

It was simply the algorithm doing its thing.

Naturally, the thought process did stop there either, because, why stop at graphics?

The same theory could be applied to sound. Every sound can be digitised, and therefore every sound that has been, or could ever be, could be digitally generated. Every great speech you've heard could be generated, in every language, in every voice, including your own, with some mellow dream-pop music running in the background that suddenly breaks into the sound of running water with a very specific viscosity.

If you bring those together in exactly the right order and do it repeatedly, you will generate every video footage, accompanied by full audio, for everything that... you guessed it, has ever been, or ever will be.

Some of the generated content would be byte-for-byte identical to actual digitised footage of recorded history; not so close that it's indistinguishable, I truly mean a verbatim, artificially generated, byte-for-byte identical artefact. Even for events that have not yet happened.

Because it has to be.

It is simply the algorithm doing its thing.

I never wrote that code.

It remained a concept that I would sometimes talk about, and one that I've consistently come back to over the years and continued to expand as an idea.

When I first came up with the concept I had access to a BBC Micro with very limited graphics and sound and I wanted to generate 8x8 sprites for games. Think for a moment about the level of technology we all have readily available now. Apply the same principle to today's contemporary technology and imagine just the single screen implementation that would generate every combination of a 4k resolution screen with each pixel being able to represent 16.7 million different colours.

Everything you have ever seen, or will ever see, or could imagine - good, bad, or downright disturbing - would be generated in ultra-realistic high definition. Even the things you couldn't imagine or wouldn't want to imagine would be generated.

Because it has to be.

It is simply the algorithm doing its thing.

Time

I mentioned that I didn't write the code. I didn't. Although that's not strictly true. I didn't write the code back in 1986, anyway.

How on Earth does all this relate to the unfathomable size of tiny things, and why has time been temporarily deleted?

I'm getting to it. Time was temporarily deleted, remember. So, technically you have all the time in the world when there's no time, and patience isn't even needed because it's a useless concept without time.

Wait, we're not ready for time...

Bits & Bytes

A monochrome 8x8 pixel grid is exactly 64-bits wide.

If that makes no sense, that's okay. It's a series of ones and zeros; 64 of them to be precise, all lined up neatly next to each other. A zero represents a pixel that hasn't been lit up, and a one represents a pixel that has been lit up.

So, going back to the paper I would draw on, I would draw an 8x8 grid, colour some blocks in and leave others empty, then write a 1 in the ones I coloured in and a 0 in the ones that I didn't, then I could take the first 8 and convert that to something called a "byte". Don't worry if you're not technical, this isn't a lesson in binary representation, maths, or anything like that.

On a fundamental level, each one of those ones and zeros is represented by a transistor that is in one of those two states. A simple switch. The fundamental building block of computing. The molecular level of all digital forms; a single 1 or 0. It's called a "bit".

I said it wasn't about molecules. It isn't... and it is... sort of. Digital molecules. Binary "bits".

I am of course ignoring quantum computing, stateless observation, and suchlike.

Digital molecules. Binary "bits".

One bit. This is our tiny thing.

If we take 64 bits and put them together, it is still a tiny thing; it's just not the smallest tiny thing.

Time

Oh, hello, we're back on time, so to speak.

I didn't write the code back in 1986 because of our helpful fourth dimension; time.

When we introduce time, our tiny thing suddenly becomes an unfathomably large thing.

This is why I needed to temporarily delete time. Sorry about that.

Everything I have mentioned, in reality, won't be possible to observe within our own lifetime. So our fourth dimension, which we helpfully used earlier to describe a big thing in three-dimensional space, is suddenly our Achilles heel because it created an unfathomably large size out of a tiny thing.

Damn you, time!

Okay, let's undelete time. It may take a brief moment to catch up.

The Unfathomably Large Size of Tiny Things

As mentioned, an 8x8 monochrome grid is exactly 64 bits.

If you turn that into a regular decimal number (it's 2^64), you get the number 18,446,744,073,709,551,615.

That's a big number.

In words, it's eighteen quintillion, four hundred forty-six quadrillion, seven hundred forty-four trillion, seventy-three billion, seven hundred and nine million, five hundred and fifty-one thousand, six hundred and fifteen.

I told you it was a big number.

In fact, I would go so far as to say that it is an unfathomably large number.

Imagine if you will that we generate every combination of the 8x8 monochrome pixel grid. That is how many combinations would be generated.

This is where time can become helpful again, to describe the unfathomably large size of tiny things.

Imagine further, that you display each combination of the grid for just 1 second.

It would take 584,942,417,355 years to view every combination.

That's over 584 billion years.

To put that in perspective, according to current scientific understanding, the Earth formed approximately four and a half billion years ago.

I ain't got time for that.

So I didn't write the code.

Instead, I continued being creative and randomly choosing the most appealing and relevant pixels to colour in.

Afterword

When I decided to write this article, I also decided I really ought to finally get around to writing the code. Let's face it, 37 years is a very long time to procrastinate, even for a professional procrastinator.

So I did.

Finally, in 2023, I have written a Python script to show every combination of a monochrome 8x8 pixel grid. You can find the code below, and I've dropped some pre-made sprites in there if you want to start from somewhere interesting.

It has been fun to finally write the code and to also see the first three or four rows generate images that make sense. My favourite so far has been a simple key that it generated... that was enough to prove the concept, and in some ways, is quite poetic.

I suspect I won't be adding an update to let you know when it's finished.

import pygame
import sys

GRID_SIZE = 8
PIXEL_SIZE = 50
WINDOW_SIZE = (PIXEL_SIZE * GRID_SIZE, PIXEL_SIZE * GRID_SIZE)
BLACK = (0, 0, 0)
WHITE = (255, 255, 255)

pygame.init()
screen = pygame.display.set_mode(WINDOW_SIZE)
clock = pygame.time.Clock()

def draw_grid(grid):
    for y, row in enumerate(grid):
        for x, value in enumerate(row):
            color = WHITE if value == 1 else BLACK
            pygame.draw.rect(screen, color, (x * PIXEL_SIZE, y * PIXEL_SIZE, PIXEL_SIZE, PIXEL_SIZE))

def generate_grid(num):
    binary = format(num, '064b')
    binary_digits = [int(x) for x in binary]
    binary_matrix = [binary_digits[i:i+8] for i in range(0, 64, 8)]
    return binary_matrix

def main():
    # key = 457126
    # pickaxe = 33790251778589312
    # spaceinvader = 1746410238856420005
 
    # start with zero if you want a blank page, or override with a predefined sprite above
    iterations = 0

    max_iterations = 2 ** 64
    
    while iterations < max_iterations:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                pygame.quit()
                sys.exit()
            if event.type == pygame.KEYDOWN:
                pause_text = f"Iteration {iterations} of {max_iterations}"
                pause_rect = pygame.Rect(0, 0, PIXEL_SIZE * GRID_SIZE, PIXEL_SIZE)
                pygame.draw.rect(screen, WHITE, pause_rect)
                pause_surface = pygame.font.SysFont('Arial', 20).render(pause_text, True, BLACK)
                screen.blit(pause_surface, (0, 0))
                pygame.display.flip()
                pygame.time.wait(1000)  # pause for 1 second
                pygame.event.clear()
        
        grid = generate_grid(iterations)
        draw_grid(grid)
        pygame.display.flip()
        iterations += 1
        clock.tick(60)

if __name__ == '__main__':
    main()

Disclosure

No ants or snails were harmed in the making of this article. I made certain assumptions on the relative scale and speed of ants and snails being subjected to an additional 0.1-mile journey. Rest assured, I did not force, or even request, any ants or snails to partake in such a journey. I may, however, have conversed with some during the writing process. I will let you decide if it was a bidirectional conversation.

Is your code eco-friendly?

Do you realise that poorly written software is a contributor to global warming?

Although I have absolutely no data to base this on, please, read on... I assure you that you will not only conclude that this is a true statement; if you code, you may even feel compelled to write better code. If that's not enough of a hook, I will introduce to and encourage you to use, this hashtag in your code reviews:

#cowfart

Let me set the scene...

There's a lot of software out there, running on billions of devices across the globe, and sadly, a lot of that code is not optimised.

Back in the day when we wrote in assembler on tiny 8-bit computers we would count clock cycles (honestly; it wasn't just me!) and refactor, refactor, refactor... until we got the best oomph out of the CPU that we could.

As if by magic, computers started popping up everywhere and hardware started to get really fast. And cheap. Suddenly, it was not about clock cycles. It was all about time-to-market. Get the code cut fast, get it released, and if it didn't run very well, upgrade the hardware!

It really was that simple for a long time; certainly for most business software anyway. Games, well that's different because if you have lag... you suck. Well, your game does, and it won't sell. In business software, having things take a while isn't necessarily a problem... more time at the water cooler, more time to do all the admin things.

Sidenote: I would encourage every budding programmer, and even seasoned programmers, who have only worked in business application development, to do some games programming. The techniques you will learn in order to survive out there are transformational.

Technology has moved along at a rapid pace. Astonishingly fast, and astonishingly cheap. And the world, largely, depends on it being that way.

The plot thickens...

Thing is, we've got big problems out there.

With climate.

Okay, not just with climate; other big problems are available.

You don't have to just believe in it anymore either; it's fact, evidenced in the starkest possible ways.

We take a tangent...

Before we get back to my main point, I did promise you a cool hashtag to use in code reviews.

#cowfart

Why? Well, because it is believed that cows are a significant contributor to global warming because there are a lot of cows out there. And they fart. Quite a lot actually. There are many articles and research studies about it.

Okay, okay... cows fart, hardware got better, people stopped counting clock cycles, and released products faster... get to the point already!

So, here's the thing...

If you write sloppy code that does not perform as well as it could, it will take longer to execute. If it takes longer to execute, or if it requires additional processing power just to make it run, you are using more hardware, more electricity, and creating more heat. On a single PC, meh... what's the difference, right? Time to hit the water cooler.

If you scale that up to run on billions of devices across the planet, well... that's a staggering amount of electricity, time, and heat being wasted by your code.

Although your code might not get to be in the next kernel of Linux, Windows, Android, or some other place like Facebook, where it might be executed trillions of times, that's not a good reason to write sloppy code.

Always write the best code you can, and educate yourself on performance tuning.

The end is nigh...

And so, without further ado, next time you review some code or give yourself a TODO:, and see code that you know the could be better performing...

do the polar bears a solid...
keep some habitat for the bees...
think of the children...

Won't somebody please think of the children?

And, as a fun way to say "Hey, this bit of code you wrote. It sucks. It might work, it might even read like a poem. Thing is, it could perform much better. So sort it out."... just put #cowfart in there.

Because the planet's worth it.

Fin.

Robotic Chimney Sweep

Cast your mind back... way back... okay, no, not that far, put the dinosaurs away and bring your mind forward a little... and... there we go.

We're in 1666. To be precise, it's September 6th, 1666. And it's Monday. Oh, and London stopped burning down yesterday.

In a bid to prevent that happening again things need to change, and one of those things was to start cleaning out chimneys on a regular basis. With children. Because they fit. And it turns out they are very cheap and will work long hours.

Bring your mind forward some 209 years. It's now September 1875. I don't know the precise date, nor the day. What I do know, is the Chimney Sweepers' Act of 1875 just got passed and now, at least throughout England, you're not allowed to go stuffing small children up chimney's to keep them clean.

What to do!?

Brushes. Brushes on a long stick. A very long extendable stick. That works, and with the dawn of the vacuum cleaner nearing, things are about to get a lot cleaner.

Well, that's been about it. Some improvements on the fuels being burned help improve the reduction of waste, smoke, tar build-up, and all manner of bad things; though fundamentally, there's a lot of cleaning that still needs to be done.

So it's about time for some innovation in chimney sweeping!

Roll up, roll up... here comes the robotic chimney sweeper.

I have a couple of concepts in mind:

Chimney Drone:
A drone-like bot that stays suspended in the chimney with extended rods that keep it positioned in the chimney and the drone moves up and down the chimney monitoring and cleaning as necessary; staying out of the way of the heat.

Chimney Bug:
A bug-like bot that sticks to the chimney wall and travels around the chimney continually monitoring and cleaning.

Both will monitor airflow and temperatures, toxin build-up, and have lights and cameras for monitoring the internal structure of the chimneys.

Concept (c) 2019 letsbuildathing.com

Data folding... yet another compression method

Recently, I wrote about a compression technique I've called "Iteratively Sequenced Data Compression" making magnificent claims about the level of compression it could attain.

I have another compression technique using [spoiler alert!] folded data structures. I'll explain what I mean by that shortly.

For this technique I won't bore you with a back-history of how I got involved in data compression, nor will I make claim to impressive compression ratios because this, currently, is a little more theoretical.

It's important to remember that when it comes to data compression, we don't really care about the data itself. The data itself can go in the trashcan because it's probably too big - after all, that's why we're looking to compress it right?! All we need is to remember is how to reconstruct the data once we're done forgetting about it. That bit is really important. So important in fact, it deserves a line of its own.

Forget the data; just remember how to reconstruct it.

So, with that in mind, what do I mean by folded data structures?

Imagine you have your data in one nice long strip. Easy enough.

Now imagine you start wrapping that data into rows, essentially like word-wrap on your fave text editor. Also easy.

Okay, so now imagine you had that data printed out on a piece of paper. Also not difficult to imagine.

Hold tight folks, here's where it starts to get a little more tricky.

Now look for a way of folding that piece of paper such that you align common values and drive a skewer through the paper to hold all those matched values in one place.

Let's further imagine you tried folding that piece of paper many different ways until you landed on the optimal folds to get the maximum continuous strip of values on your skewer.

Remember the value you skewered and the folding pattern. That's your first folded data item.

You can forget about that chunk of data now, so go ahead and erase all the data you just skewered from your piece of paper.

Go ahead and repeat this process of finding the optimal fold; remember the value and the folding pattern. That's your second folded data item. Forget the data you just skewered from the page.

Do it again. That's your third. Keep repeating this until you have skewered all your data.

You will find that as you reduce the data by skewering it off the page, your folding patterns become easier to spot, and less complex, so the process becomes faster the more you work through the page.

Once you've got all your folded data items, simply concatenate them and pop them in a small file.

Easy!

All you've had to remember is the skewered value and the folding pattern; which can then be unfolded back out all the way back out to that continuous stream of data that you started with.

So what this technique effectively does is allow you to do run-length-encoding across a landscape of data that, as a continuous strip, could not make the best use of high run lengths; which is therefore sub-optimal. By folding (or crinkling) the data in the right way, you can maximise your run-lengths to get highly effective compression.

Hypothesis:

• Compression will be intensive and slow, whereas decompression will be fast.
• Splicing byte values to enable layering of data prior to folding would improve compression and performance.
• The compression method could be iterative to produce progressively smaller content.

Concept (c) 2019 letsbuildathing.com

When you need "big" and infinity is "too big"

Today I came across a lazy calculator. I asked it a simple enough question, and in response, I was informed that the answer was "infinite"... and that is a wrong answer.

It is not infinite. It is just big. Really big. Some would even say mind-bogglingly big.

The fact is, the calculator was lazy and just could not be bothered to work it out. The thing is, I use calculators to get accurate answers fast. I don't expect my digital buddies to be workshy.

I would have preferred it display the shrugging emoji than an infinite symbol because that would have at least been truthful, and amusing.

Therefore, we need a new symbol for "not as big as infinite, just mind-bogglingly big, so big we won't even work it out, so let's just look at this snazzy symbol instead, kinda big"... you know, for when the shrugging emoji guy just won't cut it.

To which, I have come up with this:

The Less Than Infinity Symbol

Besides, I really need the answer to not be infinite because I'm writing some code and need to reserve a chunk of memory and my calculation was to tell me precisely how much memory I need, and well, I suspect asking to reserve an infinite amount of memory will not have a favourable outcome.

>Hey, computer: Remember all the stuff

Nope.
>_

Iteratively Sequenced Data Compression

I first came across data compression back in the 1980s, some 5 or 6 years after first being awestruck by this thing called a computer.

At the time I was very young, and these computer things were new - and, by all accounts, a pretty big deal. I was fortunate enough to have one at home that I shared with my brother. It was a BBC Micro Model-B from Acorn Computers. Go Acorn!

It was awesome.

This thing was competing with a trusty BMX, He-Man, haunted house exploring, and lots of shenanigans. So it really did have to be awesome to get a look in.

It had 8K of memory. Eight freaking K. An unimaginably large number of transistors that you couldn't dream of filling! Right? Well, no. It turns out that you run out of 8K memory pretty fast... even back in the day.

Back then I was writing 6502 assembly language, dabbling in the 8-bit games industry, designing graphics on graph paper with a pencil (until I wrote a frickin' nifty sprite editor), and maps, because dammit, I was a games programmer! High school and night classes at college were just a necessary pastime. Game programming. That was going to be my life.

And then in 1988 Superior Software brought out an amazing new game called Exile. It was literally a game changer - no pun intended. It was a great game, with many hours and fond memories with buddies spent trying to get to the end of it. I believe it was the first, or one of the first games, that sported an early physics engine... gravity and inertia at least. It was amazing. The graphics were, by 8-bit 8-colour standards, intricate and plentiful. The map was huge. It broke so many rules of what was possible.

All of that was, of course, fascinating and made for a great game.

Only I like to learn and tinker. So tinker and learn I did.

One of the claims that this game made was around the size of the graphics, the size of the sound, and the map. The numbers were staggering - significantly more than 8K!

Whoa!?

How does that work, I thought. It won't fit.

I pondered.

I reached out to friends.

We pondered together.

It did not make sense.

Back then, there was no Internet. There was not an endless amount of resources, or of people doing this stuff to reach out to, especially not outside of universities.

So I took to doing what I would always do in these scenarios. Take it apart. Take a look. Figure out what the hell was going on.

I was pretty resourceful when it came to getting into computers and code, and by this time I was a dab hand at reading machine code; not just assembly language, I mean actual hexadecimal compiled code and understanding exactly what it was up to - often printed out on reams of dot matrix printer paper that I would read travelling about in the back of my parents awesome Rover 2600. Pretty geeky. To be fair, 6502 machine code was a hell of a lot more simple compared to today's machine code, and you had to know it back then to figure out clock cycles and see where you could save the odd millisecond.

Anyway, the point is, Superior Software got all of this content that was bigger than the thing it was going in to by using compression. I genuinely agonized for some time how a bit would get smaller than a one or a zero, came to the conclusion it couldn't, so went to the byte level, and I was still stuck. So then I did some tinkering. You may choose to call it hacking. Whatevs. That, along with talking it through with a buddy, came up with the realisation of what is now known as RLE; run-length encoding.

It's a very basic compression technique whereby you don't store the data itself, rather, you store the pattern of data.

Mindblown.

You don't store the data.

Mind. Blown.

These days it's unbelievably obvious of course and a very basic algo.

Data is quite often a series of the same value, particularly in graphics, maps, and sound. So for example, instead of storing 28 zero's for a strip of black in your image, you store the number 28, followed by the number 0. Then your code decompresses it by reading the 28, 0, and expands it back out to 28 occurrences of 0. The thing is, you only stored two bytes (the 28 and the 0) instead of 28 bytes (28 consecutive 0's). So if you do that across your entire set of graphics, sound, and map data, you get something big into a smaller space.

Easy.

Or rather, easy when you know how.

So... that's a very long intro to a possible new compression technique that, I hasten to add, would make a pretty neat TED talk intro. The important part to take away is this...

You don't store the data.

So I've concluded that if we don't care about the data, and we're never truly going to store the data, and we want lossless compression, well then, let's not even go down that path. Let's not figure out clever algorithms to look at data in different ways - for example looking in cubic data not strips of data; losing quality by deciding on a loss factor and figuring out if the value is "close enough" for the loss, or breaking graphical data into meaningful slices, e.g. filter the red component of the RGB values, compress, then do green, and so on and so forth... there are many techniques.

Here's the thing...

All we need is an established public library of bit sequences.

I've long held the belief that every image, and indeed every sound, can be generated through tedious incremental change; therefore all the patterns of data already exist.

To that point; all data that will exist already exists.

Don't dwell on that, just run with it and maybe I will write about that some other time.

To record massive data sets as single sequences would be borderline silly because of the indexing time required. So we break the full data set into sensible, optimised, chunks. Then we simply index them against our established public sequence library. We do not store the sequence library itself because they are established public sequences, and remember; we do not store the data.

Then you repeat the process on your newly generated sequence because there will be a public library sequence for that too. You continue to repeat the process until you have an incredibly tiny amount of metadata where all you need to store is the iteration depth and the sequence numbers starting with the seed sequence. Decompression would be the reverse process.

The resulting compressed data only ever consists of metadata to describe what the data was when it did exist.

The public sequence library would contain multiple sequence lengths, so you would check your data size, compare it to the library length that best fits, and as you iterate and compress you step back to the next smallest size until you run out of practical compression.

So depending upon the chunk size for the sequences, a multi-gigabyte file could easily be compressed to a few kilobytes at most, possibly even bytes.

Imagine being able to download an entire UHD film in mere seconds over the crappiest of narrow bandwidth, and saving a decade's worth of images on a tiny drive.

That would be awesome.




Concept (c) 2019 letsbuildathing.com

Digitally Synthesised Telepathy

Telepathy has long been debated and routinely unproven and proven, and has oft been a subject of ridicule; left to sci-fi and the paranormal to explain.

I think it's a very simple concept to comprehend without getting all weirded up.

Ansonlobo [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)], from Wikimedia Commons

Telepathy is the ability to communicate between two or more minds without written or oral communication; mind-to-mind communication. The dictionary definition refers to "known senses".

A typical telepathy test would consist of two subjects purporting to be telepathic and an observer. The subjects are not able to see, hear, or "knowingly sense" each other. "Subject A" looks at a randomly selected card with a shape on it. They communicate what they see telepathically to "Subject B", and "Subject B" describes the shape. The observer, well, they observe. They see the card that "Subject A" saw and they validate that "Subject B" describes it accurately. A 100% consistent pass rate over multiple repetitions means that it's possible. Goof it up and it's not possible.

Some argue it's possible. Others argue it's not. Both cohorts will be able to prove their agenda.

Here's the thing...

When people talk about telepathy, what they really mean is "organic-telepathy". The ability of two organic beasties to communicate together. Doesn't have to be humans.

That's the same definition, only I added "organic" to the mix... so you know what's coming up next, right?

Let's talk about "digital-telepathy" for a moment.

Whoa! What!? Digital telepathy!? Pfft.

Okay, okay, wild, I know... wait up though... let's talk about, radio. Let's talk about WiFi. How about Bluetooth? The list goes on.

Yeah, yeah, I hear you definition-sticklers out there... there's no "magic" in this supposed "digital-telepathy" because it's "known" senses. It's clever stuff that has taken a lot of clever minds to master and get to the point we're at now with digital communication.

So if it's not hard to comprehend digital devices communicating without any physical contact potentially across millions of miles (think space stuff floating around out there talking back to Earth) without a problem, then why is it so hard to comprehend that this is not possible to do organically?

Fact: We know there is a lot about our bodies that we don't fully understand yet.

Fact: We know we have body parts that we can't really explain the purpose of (the appendix for example).

Fact: We even have new organs being discovered that we've so far missed (the interstitium for example... for real!)

Fact: We know some people have different physical attributes than others... we're all very similar; we're not the same though, and that's one of the things that makes us all interesting and beautiful in our own ways.

So, how hard is it to imagine that some people may possess the ability to communicate thoughts (data) through a similar method to those that our digital creations already can?

Imagine some people have a "thing" in their body that allows telepathy that others don't. Imagine that we all do, only we don't know that they're there. Imagine that some us know how to use the "thing" and others don't.

It's not that far fetched.

Based on the facts above we could legitimately come to the conclusion it would be nonsense to suggest that it is simply not possible.

Now here's where it can get really interesting...

Imagine that organic-telepathy is possible between species. In a digital-telepathy world, a smartphone happily communicates with a TV or a sound system, or even a printer (really people?! printers... still!?) not just other smartphones.

Imagine an augmented-telepathy world where we've not really mastered or realised organic-telepathy so we augment our bodies with a digital capability to provide digitally-synthesised-telepathy.

Stop! Let's take a moment to marvel at the name "digitally-synthesised-telepathy". Okay, move along...

Shock horror... we're almost there folks! The smart devices that are emerging today are not too distant from digitally-synthesised-telepathy. We can control digital devices through thought - controlling a mouse cursor on a screen through thought alone is practically old hat now. We can communicate through an array of digital technologies. That's so old it's almost dull. We have digitally enhanced experiences all around us. So really, what I'm suggesting, is that we just need to glue a few clever things together and digitally-synthesised-telepathy will be a thing.

Still got an appetite for something even more interesting to ponder over?

Imagine a world where digitally-synthesised-telepathy is here... okay, here goes...

How do we retain our right to privacy?

How do we ensure our thoughts are not hacked?

How do we differentiate between thought and communication? The voice in your head when you read... that's a thing... would digitally-synthesised-telepathy know the difference?

How do we legislate?

How do we avoid pop-up-thoughts becoming a thing, like the pop-up-ads of today?

How do we avoid DDoS (Distributed Denial of Service) style attacks on our inbound comms? Imagine the noise that would create in your mind. Are people suffering from particular mental illnesses actually organic-telepaths that are experiencing a form of organic-telepathy hacking? Those unexplained head-voices might just be a filter-door swinging on the hinges, or being forced open.

It all sounds so very useful and exciting until you start asking the tricky questions.

Concept (c) 2019 letsbuildathing.com