Dark Energy: One of Science's Great Mysteries

You cannot understand the universe if you are afraid of the dark.

There is an energy that permeates the entire universe, from your bones and skin to the deepest reaches of outer space. We cannot see it, hear it or feel it. We do not understand where it comes from or why it is there. In any region of space, this energy is incredibly weak — one hundred trillion times weaker than the heat energy of the air around us.¹ Yet what it lacks in strength, it makes up for in ubiquity. Because it exists not just on Earth but also across even the unthinkably vast void of outer space, where no other energy or matter can be found, its total strength swamps all else. For this reason, its power will shape the future of the universe and determine the fate of all things.

This may sound like science fiction or even new-age spirituality. But by our best understanding of physics and cosmology, this mysterious energy is a scientific reality. It is called dark energy. Though we do not understand its nature and have never directly detected it, its existence is necessary to explain one of the most remarkable discoveries of recent decades — the accelerating expansion of the universe.

What Goes Up Can Keep Going Up

What happens when you throw a tennis ball into the air? Initially, the ball flies upwards very quickly. However, as it rises, the pull of gravity slows it down. This leads to two possibilities. Typically, the pull of gravity is strong enough that it eventually stops the ball from moving upwards altogether, and the ball then falls back to Earth. However, if you are strong enough to throw the ball so hard that it leaves your hand at a speed of more than eleven kilometres per second, there is another possibility. In that case, the ball is moving so fast that gravity is not strong enough to stop it altogether; instead of falling back down, the ball will escape from the Earth and fly out into space, never to return. You are extremely unlikely to manage this, as it would require throwing the ball ten times faster than a gun fires a bullet. But it is at least physically possible.

In either case, what is certain is that the ball will slow down as it rises. If you ever throw a ball into the air and it speeds up as it rises, you will know that something strange is afoot. The only way an object can accelerate while moving upwards from the Earth is if it is powered, meaning it has a source of energy that can propel the object against the pull of gravity. This is how rockets work; they release energy as they travel by burning fuel. If you saw the tennis ball accelerating into the sky, you would have to conclude that it too had some hidden source of energy powering it. But what this energy source could be would be a complete mystery. A physicist who witnessed this phenomenon would be justifiably baffled.

Rockets can accelerate as they travel upwards because they are powered by burning fuel to release energy. This is clearly visible by the flames that can be seen coming out of the back of the rocket. By contrast, an object like a tennis ball with no energy source must be slowed down by gravity as it travels upwards. Image Source: SpaceX and NASA Goddard Space Flight Center, CC BY 2.0, via Flickr.

In 1998, astrophysicists Brian Schmidt, Saul Perlmutter, Adam Riess and their colleagues made just such a perplexing observation. But it was not a mere tennis ball acting so strangely. It was the entire universe! It has long been understood that the universe began, at The Big Bang, in a state of expansion. We can imagine that space was thrown outwards at the beginning of time, and has been growing ever since. However, there is a gravitational attraction between all of the matter and energy in the universe, pulling it all back together and so resisting the universe’s expansion. This is just like how gravity resists a rising tennis ball. Just as a tennis ball may fall back to Earth, the universe could one day reach a maximum extent before contracting back to a point in an event known as The Big Crunch.² Alternatively, like the tennis ball flying off into space, the expansion of the universe could continue forever. However, in either case the expansion should necessarily be slowing over time, just as a rising tennis ball necessarily slows.

Instead, what Schmidt, Perlmutter and Riess discovered was that the expansion of the universe is actually accelerating. Just like a rising ball accelerating away from the Earth, the best explanation for this was that the universe has some hidden source of energy that powers its expansion. Following the earlier example of the similarly enigmatic dark matter, this mysterious energy came to be called dark energy.

From Einstein’s Biggest Blunder to The Worst Prediction in the History of Physics

Remarkably, the existence of such dark energy was actually anticipated by Albert Einstein in 1917. Like most scientists of the time, Einstein assumed that the universe was permanent and unchanging. However, his own theory of general relativity contradicted this, instead describing a universe that is either expanding or contracting. Desperate to correct this apparent defect and reconcile his science with his preconceptions, he added a correction to his equations. This was called the cosmological constant. Interpreted physically, it described a constant background of energy permeating all of space. By setting the cosmological constant to have just the right value, Einstein hoped that this energy could balance out gravity in just the right way to hold the universe steady.

Since then, the cosmological constant has had a curious history. By 1929, astronomical observations had shown that the universe actually is expanding. As a result, the cosmological constant seemed unnecessary to Einstein and so he eliminated it from his equations. In fact, even then there was evidence that this may have been premature, as colleagues pointed out that it could help account for a discrepancy between the predicted and observed age of the universe. But Einstein himself is famously reputed to have lamented his addition of the cosmological constant as “the biggest blunder he made in his entire life” and scientific consensus followed his lead.

However, with the discovery of the accelerating expansion of the universe in 1998, this changed dramatically. Suddenly, the presence of energy permeating all of space was exactly what was needed, and the cosmological constant was the perfect way to account for it. So, the cosmological constant has now been reinstated as the mathematical expression of dark energy and is a vital part of cosmology. As is typical of Einstein, even his biggest blunder turned out to be a stroke of genius.

The evidence for the expansion of the universe, and for the acceleration of the expansion, came from astronomical observations of so-called “standard candles”. These objects shine with a known brightness, so we can determine how far away they are by observing how bright they appear. The galaxies in this image contain examples of standard candles, which are currently being studied to refine our understanding of the expansion rate of the universe. Image Source: European Space Agency, CC BY 2.0, via Flickr.

However, the cosmological constant is merely a mathematical tool, not a scientific explanation. We are still left to wonder: where does dark energy come from? How can even the deepest voids of space, which were assumed to be completely empty, contain this energy?

The good news is that quantum mechanics offers an explanation. The bad news is that the quantity of dark energy predicted by this explanation is so inaccurate that it has been called “the worst theoretical prediction in the history of physics”. According to Heisenberg’s Uncertainty Principle, the energy of a physical system at any particular moment cannot be known exactly. This means that nothing can ever have exactly zero energy (since otherwise we could know that this was the exact amount of energy); there must always be some residual energy present. This is even true of empty space! This means that even empty space must contain some low level of energy. This is called vacuum energy.

Vacuum energy seems an ideal candidate for dark energy! It offers a solution to the puzzle of how empty space can have energy. Peculiarly, however, it does this job too well. When physicists calculate the amount of vacuum energy, they find a number that is at least a hundred trillion trillion trillion trillion times too large. Given such an unprecedentedly poor prediction, it is difficult to conclude that vacuum energy has solved the mystery of dark energy. It has arguably just made it worse.

More Questions Than Answers

Twenty-five years after its discovery, what can we say about dark energy? It has a name, but one that only highlights its mystery. It has a mathematical formulation through the cosmological constant, but one that has no justification from physics. It has a potential explanation as vacuum energy, but one that is so inaccurate that it just leaves more questions than answers. The truth is that dark energy remains an enigma. It is one of science’s greatest mysteries.

That we remain so ignorant of the phenomenon that accounts for the majority of energy in the universe may seem dispiriting. However, it can also be seen as a positive sign for the future of science. Major advances in the history of physics have often emerged from unexplained phenomena or sharp contradictions between expectations and reality. The anomalous motion of planets in the sky inspired Nicolaus Copernicus to conclude that the Earth revolved around the Sun rather than vice versa. The failure of classical physics to explain the emission of light from hot objects helped to inspire the development of quantum mechanics. Perhaps the puzzle of dark energy will similarly motivate the next great scientific theory.

When observed from Earth, Mars sometimes undergoes apparent retrograde motion, meaning it path across the sky doubles back on itself. This anomalous movement of planets motivated scientists to question the geocentric model of the universe at which the Earth was assumed to be at the centre. Perhaps the discovery of the anomaly of the accelerating expansion of the universe will eventually motivate a similar revolution in our understanding of nature. Image Source: Eugene Alvin Villar, CC BY-SA 4.0, via Wikimedia Commons.

Science thrives on difficult questions. New understanding rarely comes from the familiar; it is found instead in the unknown. Probing dark energy can be unsettling because it sharply exposes our ignorance. When confronted with the vast mysteries of the universe, it seems that centuries of scientific progress amount only to a dim torch illuminating a fraction of all there is to know. To learn more means exploring what remains hidden. After all, you cannot understand the universe if you are afraid of the dark.

1. The heat energy of a cubic metre of air is approximately 100 kJ (see The Surprising Physics of the Star Wars Force), while a cubic metre of space contains approximately one trillionth of a kilojoule of dark energy.

2. Of course, a ball hitting the ground does not have to stop; it can bounce back off the ground and into the air again. Similarly, it has been hypothesised that a Big Crunch could be followed by a Big Bounce which would serve as a Big Bang for a new universe.