Colombia, who join group winners France in the last four, needed all three points at the Stade Geoffroy Guichard to go through and seized them in the 68th minute when Hernandez pounced after a mix-up in the Japanese midfield.
Japan, who had gone into the game requiring only a draw to qualify, leave the eight-nation tournament after having kept the South Americans in check before Hernandez's strike and hitting the woodwork just after the re-start.
France, who next face the Group B runners-up, crushed New Zealand 5-0 in Sunday's other tie in Paris. Colombia will play the winners of Group B, currently led by Cameroon, who have already qualified.
Colombia deserved their win for having chased the game against a well-organised Japan side who showed discipline at the back, until the blunder, and more class going forward.
Although they failed to create a clear-cut scoring chance, Francisco Maturana's men kept the Japanese defence on their toes throughout the first half.
HERNANEDEZ KEY
The South Americans, whose best moves all passed through Hernandez, came closest to scoring with an angled chip shot from Victor Aristizabal which beat Seigo Narazaki but sailed over the bar.
Japan's defence looked otherwise comfortable with the Colombian pressure while skipper Hidetoshi Nakata provided the menace at the other end.
The creative midfielder should have done better with a glancing header from Nobuhisa Yamada's cross, but made up for it with a deft through-ball for Yoshito Okubo, who jinked twice before firing wide.
The match finally sparked into life within minutes of the re-start when Yamada provided a long ball from the right flank and Naohiro Takahara met it with a header that slammed against the post with Oscar Cordoba beaten.
Colombia replied immediately when a defensive lapse allowed Aristizabal a half-volley which Narazaki blocked superbly with his legs.
But Hernandez made no mistake in the 68th minute after a mix-up between Yasuhito Endo and Tsuneyasu Miyamoto.
Miyamoto gave the ball away, Hernandez pounced and then swivelled to steer a low shot past Narazaki.
Japan promptly laid siege to a Colombian defence which, though missing its skipper Ivan Cordoba through suspension, weathered the storm to book their place in the last four.
Colombia, who had lost 1-0 to France and beaten New Zealand 3-1, finished with six points to Japan's three, after a win over the Kiwis, while France topped the group after three wins.
Cameroon, on six points, need only a draw on Monday against the United States to finish top of Group B, while the other semi-finalists will be either Turkey (3pts) or Brazil (3pts), who also meet on Monday.
