There's a Japanese proverb that warns: "The nail that sticks out gets hammered down."

And it's this phrase I'm thinking of as I trip over an ancient spike on a thousand-year-old pilgrim path in Japan's emerald Kii Mountains.

Hiking along the ancient Kumano Kodo trail – which connects tiny hamlets with winding forest paths, babbling brooks, thatched teahouses and stunning views – the rusty spike is my introduction to Japan's "abode of the gods", a region considered so sacred that everyone from Edo emperors to contemporary octogenarians has graced its mountain paths.

While still relatively unknown to international visitors, the Kumano Kodo is a particularly well-trodden trail: Japanese have been hiking these mountains for more than a millennium, and antique rice paper paintings depict crowds of samurai, Kyoto emperors and kimono-clad women in veils and straw sandals, traversing the rocks and cedars, in search of enlightenment.

The Kumano Kodo comprises seven sacred trails along the Kii mountain range, taking in Buddhist and Shinto shrines where trekkers have historically come to pray, refocus and engage in purification rituals.

It's one of only two pilgrimages in the world with UNESCO world-heritage status – the other being Spain's Camino de Santiago – and its cultural significance shouldn't be overlooked.

Japan can sometimes seem overwhelmingly impenetrable to Western visitors. It's a place steeped in ancient rituals, yet also formidably alien. There are ghost-faced geishas ducking into alleyways, toilets with heated seats that flush automatically and cafes where you can pay women in pyjamas to give you a cuddle or remove your earwax.

But the nation's staggering beauty, delicious cuisine, and opportunity to step both back and forward in time, is truly a rare treat.

We begin our trip in Tokyo, at one of the capital's newest and most chic hotel. Built on the site of a former castle in the upscale Toranomon neighbourhood, the Andaz Tokyo is an ear-popping elevator ride up Tokyo's second-highest building.

By day, the city stretches out as far as the eye can see, the Paris-inspired Tokyo Tower and luscious royal parks serving as landmarks. At night, the glittering capital below is best enjoyed with a cocktail from the 52nd-storey rooftop bar.

Around the Andaz are tiny vestiges of old Japan: a shop selling 11th-century samurai swords with gloriously intricate designs and a 400-year old Shinto shrine where locals give thanks to the god of fire as tourists sit amid koi ponds and fluttering butterflies.

But Tokyo wouldn't be Tokyo without a trip to some of its wackier destinations, so we spend a day in Shibuya, hobnobbing with Japanese teens dressed as Victorian dolls, then head to the nation's only "goat cafe", where the two resident beasts stare curiously as we sip our bubble tea.

We end the evening at the Robot Restaurant, where a fire-breathing gorilla swings onto the stage on a trapeze and battles a spider robot.

It's a mad futuristic mix of kabuki theatre, Japanese anime, Liberace sequins and retro Vegas decor – the best fun I've had in ages.

After a few days in the city, it's time to seek some spiritual salvation, so we take a bullet train to Osaka, and connect to another train to Tanabe city in the Kii peninsula, to start our Kumano Kodo hike.

Our five-day 40km trek follows one of the most popular routes, starting at Takijiri-oji (a short drive from Tanabe) and ending at the waterfall shrine in Kumano Nachi Taisha.

While much of the Kumano Kodo is accessible by car or bus, our goal is to take in as much as possible by hiking, while sleeping in traditional ryokan (Japanese inns) and minshuku (family-run guesthouses).

We hike along dappled paths by day and at night, exhausted, are treated to eight-course kaiseki dinners by our kind hosts-cum-chefs, gorging on a splendid mix of freshly prepared sushi, locally grown vegetables and home-made soba noodles.

These days, it's mostly foreigners who engage in the longer treks that can take anywhere from four to 14 days, depending on the route, ambition and amount of time one has. The Japanese prefer weekends or day-long hikes, simply because of their long working hours, says Tanabe city's Kumano tourism manager Brad Towle.

"The Japanese consider these shrines 'power spots' where they can recharge with good energy," he says, as we begin our steep hike up into the mountains, a cushion of cypress leaves at our feet.

We have the trail almost to ourselves, but along the way meet Shizu, a Japanese nurse from Nagoya.

"Since I turned 40 – an important milestone in my life – I've been thinking a lot about my past and my future," Shizu tells us over a picnic lunch of rolled sushi, hot tea and pickled salad, packed earlier for us by our minshuku hosts.

"Something called 'God' has kept me alive all these years here in Japan, and I want to show my gratitude – I want to live the rest of my life with humility."

The final leg of our journey is an ancient one, as we travel like pilgrims on a traditional boat along the turquoise Kumano River. When we arrive at the spectacular Kumano Nachi Taisha shrine, set against a 133m-long waterfall cascading over a huge granite and cypress outcrop, hundreds of locals have gathered to pray to the "mother goddess", asking for help trying to conceive or to bless newly pregnant women.

As I take in the scene around me, one of the Shinto priests comes over to offer prayer "tips". "You must pray with your full name and address so that the goddess can find you and deliver your wish," she tells me.

So, despite the raised eyebrows from my partner, I do what the locals do: I throw a gold coin into the box, ring the big bell, bow twice, clap twice, tell the goddess my prayer (along with my address, quietly wondering if she delivers internationally), and bow again. Our trekking friend Shizu makes a separate prayer for me and claps delightedly with hope.

Let's just say the Japanese have their rules for a reason: if you follow instructions, sometimes you're lucky enough to get what you're looking – or praying – for.

* Kate Hodal was a guest of Inside Japan Tours (www.insidejapantours.com)