So, aged 18, he took himself off to a kibbutz in Israel and then backpacked around Europe, before ending up back in Hampstead, wondering what to do.
"It was a glorious summer, and I now had a taste for the outdoor life, so I started with a local landscaping firm doing garden maintenance and then building gardens. I worked with a lovely hippie gardener, Antonia Sturges, who taught me something new every day. It was a really fun two years."
But then wanderlust took hold again, and he was off to Australia, crewing a yacht and working as a jobbing gardener, eventually returning to London and setting up a stall at Camden Lock market.
It wasn't until 1990, when he was 25, that he decided to take gardening seriously, and enrol on the design course at the English Gardening School in Chelsea. Ever the entrepreneur, he supplemented the income from his few design jobs by setting up a company that could build the gardens designed by his fellow students. And he started a retail outlet, a contemporary garden shop in Islington called The Plant Room.
The Modular Garden concept crystallised in 2004. "Every year, we had people ringing us up, saying they had seen a design makeover on television, and asking if we could design and build a garden for them for a few thousand quid.
We couldn't; the costs of a traditional designing job, where you work from scratch each time, and have to go through brief, tender, specifications, etc, were just too high. But I hated turning them away. There had to be a way of bringing good design to people with a more modest budget, as Conran did with Habitat."
Finally, after discussions with his business partner, Nick McMahon, and construction director, Allon Hoskin, the formula emerged.
"By standardising a lot of the construction details, using the same process and in-house team for every job, and knowing in advance what everything is going to cost, we could work more quickly, and have a slicker operation."
And linking up with house builders, such Laing Homes and David Wilson Homes, would enable him to crank up the volume of gardens made, and again keep prices down.
Today, the business employs half a dozen construction teams, plus office staff, and designs and builds more than 100 gardens a year. At present it works only within a two-and-a-half hour radius of London, although the plan is to go national. Gardens start at £7, 000.
You might expect a very "cloned" product, but because the design is tailored to each client and site, and the "finish" - the plants and furnishing - is a relatively small part of the overall cost and can be tweaked in any number of ways, the gardens do end up fairly different.
In spite of all the television makeovers, the quality of most of the nation's small gardens, as judged by what one sees on train journeys, remains pretty basic. This is not idleness, suggests Joe charitably, but bafflement.
"People don't know how to get started and they are confused by all the mystique around plants and gardening. The really satisfying thing for me is being able to help these people get a better garden. Why should professional design be confined to the very rich?"
I agree, and if he can make good gardens as commonplace as good kitchens, it will be a wonderful thing.
Joe's design tips for small gardens
- Make a masterplan before you begin.
- Break the garden up into three distinct areas to make it more inviting to explore: for example, have stepping stones through plants in the central space, and sunny and shady sitting areas at either end.
- Create a sense of space by taking paving the full width of the garden.
- Incorporate diagonals, angles and asymmetry in the design to disguise the rectangular shape of the garden and make a more interesting route through it.
- Introduce variety by having changes in level - old railway sleepers are an easy construction material, and don't usually need foundations.
A good lawn is high-maintenance: think hard about whether you want one, and if you do, ensure there is a way of walking down the garden without having to step on the lawn when wet or frosty.
- Verbena bonariensis: elegant lilac perennial.
- Pittosporum tobira 'Nanum': compact evergreen shrub.
- Amelanchier lamarckii: small tree that performs in spring and autumn.
- Polystichum setiferum: fine-fronded hardy fern.
- Astelia chathamica: hardy perennial with silver, sword-shaped leaves.
- Olea europaea: olive tree.