Taliban insurgents in the northern Kunduz Province last month Photograph: STRINGER/AFGHANISTAN/REUTERS

Eight years after they were overthrown by US air power, a drumbeat is starting to sound across Afghanistan in favour of talking to the Taliban, the country's once-hated former rulers. An idea that used to seem absurd, if not defeatist, is coming to be seen as the only credible way to end an ever-widening war. Moreover, the proposed agenda of negotiations is not a Taliban surrender, but an offer to share power in Kabul.

President Hamid Karzai and other senior Afghan politicians support the idea. So too do a growing number of foreign governments, including Britain's – at least tentatively – now that British troops are being killed at twice the rate they were in early 2009.

Perhaps most surprisingly, even among Afghanistan's small but determined group of woman professionals, the notion of making a deal with the ultra-conservative men who forced them into burkas and denied them the right to work outside the home is no longer anathema. A desperate desire for peace is trumping concern over human rights.

Given the sense of liberation that accompanied the Taliban's defeat in 2001, the new mood seems barely credible. For five years, the 20th century's most brutal form of male chauvinism had held sway across 90% of Afghanistan. It was accompanied by other bizarre efforts to revert to a pre-modern age.

I was one of the few journalists in Kabul as the Taliban swept up from Kandahar to take control of the Afghan capital in 1996, prompting the mujahideen warlords to abandon resistance and flee. The sudden shift left everyone stunned, but the crowds that came out to watch the Taliban's pick-up trucks roaring around the streets were mainly supportive. The bearded young Islamists with their promise of social justice seemed to offer an end to the fighting between rival mujahideen leaders that had devastated large parts of the city and forced hundreds of thousands into refugee camps abroad.

We watched in wonder as they ripped cassettes out of cars and hung the tapes from lampposts like brown streamers fluttering in the breeze. Crates of whisky and brandy were dragged out of the cellars of the Intercontinental hotel and dumped into the road for a tank to roll over, a ceremony of fundamentalist solemnity that rapidly became farcical as its Taliban driver, succumbing to the fumes, backed and advanced on an increasingly erratic orbit.

Young Taliban gunmen ran into hospitals, ordering male doctors to grow beards and female doctors to go home. Burkas, once worn only by poorer women in the bazaar, became compulsory for all women. Taliban thugs flayed the ankles of anyone who showed even an inch of bare skin below the regulation new hemlines. But even as repression grew women could still be heard saying that their family's new-found safety from the civil war's shells and rocket-fire made it worth it.

A similar calculus of security-versus-rights is re-emerging now. Three years ago, when I was last in Kabul and the Taliban were only just starting their comeback on the battlefield, defeating them was the watchword of the day. There has been a tectonic shift in Afghanistan's public mood since then. It is prompted by a host of factors: growing disappointment with western governments and the ineffectiveness of billions of dollars in aid that seems to go nowhere except into the bank accounts of foreign consultants or local politicians; a sense that there can be no military solution to the new civil war and that outsiders are deliberately prolonging it; grief and despair over the mounting toll of civilian casualties, many caused by US airstrikes; rising nationalist anger and a feeling of humiliation; and a desire to return to an Afghan consensus in which Afghans create their own space and find their own solutions. Karzai's recent outbursts against the Americans and other foreigners are no aberration. They reflect a widely held mood.

Over two afternoons, I sit down over tea with a group of six women professionals. If anyone should be suspicious of the Taliban, it would be educated women like these. In varying degrees they all favour negotiations. Though they do not want their names used, so I will identify them by the letters A to F.

A is a Pashtun, Afghanistan's largest ethnic group and the one from which almost all Taliban come. She was already a refugee in Pakistan when the Taliban took over, having fled in 1993 at the height of the civil war. She only returned to Kabul after the Taliban were overthrown.

B, also a Pashtun, lived under Taliban rule. She feels the US, Pakistan and other foreigners are manipulating the war and even have the elusive Taliban leader, Mullah Omar, under their influence. I encounter this sense of the Taliban as puppets, even victims, in numerous conversations with Afghan men as well as women.

"It's an excuse for foreigners to occupy Afghanistan and stay here," says A. "That's why the war continues. It's not a war against the Taliban. It's a war for their own objectives."

B says Taliban rule had positive as well as negative sides. As a woman, you couldn't work, "but if you were walking in the street no one could kidnap you. We felt safer than now, when there are all these security guards and other people with guns who can abduct a woman at any time."

C raises Osama bin Laden, the al-Qaida leader who is as invisible as Mullah Omar in his presumptive hideout over the border in Pakistan. "There is something going on behind the curtain. It's politics. They could find Saddam, but they can't find Bin Laden. When they wanted to find Saddam, they did."

C is a Tajik, one of Afghanistan's three other ethnic groups, along with the Hazara and Uzbek people. She was 13 when the Taliban captured Kabul and sent all schoolgirls back home. Because of a health emergency her father took the family to Karachi for a year. She recalls her fear when a Taliban militiaman stopped their taxi on the way and found a pin-up of a young woman stuck to the back of the mirror. He beat the driver mercilessly. When the family returned to Kabul, she could not go to school for four years. Yet, in spite of these bitter memories she now supports negotiations with the Taliban.

D, another Pashtun, spent the Taliban period as a refugee in Pakistan. "When we were there, we were afraid of the Taliban. I came back here in 2002 and just didn't want the Taliban to exist. Then I began to realise they are also Afghans and Pakistan is using them," she says.

A key question is whether the Taliban leadership's eight years out of power have changed their thinking. Would they really try to turn the clock back for women a second time? E, a Pashtun, says there were always good and bad Taliban. "Some were educated and religious, but others joined them from Pakistan, and then criminals piled in."

F, a Tajik, says she has noticed Taliban members presenting themselves as nationalists more than Islamists these days. "There are two kinds of Taliban: those who want a strict implementation of sharia law, and those who want to get rid of US forces. I'm not very hopeful that the Taliban leaders who want to negotiate won't be killed by our neighbours," she says. She points out that Mullah Abdul Ghani Baradar, one of the Taliban's top commanders, was recently captured in Pakistan. Now being held incommunicado, he had helped to initiate preliminary talks between Taliban representatives and intermediaries of the Karzai government in Saudi Arabia.

Some western experts on Afghanistan also claim to detect a difference between the old Taliban and the "neo-Taliban". The movement has certainly changed its position on communication technology. Where it used to ban TV, it now has a sophisticated propaganda machine regularly commenting on the latest developments, as well as a website that offers statements, interviews and DVDs. The Taliban are also more diverse and fragmented. In some areas commanders ban music at weddings; in others they permit it.

Anders Fänge, the country director of the Swedish Committee for Afghanistan, a large aid agency, has spent around 20 years in the country, also working as a journalist and a UN official. The Taliban should never have been portrayed in the black-and-white terms that Bush and Blair used, he says. During their period in power they often turned a blind eye to the discreet "home schools" where teachers taught girls in people's flats or family compounds. "In 1998 the Taliban governor of [the central Afghan city] Ghazni told me, 'We know you have these girls' schools, but just don't tell me about them.' A Taliban minister even approached me and said, 'I have two daughters. Can you get them in?'" he recalls.

Similar attitudes exist today, he says. In Wardak, a province close to Kabul that is heavily contested by Taliban and Nato forces, "we don't have much problem with the Taliban," says Fänge. "They accept girls' schools and women doctors. They just ask for two hours of Islamic education in schools, that teachers grow beards and not spread propaganda against the Taliban."

The difficulty comes from foreign Taliban, the Pakistanis and Arabs, or Taliban from other provinces. "At the local level, it's a patchwork, a mosaic of local commanders, who may recognise Mullah Omar as their spiritual leader but are not under his control," he adds.

Fänge's points support the case, rarely mentioned by western politicians, that Taliban conservatism differs from the rest of the country in degree, not in kind. Afghanistan is a largely rural society where the oppression of women runs deep. Even in villages populated by Tajik, Hazara and Uzbek, Afghan women are routinely banned by husbands or fathers from leaving the family compounds, and girls are kept out of school, according to Afghan women reporters.

One morning I drive to a marble-fronted two-storey house in a Kabul suburb that was ruined in the mid-1990s fighting but is now reviving as a favoured address for Afghanistan's nouveaux riches. Wearing tinted glasses and a long green-and-purple-striped chapan (the signature garb of the elderly that President Karzai has made famous) a tall dignified man greets me. Arsalan Rahmani was deputy minister of higher education and later minister of Islamic affairs in the Taliban government. Four years ago Karzai invited him back to Kabul and made him a senator. He accepts the Taliban made a string of mistakes. "They didn't have good management, they were young, they had no experts, doctors, and couldn't run ministries. My boss was a boy of 25, who couldn't even sign an official letter."

He describes reports of restrictions on girls' education and women being denied the chance to work as false. "That wasn't their idea, then or now. We didn't let girls go to school because of lack of security. There was a war on. But now in Pakistan, Taliban girls go to school and university. My son is a doctor and I want him to marry a lady doctor. I've got three daughters. During the Taliban time they were in Pakistan and all studied there."

He goes on to tell an incredible story. "When I was deputy minister of higher education, people came to me and said they had girls who had finished school and wanted to study medicine. I consulted Mullah Omar and he authorised us to set up rooms in a central Kabul hospital, now called Daoud Khan hospital, where women could study to become doctors. Around 1,200 graduated, and if you track them down you'll see my signature on their degree certificates," he says.

I have no time to follow his advice but I do locate Shukria Barakzai, an independent woman MP who stayed in Afghanistan throughout the Soviet occupation, the four-year rule by mujahideen warlords, and the Taliban period. She confirms the senator's story.

Like many educated Kabulis, she criticises the warlords as strongly as the Taliban (during the warlords' clashes she lost a son and daughter). She too favours talks with the Taliban. "I changed my view three years ago when I realised Afghanistan is on its own. It's not that the international community doesn't support us. They just don't understand us. Everybody has been trying to kill the Taliban but they're still there, stronger than ever. They are part of our population. They have different ideas but as democrats we have to accept that. Every war has to end with talks and negotiations. Afghans need peace like oxygen. People want to keep their villages free of violence and suicide bombers."

Her relaxed attitude to the Taliban stems, in part, from confidence that they cannot win again. "They no longer have the support and reputation they had back then. Taliban is an ideology. It's no longer a united force," she says.

If Afghan women now overwhelmingly want talks with the Taliban, the same is true of many of the country's male politicians, particularly the Pashtun. They want "a rebalancing of forces" in Afghan society, as a former minister who wanted to remain anonymous put it. The US invasion in 2001 put the warlords of the so-called Northern Alliance in power, but failed to produce stability. "In October 2001 the Taliban controlled 90% of Afghanistan, while the Northern Alliance had 10%. After December 2001 the Northern Alliance had 70% and the country's majority group, the Pashtun, were marginalised. Now this needs to change. There's an Afghan consensus on that," he says.

In recognition of the new mood, and in the hope of getting the Tajik warlords on board, Karzai has called 1,200 leading Afghans – politicians, tribal elders, and representatives of civil society – to a national consultation (known as a jirga) on peace later this month. The aim is to get their endorsement for his strategy of talks with the Taliban and the rest of the armed opposition, which also includes Hezb-i-Islami (the party of Islam), run by a fierce Pashtun warlord, Gulbuddin Hekmatyar.

There are many roadblocks, not least the reluctance of the Obama administration. It has nailed its colours to two masts. One is a strategy of "re-integration", aimed at winning the so-called $10-a-day Taliban footsoldiers back to the government side, partly by the "hearts and minds" investment in schools, clinics and other government services that is supposed to follow the current offensives in Helmand and Kandahar, and partly by offering them money to start new lives. The cash bribes irritate other Afghans who never joined the Taliban and now feel unrewarded. They also doubt the strategy will work since it requires surrender before the main Taliban grievance, the US occupation, is resolved.

The other US strategy, led by the senior commander in Afghanistan, General Stanley McChrystal, is to use a massive surge of US forces (by the end of this year they will have tripled from the number left by Bush) to knock the Taliban back. He pays lip service to the notion of negotiation but he wants to inflict a severe blow on the Taliban first.

Yet senior European diplomats in Kabul have little hope in the surge. In February it took 16,000 US troops to capture Marjah, a district that is home to a few thousand people in Helmand province. General McChrystal's plan is to recapture 40 districts this year and another 40 next year, but if progress is as slow as the Marjah operation, he is going to need 20 years, not two. In any case, reports from the ground in Marjah suggest US "control" is patchy. The Taliban went underground during the offensive but emerge after nightfall to punish or kill people who collaborate with the Americans.

Having invested in the surge, Obama is bound to stick to it until after the mid-term Congressional elections in November. He has promised to review his Afghan policy in December. "The Democrats will suffer losses, and I fear there'll be a panic in Washington. Obama will realise the surge hasn't worked, but then what?" one European diplomat in Kabul tells me.

He may decide to muddle through, on the grounds that if he starts to withdraw significant numbers of troops, the Republicans will accuse him of weakness as the 2012 presidential campaign begins. Then he risks losing his core Democratic base, which is increasingly disllusioned with the Afghan war. The alternative is to start talking to the Taliban leadership about a ceasefire and power-sharing in Kabul.

The trouble, as diplomats see it, is that Obama has not even authorised the CIA to put out feelers to the Taliban leadership on a "deniable" basis, a common way of initiating contacts. Nor has he begun to prepare the American public for the notion that the Taliban may not be demons but necessary negotiating partners. It would be as massive a U-turn in US policy as it was for the British government to talk to the IRA.

In the meantime, who can take the first steps on what will be a long and difficult road? David Miliband, Britain's foreign secretary, made a cautious effort to differentiate UK from US policy in March when he loyally supported the surge but advocated talks with the Taliban, though he did not mention their name in his key sentence. "The idea of political engagement with those who would directly or indirectly attack our troops is difficult," he said. "The Afghans must own, lead and drive such political engagement. It will be a slow, gradual process. But the insurgents will want to see international support. International engagement, for example under the auspices of the UN, may ultimately be required."

The UN ran the negotiations that got the Soviet Union's agreement to withdraw in 1988. UN diplomats later tried to broker a deal for a coalition government in Kabul in 1992. Today the man best placed to do the job is Staffan de Mistura, a dapper diplomat of Swedish and Italian origin with long experience of Afghanistan. He took up his new post as the UN special representative in March and holds court in a spacious rented villa known as Palace Seven. With a garden that could take six tennis courts, it hides behind three checkpoints in the heart of what the army of security guards in Kabul call the city's green zone – though, in truth, it is less formidable than Baghdad's version.

De Mistura's first job is to overcome the rows over fraud in last year's elections which pitted Karzai against the UN. The new UN envoy will be helping to organise parliamentary elections, due for September, which could be equally fractious. But De Mistura also wants to help in facilitating negotiations with the armed opposition.

Getting talks going with the Taliban will be not be easy. Their current position is to reject them on the grounds that Karzai is a puppet and the US is still committed to war. ("They are nurturing to turn Afghanistan into a hotbed of long-term colonialist conspiracies and ambitions," they said on their English-language website last month.)

Even if talks were to start, the agenda will be challenging. The first item must be for all sides to agree to a ceasefire. Bringing Afghanistan's neighbours into the process via a regional contact group will also be crucial. Yet Obama's most recent speech on Afghanistan failed to mention the regional dimension, even though it is clear that Pakistan, and to a much lesser extent Iran, have the potential of stoking tension if their views are ignored.

For the US, a timetable for withdrawal will be tough for Washington to accept. On the Taliban side, decisions will have to be made on whether to abandon the hope of victory and share power, especially if it means accepting the new Afghan constitution as the basis for a deal. They are bound to demand changes, so what concessions on sharia law and women's rights are Karzai and Afghan's other politicians likely to make?

Human rights groups hope to use this month's peace jirga to lay down red lines. "Women are afraid that the Taliban will come in without any conditions being put by the government," says Farida Nekzad, who runs the Wakht news agency and is widely regarded as one of the country's bravest journalists.

But the obstacles to a peace deal will not get smaller as time goes by. By and large, Afghans are ready. When will Obama get on board? Karzai's visit to the White House next week would be a good moment but, sadly, it seems unlikely.