Thousands were invited to the ceremony in Delhi, including regional leaders and Bollywood stars.
But notable names were excluded, including Pakistan’s PM Imran Khan, and some opposition leaders stayed away.
The election was seen as a referendum on Mr Modi, 68, who is adored by many but also accused of stoking divisions.
He ran a campaign dominated by his personal image and national security after tensions with Pakistan spiked in February in the disputed region of Kashmir.
The ceremony is taking place at the presidential palace, a colonial-era building in the centre of the capital.
A new cabinet has also been sworn in, and includes the chief of Mr Modi’s Hindu nationalist Bharatiya Janata Party (BJP). Amit Shah is credited with steering the party to victory in the election, the results of which were announced last week.
Mr Modi retained his seat of Varanasi by a margin of nearly half a million votes. He has become the first leader since 1971 to secure a single party majority in back-to-back elections.
His government will be under pressure to deal with a slowdown in India’s economy and provide more jobs.
The new government suffered a blow on Wednesday when Finance Minister Arun Jaitley said he was standing down for health reasons.
The majestic presidential palace or Rashtrapati Bhawan, home to the Indian president, was the venue for the ceremony.
Everyone wanted to get a glimpse of Mr Modi, who earned a massive mandate in the recent elections. Many of the over 7,000 guests had arrived early.
As Mr Modi was being sworn in for a second term, his supporters rose from their seats and applauded enthusiastically.
Loud cheers were also reserved for Mr Modi’s friend and BJP president Amit Shah, who was credited with securing this resounding victory for the party.
Foreign leaders and key opposition and regional politicians were in attendance, as were industrialists, Bollywood stars and sporting legends.
Karan Johar, Rajinikanth and Kangana Ranaut were among the film stars attending, as were leaders from several neighbouring countries – including Bangladesh, Sri Lanka, Myanmar, Nepal and Bhutan.
Congress party leader Rahul Gandhi – who led the main opposition that lost to Mr Modi and the BJP – attended alongside his mother, Congress veteran Sonia Gandhi.
State leaders, some of whom who had run against the BJP in the elections, were also invited. Local media reported that an invitation had also been extended to Bill Gates.
The chief minister of West Bengal, Mamata Banerjee, a strong critic of Mr Modi, was one of the opposition leaders who turned down the invitation.
She said she would not be attending after reading “media reports” quoting the BJP as saying 54 people had been murdered in “political violence” in her state.
“This is completely untrue,” she said in a statement.
During the campaign, Ms Banerjee and Mr Modi regularly traded barbs as the BJP courted voters in West Bengal.
It is not surprising that Pakistani PM Imran Khan was not invited. Tensions between Pakistan and India dominated the election after more than 40 paramilitary troops were killed in Indian-administered Kashmir on 14 February.
It was the deadliest attack against Indian forces in Kashmir in decades and brought the nuclear-armed powers to the brink of war.