More than 180 people were injured in the accident, the country’s civil defence authority said on Twitter.
Images from the scene showed part of a huge red crane crashed through the mosque’s roof. The cause of the collapse is not known.
Mecca is currently preparing for the annual Muslim Hajj pilgrimage.
Image caption Images posted on social media showed what appeared to be the collapsed crane
Hundreds of thousands of people are expected to arrive in the Saudi city from all over the world later this month.
Reports suggested the crane fell on the east side of the mosque following high winds and rain. The Arabian peninsula has been has been hit by strong sand storms over the past week.
A video posted on YouTube, which could not be independently verified, appeared to record the moment the crane fell, with a loud crash heard in the background followed by panic and shouting.
Images circulating on Twitter appeared to show numerous bodies and blood on the floor of the mosque.
The mosque is Islam’s most sacred site and the destination for millions of Muslims undertaking the Hajj pilgrimage every year.
Saudi authorities began a major expansion of the site last year to increase the area of the mosque by 400,000 square metres (4.3 million square feet), to allow it to accommodate up to 2.2 million people at once.
More than three million people undertook the Hajj pilgrimage to Mecca in 2012. Saudi authorities have taken steps in 2013 to limit the number of people involved. Large numbers of people have resulted in several tragedies over the years, including a stampede in 2006 that killed nearly 350 people.
At the heart of the Grand Mosque is the Kaaba, a black cube-shaped building which Muslims all over the world face when they pray.
Islam requires that every Muslim capable of doing so performs a pilgrimage to the site at least once in their lifetime. Once at the mosque, pilgrims perform “tawaf” – walking seven times around the Kaaba in a counter-clockwise direction.