The Williams sisters were seeking their 14th Grand Slam doubles title.
They hadn't played together for two years until joining up to win at Wimbledon in July, then followed that by taking the doubles gold at the London Olympics, where they beat Kirilenko and Petrova in the semifinals. Kirilenko and Petrova wound up taking the bronze.
Monday's defeat ends Venus Williams' U.S. Open. She lost in the second round of singles.
Serena Williams moved into the singles quarterfinals with a 6-0, 6-0 victory over Andrea Hlavackova earlier Monday.