The short answer is no … That said, you don’t come to Rules Guy for short answers, so let me expand a skosh. (Not that Rules Guy is paid by the word or anything!)
There are no restrictions on removing loose impediments, but be careful not to accidentally move your ball, which would result in a one-stroke penalty (except when the ball is on the putting green) and require you to return the ball to its original position.
Also, don’t take so long that you’re put on the clock, please.
Got a question about the Rules? Ask the Rules Guy! Send your queries, confusions and comments to rulesguy@golf.com. We promise he won’t throw the book at you.