Mine also does this. Also clunks when switching out of 4wd back to 2wd more so when stopped. Started switching down to 2wd when coasting. Don't think I've heard it clunck switching into 4wd. Non-hybrid, if that makes any difference.
Some clunking when switching out of 4wd does not surprise me. The clunk when it is already in 4wd does and makes me wonder about the transfer case. I hope Toyota has some explanation or remedy for this.