Crystal shards has a cast time and an animation timed to coincide to with that cast time. If you are spamming that ability, what difference does it make if you press the key for it during the current cast or waiting until the cast ends? Either way, it doesn't start casting again until the cast time is over.
Since we are in arm chair dev mode, let me put on my systems analyst and QA hat, which I happen to have a little experience in.
What you said about no GCD and spamming ability keys is appropriate is your expected behavior as an end user. Unless Grave chimes in, no one here knows what the designed behavior actually is.
If an animation is playing and the user presses another key, what is supposed to happen? Should the animation get clipped and the next ability immediately trigger or should the next ability queue up and not trigger until the previous animation has completed. I certainly don't know what the official designed behavior is.
I can speculate that one possible explanation for those of you who are having weapon swap delay problems and who also happen to be ability spammers is that the weapon swap keystroke is put in the same queue as your ability keystrokes and the weapon swap doesn't occur until all of the preceding abilities you spammed are completed.