I left the skin on mine too. It certainly won't come out crunchy, but it was delicious.
How do you use your dutch oven to cook the chicken, exactly? I'm sure the final product is better with other methods, but the set it and forget it nature is just too easy with the slow cooker and it is...