I'm going to give basic information here that will be broad, please do not take any of this information as an insult to your intelligence. I don't know what you know and I just want you and others to be safe, I'd hate to see anyone here get hurt or killed.
Chatter can be caused by a multitude of issues and these things should be considered first before moving forward with such a large bit/job. Running long bits is dangerous and shouldn't be used unless your machine/collet/spindle are designed to run such things. I believe I have mentioned it here before but I can't remember so I'll just tell it again. We had a guy running a test on the production floor with a large bit that was not designed for the machine/spindle type and caused it to come out of the chuck at a very high RPM, it shot across the floor bouncing all over the place and impaled in his leg barely missing his main artery. We were forbidden to run anything over 2.5" in length unless the machine was surrounded by a bulletproof glass barrier. Be very careful here, just because you have done it before doesn't mean something similar won't happen to you, it just takes once to hurt, maim or kill. Also keep in mind that drywall will not stop something moving so fast, so keep your loved ones safe if you're doing this in your garage. It's some scary shit when you see a bit bouncing around at 18k-20k rpms and having no idea where it's going to go next.
Chatter Main issues
- Collet depth
- If you do not have enough of the tool inside the collet this will cause wobble/chatter. This can cause the collet to become worn out and eject the bit. This will also cause your spindle bearings to wear out.
- Carriage harmonics
- This is the best way I know how to explain this, if the carriage is not built to withstand the type of pressure being applied at those lengths this will cause the entire carriage to chatter and tear itself apart. Only our largest machines were capable of using such large bits.
- Spindle size
- This ties in with the two items above, larger spindles are designed for larger bits. They can withstand the horizontal pressure applied with such lengths. Think of a cheater bar used to bust a nut loose, you're applying a ton more pressure to the bearings/collet/carriage/etc. When using a bit of this length.
- Bit quality
- Ensure you are using a good quality bit, if a bit at that length is not balanced properly there is nothing you can do to stop the chatter.
- Bite size
- Ensure the cut depth and bite size being cut out per pass are very shallow. This will help reduce chatter.
- Flutes/chip load/etc.
- Number of flutes, upcut/downcut, RPM, Cut speed, etc. will all play an important role in reducing chatter.
- Going too slowly with a cut-speed will cause the bit to bounce off the wood, it will also cause the bit to overheat which can increase the size of the bit and cause it to make more contact with the wood causing even more chatter. This will also burn up/dull the bit very quickly which will compound the issue further.
- If your bit is a downcut this will also put more pressure/load on the bit which will cause more chatter.
I'm not saying you can't do this, you can. You just need to be careful, very careful. Mechanics will play a huge roll in the chatter, Check all your bolts make sure they are tight, ensure your collet is not worn out, ensure your spindle and collet are designed for such a lengthy bit. if all this looks good you need to experiment with your RPM/Bit Site/Cut Speed until you can reduce chatter, more than likely with a smaller machine and a bit that long you are going to be cutting very slowly. Upcut/Downcut bits can have dramatic effects upon the wobble as well. If you are having to go so slow that it's burning the wood then you may need to re-think your process. If the bit is too long there is simply nothing you can do to prevent chatter and it will have to be cleaned up post process.
The best thing about CNC machines is there is always a undiscovered solution to your problem or a common one. If you plan on cutting things this deep in the future you can always get two pieces that are half as thick and cut them in two jobs then use a dowel pin or whatever to join them back together. You could also try to use a shorter bit to go down as deep as you can to reduce the amount of material you are cutting out then finish out with the longer bit.
As for feed and speed tinkering, I'm not as well versed as others in this field. I typically looked at the Onsrud manual, changed my settings and went on my way unless it sounded funny. Then I'd adjust on the fly until it sounded right, which were typically minor adjustments.
Please be careful.