Always brush felt with soft, brisk strokes in one direction only, toward whichever way the "nap" (the direction of the cloth along which pool balls roll fastest) runs.
Nap, the grain of the cloth can be discerned with the palm of the hand. It is the direction toward which the grain feels smoothest, providing the lowest possible resistance to a rolling ball. The nap is installed in one direction, usually from the head rail toward the table's foot, for a fast break shot.
A felt brush may not do enough work for pool shooters with sensitive skin. Chalk or talcum, and dirt on the table, mixed with perspiration, can irritate hands and discolor them and the cloth.
If you need to, gently vacuum the felt with a dust buster or tube vacuum. Vacuum sparingly, because felt can become pried loose from the slate.
