In species with a heterogametic sex, population genetics theory predicts that DNA sequences on the X chromosome can evolve faster than comparable sequences on autosomes. Both neutral and non-neutral evolutionary processes can generate this pattern. Complex traits like gene expression are not predicted to have accelerated evolution by these theories, yet a "faster-X" pattern of gene expression divergence has recently been reported for both Drosophila and mammals. Here, we test the hypothesis that accelerated adaptive evolution of cis-regulatory sequences on the X chromosome is responsible for this pattern by comparing the relative contributions of cis- and trans-regulatory changes to patterns of faster-X expression divergence observed between strains and species of Drosophila with a range of divergence times. We find support for this hypothesis, especially among male-biased genes, when comparing different species. However, we also find evidence that trans-regulatory differences contribute to a faster-X pattern of expression divergence both within and between species. This contribution is surprising because trans-acting regulators of X-linked genes are generally assumed to be randomly distributed throughout the genome. We found, however, that X-linked transcription factors appear to preferentially regulate expression of X-linked genes, providing a potential mechanistic explanation for this result. The contribution of trans-regulatory variation to faster-X expression divergence was larger within than between species, suggesting it is more likely to result from neutral processes than positive selection. These data show how accelerated evolution of both coding and non-coding sequences on the X chromosome can lead to accelerated expression divergence on the X chromosome relative to autosomes.