And that's exactly what get_all_category_ids() does internally, except it passes 'category' instead of 'post_tag' for the taxonomy name.

Seems like there should be a get_all_term_ids( $tax_name = 'category' ) function added to core, and possibly deprecate get_all_category_ids.