Finance Minister Arun Jaitley on Thursday said banks should have acted earlier to stop Vijay Mallya from going abroad, even as he warned of "appropriate action" against those responsible for any "inaction" in recovery of dues totalling over Rs 9,000 crore in the case.