Earlier this year, Warren Buffett invested in Capital One Financial (COF 0.22%) for the first time and cut back on other bank stocks, like US Bancorp. The move may have taken some by surprise as the bank -- the ninth largest in the country with some $469 billion in assets -- has been a spotty performer over the years. 

Over the last 20 years it has posted an annualized return of about 3.6%, and over the past 10 years, the annualized return is just 4.6%. Both of these numbers trail the S&P 500

However, during the past three years, Capital One has doubled the large-cap benchmark, posting an annual return of 17.2%. As one of the largest credit card issuers, Capital One has benefited in recent years from a post-pandemic surge in digital spending, and it should continue to take advantage of the gradual shift to digital payments. It is also dirt cheap, with a price-to-earnings ratio of less than 8 while trading below book value with a price-to-book ratio of 0.73. Maybe those factors appealed to Buffett, the chairman and chief executive officer of Berkshire Hathaway

So, let's look at the past three years. If you invested $10,000 in Capital One just after the start of the pandemic in August 2020, how much would you have today?

Beating the benchmark -- and its rivals

On Aug. 20, 2020, Capital One closed at $63.80 per share. Three years later, on Aug. 20, 2023, the stock opened at $105.27. That's a 65% increase with a roughly 17.2% annualized return per year. If you reinvested the dividend over those three years, that's about a 20% return per year. And that includes the bear market of 2022, when Capital One fell 35%.

Over that same three-year period, the S&P 500 was up 8.9% on an annualized basis, and 10.6% with the dividend reinvested. So, Capital One has far outperformed the benchmark in the past three years.

If you invested $10,000 in Capital One back on Aug. 20, 2020, you'd have seen that increase to about $17,280 -- without adding a cent to the investment. If you added $100 per month to that initial $10,000 investment, you'd have about $22,035.

COF Chart

Data source: YCharts

Among the top 10 U.S. banks over that three-year period, only Wells Fargo had a better return. As the chart shows, it beat such banking stalwarts as JPMorgan Chase and Goldman Sachs, along with everyone else.

Buffett buys more

Based on these numbers, Capital One has clearly been a buy over the past three years. But is it a buy right now? Buffett certainly thinks so, as he added Capital One in the first quarter for the first time and then boosted his position in the second quarter.

I think it is a stock worth considering. But it has been very cyclical, even more so than typical banks because of its reliance on consumer spending, since credit cards are its primary revenue generator.

While the economy has performed better than expected, uncertainties remain. In the second quarter, Capital One has seen increases in net charge-offs and nonperforming loans, and it has boosted its provisions for credit losses. I certainly wouldn't sell it, but there may be better options out there among bank stocks, or credit card issuers.