The number of warriors of the Israelites who came out of Egypt were 603,550, but the number who came out of Babylonian captivity did not come close to that number. Ezra 2.2–58 gives the families and their numbers,

Those who came with Zerubbabel were Jeshua, Nehemiah, Seraiah, Reelaiah, Mordecai, Bilshan, Mispar, Bigvai, Rehum, and Baanah. The number of the men of the people of Israel… (Ezra 2:2–58).

Then in verse 64, Ezra gave the number as 42,360. That was the sad tale of the sin of Israel and Judah. The once great nation had been reduced to almost nothing, and during the 70-year captivity, they grew some, but not as in the former days.