Caldwell was a 4-year starter at Defensive Back while with the Hawkeyes. He became a graduate assistant in 1977 under Bob Commings and that began his coaching career. He would then leave Iowa for Southern Illinois as their Wide Receiver Coach, a title he would hold for Colorado and Louisville. He also pit-stopped for a year at Northwestern as an Offensive Assistant and found his way to Penn State as a Wide Receiver Coach, only to take over the next year as Quarterbacks Coach under Joe Paterno. It was there that he coached Kerry Collins when he was named the best Quarterback in the country by winning the Davey O’Brien Award. He remained there for 7 years before being named the Head Coach of the Wake Forest Demon Deacons. He remained there for 7 years as well, while giving the Demon Deacons their first winning season in 6 seasons and being selected for a bowl game as well. Jim eventually made his way into the NFL in 2001 with the Tampa Bay Buccaneers as their Quarterbacks Coach, but followed Tony Dungy to Indianapolis in the same role. In 2005 he added the title of Assistant Head Coach and was a big part of the Colts winning the Super Bowl with his tutelage of Peyton Manning. In 2009 he took over as Head Coach for the Indianapolis Colts when Dungy retired after the season. He became the first ever rookie Head Coach to start a season 14-0 in 2009, before the Colts were beaten in Week 16 by the New York Jets. The game was considered controversial because Coach Caldwell decided to rest the majority of his starters to keep them healthy for the playoffs, thus not striving for the perfect season.  However, that move looked to be the right one as the Colts beat the Ravens and the Jets on their way to Super Bowl XLIV, where they were defeated by the New Orleans Saints. Caldwell coached the Colts to a 10-6 record in the 2010 season, but a timeout blunder late in the game contributed to the 1st round playoff loss to the New York Jets. His job became significantly tougher in 2011, when Peyton Manning was forced to miss the entire season due to a neck injury. His 2011 results:

Week 1 vs. Houston Texans: Loss (34-7)
Week 2 vs. Cleveland Browns: Loss (19-27)
Week 3 vs. Pittsburgh Steelers: Loss (20-23)
Week 4 vs. Tampa Bay Buccaneers: Loss (17-24)
Week 5 vs. Kansas City Chiefs: Loss (24-28)
Week 6 vs. Cincinnati Bengals: Loss (17-27)
Week 7 vs. New Orleans Saints: Loss (7-62)
Week 8 vs. Tennessee Titans: Loss (10-27)
Week 9 vs. Atlanta Falcons: Loss (7-31)
Week 10 vs. Jacksonville Jaguars: Loss (3-17)
Week 11: Bye
Week 12 vs. Carolina Panthers: Loss (19-27)
Week 13 vs. New England Patriots: Loss (24-31)
Week 14 vs. Baltimore Ravens: Loss (10-24)
Week 15 vs. Tennessee Titans: Win (27-13)