Brown, 24, has 104 wickets in 85 T20 appearances. He played a key role as Worcestershire won the Vitality Blast in 2018, in only his second full season, and made his England debut in New Zealand in 2019-20, before a succession of back injuries checked his progress.
He returned to form this season, finishing as Worcestershire's leading wicket-taker with 24 at 22.41.
"I'm very thankful to Worcestershire for providing me with the opportunities and exposure from the age of 17 to become the cricketer I am today," Brown said. "I wish them all the best going forward.
"I'm really excited to be joining Derbyshire and working in a new environment under a proven international coach, Mickey Arthur, who I know will get the best out of me as a player. Becoming a part of the project which is building at Derbyshire and looking to compete for trophies over the next three years appealed to me greatly.
"Having played at The Incora County Ground in this year's Vitality Blast, I was enthused by the prospect of playing my home games here and making full use of the great facilities the ground has to offer in my pursuit of becoming the best cricketer I can be.
"I'm keen to push on in my career and become a valuable asset to the team in all formats. Mickey made it very clear to me how he believes I can become a successful red-ball bowler at Derbyshire with his and the coaching team's help; that vision is extremely exciting for me and played a huge part in my decision to join the club."
Derbyshire finished sixth in the North Group of the T20 Blast this season, losing out on a quarter-final spot to Worcestershire after being beaten in their final group game - in which Brown claimed figures of 3 for 35.
Arthur said: "Pat is one of the most exciting and skilled bowlers on the county circuit and it's a huge boost to be able to bring him in.
"Our bowling attack is versatile and can adapt across formats, but adding a player who has won trophies with his performances just takes us up a level. He has international experience, that's where a lot of our bowlers want to get to, so Pat will fit right into our ambitious setup."