Flybye. i usually take 84 then head upriver to 19 on the thruway. Im not sure what shops are near newburgh i usually fish furthur north. there is certified on the east side up in port ewen and on the other side there is a shop in red Bank. Both have herring but I dont know if the red bank shop has herring certified for use as bait as per new regs. Certified you can purchase them by boat and use them right there in the river. NO transport by car and that includes eels.
There is a bait guy in Norrie mills state park at the Marina who arrives at 4 30 AM. He was there last year with herring. Outside of those options. You will have to fish your own with a sabiki as gudice suggested or net them via use of a scap net. another option is packaged cut shad or herring. preserved by a method other than just freezing is acceptable.
I plan to use crippled herring lures, storm shads, bloodworms and clams both of which wont be regulated because they arent considered bait fish. and chunking is another term used for cut bait.
Good luck. How far up on 84 are you? I take the Lords Valley, Dingmans Ferry exit home.