fuel-gallery

Fuel Gallery does exactly what I was looking for. A slider with thumbnails, captions below my images, creating folders, show my images by tags, easy to manage and a great support. The only downside is that this plugin doesn't get updates regularly. It still works great on the new WordPress version but by installing you will see the text that isn't tested with the new version yet. However Matt told me the update is coming soon.

EDIT 9/1/2020 The developer Matt has been super responsive with resolving issues, you can tell he takes pride in this product. Like most plugins on WordPress, there are some responsive issues on Mobile on the backend, but I suspect those will be resolved soon. I believe this will become one of the better gallery plugins on WordPress in the future. 5 stars because of the support from the developer, the free trial, and the very reasonably price, and the quickly increasing features.