The hotel we stayed (which was VERY clean and nice for an older hotel) had a shuttle service to downtown Victoria. We were dropped off a couple blocks away from the harbor and we just walked around the city.
The Empress Hotel. The Parliament buildings, etc. We walked around the old city and into China Town - at one point using a very narrow alley.
We decided that we'd do the Butchart Gardens in the evening. We were told that the gardens are very pretty when the landscape lighting comes on (around 8pm).